Sur les conjectures de Gross et Prasad. I

Table of contents :
1. Introduction
2. Classical groups and restriction of representations
3. Selfdual and conjugate-dual representations
4. The centralizer and its group of components
5. Local root numbers
6. Characters of component groups
7. L-groups of classical groups
8. Langlands parameters for classical groups
9. Vogan L-packets - Desiderata
10. Vogan L-packets for the classical groups
11. Vogan L-packets for the metaplectic group
12. The representation of H and generic data
13. Bessel and Fourier-Jacobi models for GL(n)
14. Restriction Problems and Multiplicity One Theorems
15. Uniqueness of Bessel Models
16. Uniqueness of Fourier-Jacobi Models
17. Local Conjectures
18. Compatibilities of local conjectures
19. Reduction to basic cases
20. Variant of the local conjecture
21. Unramified parameters
22. Automorphic forms and L-functions
23. Global Restriction Problems
24. Global conjectures: central values of L-functions
25. Global L-parameters and Multiplicity Formula
26. Revisiting the global conjecture
27. The first derivative
References
1. Introduction
2. Discrete series parameters
3. Depth zero supercuspidals
4. Branching laws for GLn(Fq)
5. Branching laws for Un(Fq)
6. Langlands-Vogan packets for small unitary groups
7. Theta correspondence
8. Endoscopic packets and theta correspondence
9. Skew-hermitian case: U(1) U(1)
10. Restriction from U(2) to U(1)
11. Theta correspondence for U(2) U(2)
12. Trilinear forms for U(2)
13. Restriction from U(3) to U(2): endoscopic case
14. Restriction from U(3) to U(2): stable case
15. A global argument
16. A finer global argument
References
Introduction
1. Notations et rappels
2. Fonctions très cuspidales
3. Majorations pour le groupe linéaire GLk
4. Majorations pour un groupe spécial orthogonal
5. Entrelacements tempérés
6. Expression spectrale de la limite d'une intégrale
7. Une formule intégrale calculant la multiplicité; application
Références
Références

Citation preview

Astérisque 346, 2012, p. 1–109

SYMPLECTIC LOCAL ROOT NUMBERS, CENTRAL CRITICAL L-VALUES, AND RESTRICTION PROBLEMS IN THE REPRESENTATION THEORY OF CLASSICAL GROUPS by Wee Teck Gan, Benedict H. Gross & Dipendra Prasad

Abstract. — In this paper, we provide a conjectural recipe for the restriction of irreducible representations of classical groups (including metaplectic groups), to certain subgroups, generalizing our earlier work on representations of orthogonal groups. Our conjectures include the cases of Bessel and Fourier-Jacobi models. In fact, it is the standard representation of the classical group, together with its orthogonal, symplectic, hermitian, or skew-hermitian form, that plays the primary role, and not the classical group alone. All of our conjectures assume the Langlands parametrization. For classical groups over local fields, the recipe involves local epsilon factors associated to the Langlands parameter and certain summands of a fixed symplectic representation of the L-group. For automorphic representations over global fields, it involves the central critical value of this symplectic L-function. Résumé (Nombres de racines locales symplectiques, L-valeurs critiques centrales et problèmes de restriction en théorie de représentation des groupes classiques) Dans cet article, nous donnons une recette conjecturale pour la restriction à certains sous-groupes des représentations irréductibles de groupes classiques. Cela inclut les groupes métaplectiques et généralise notre travail antérieur pour les groupes orthogonaux. Nos conjectures comprennent les cas des modèles de Bessel et Fourier-Jacobi. En fait le rôle principal est joué, non par le groupe seul, mais par la représentation naturelle de ce groupe classique, munie de sa forme bilinéaire-orthogonale, symplectique, hermitienne ou anti-hermitienne selon le cas. Dans toutes nos conjectures, nous admettons que la paramétrisation de Langlands est établie. Notre recette, pour les groupes classiques sur les corps locaux, fait intervenir les facteurs epsilon locaux associés au paramètre de Langlands et certains facteurs d’une représentation symplectique fixée du L-groupe. Pour les représentations automorphes sur des corps globaux, elle fait intervenir la valeur, au centre de la bande critique, de la fonction L-symplectique correspondante.

1. Introduction It has been almost 20 years since two of us proposed a rather speculative approach to the problem of restriction of irreducible representations from SOn to SOn−1 2010 Mathematics Subject Classification. — 22E50, 22E55, 11F70, 11R39. Key words and phrases. — Classical groups, metaplectic groups, branching laws, Gross-Prasad conjectures, local root numbers, central critical L-value.

© Astérisque 346, SMF 2012

2

W. T. GAN, B. H. GROSS & D. PRASAD

[24, 25]. Our predictions depended on the Langlands parametrization of irreducible representations, using L-packets and L-parameters. Since then, there has been considerable progress in the construction of local L-packets, as well as on both local and global aspects of the restriction problem. We thought it was a good time to review the precise conjectures which remain open, and to present them in a more general form, involving restriction problems for all of the classical groups. Let k be a local field equipped with an automorphism σ with σ 2 = 1 and let k0 be the fixed field of σ. Let V be a vector space over k with a non-degenerate sesquilinear form and let G(V ) be the identity component of the classical subgroup of GL(V ) over k0 which preserves this form. There are four distinct cases, depending on whether the space V is orthogonal, symplectic, hermitian, or skew-hermitian. In each case, for certain non-degenerate subspaces W of V , we define a subgroup H of the locally compact group G = G(V ) × G(W ) containing the diagonally embedded subgroup G(W ), and a unitary representation ν of H. The local restriction problem is to determine d(π) = dimC HomH (π ⊗ ν, C), where π is an irreducible complex representation of G. The basic cases are when dim V −dim W = 1 or 0, where ν is the trivial representation or a Weil representation respectively. When dim V − dim W ≥ 2, this restriction problem is also known as the existence and uniqueness of Bessel or Fourier-Jacobi models in the literature. As in [24] and [25], our predictions involve the Langlands parametrization, in a form suggested by Vogan [70], and the signs of symplectic root numbers. We show that the Langlands parameters for irreducible representations of classical groups (and for genuine representations of the metaplectic group) are complex representations of the Weil-Deligne group of k, of specified dimension and with certain duality properties. We describe these parameters and their centralizers in detail, before using their symplectic root numbers to construct certain distinguished characters of the component group. Our local conjecture states that there is a unique representation π in each generic Vogan L-packet, such that the dimension d(π) is equal to 1. Furthermore, this representation corresponds to a distinguished character χ of the component group. For all other representations π in the L-packet, we predict that d(π) is equal to 0. The precise statements are contained in Conjectures 17.1 and 17.3. Although this material is largely conjectural, we prove a number of new results in number theory and representation theory along the way: (i) In Proposition 5.2, we give a generalization of a formula of Deligne on orthogonal root numbers to the root numbers of conjugate orthogonal representations. (ii) We describe the L-parameters of classical groups, and unitary groups in particular, in a much simpler way than currently exists in the literature; this is contained in Theorem 8.1. (iii) We show in Theorem 11.1 that the irreducible representations of the metaplectic group can be classified in terms of the irreducible representations of odd special

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

3

orthogonal groups; this largely follows from fundamental results of Kudla-Rallis [44], though the statement of the theorem did not appear explicitly in [44]. (iv) We prove two theorems (cf. Theorems 15.1 and 16.1) that allow us to show the uniqueness of general Bessel and Fourier-Jacobi models over non-archimedean local fields. More precisely, we show that d(π) ≤ 1 (cf. Corollaries 15.3, 16.2 and 16.3), reducing this to the basic cases when dim W ⊥ = 0 or 1, which were recently established by [4], [64] and [76]. The same theorems allow us to reduce our local conjectures to these basic cases, as shown in Theorem 19.1. One subtle point about our local conjecture is its apparent dependence on the choice of an additive character ψ of k0 or k/k0 . Indeed, the choice of such a character ψ is potentially used in 3 places: (a) the Langlands-Vogan parametrization (which depends on fixing a quasi-split pure inner form G0 of G, a Borel subgroup B0 of G0 , and a non-degenerate character on the unipotent radical of B0 ); (b) the definition of the distinguished character χ of the component group; (c) the representation ν of H in the restriction problem. Typically, two of the above depend on the choice of ψ, whereas the third one doesn’t. More precisely, we have: — in the orthogonal case, none of (a), (b) or (c) above depends on ψ; this explains why this subtlety does not occur in [24] and [25]. — in the hermitian case, (a) and (b) depend on the choice of ψ : k/k0 → S1 , but (c) doesn’t. — in the symplectic/metaplectic case, (a) and (c) depend on ψ : k0 → S1 , but (b) doesn’t. — in the odd skew-hermitian case, (b) and (c) depend on ψ : k0 → S1 , but (a) doesn’t. — in the even skew-hermitian case, (a) and (c) depend on ψ : k0 → S1 but (b) doesn’t. Given this, we check in §18 that the dependence on ψ cancels out in each case, so that our local conjecture is internally consistent with respect to changing ψ. There is, however, a variant of our local conjectures which is less sensitive to the choice of ψ, but is slightly weaker. This variant is given in Conjecture 20.1. Finally, when all the data involved are unramified, we state a more refined conjecture; this is contained in Conjecture 21.3. After these local considerations, we study the global restriction problem, for cuspidal tempered representations of adelic groups. Here our predictions involve the central values of automorphic L-functions, associated to a distinguished symplectic representation R of the L-group. More precisely, let G = G(V ) × G(W ) and assume that π is an irreducible cuspidal representation of G(A), where A is the ring of adèles of a global field F . If the vector space HomH(A) (π ⊗ ν¯, C) is nonzero, our local conjecture implies that the global root number (π, R, 21 ) is equal to 1. If we assume π to be tempered, then our calculation of global root numbers and the general conjectures of Langlands

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

4

W. T. GAN, B. H. GROSS & D. PRASAD

and Arthur predict that π appears with multiplicity one in the discrete spectrum of L2 (G(F )\G(A)). We conjecture that the period integrals on the corresponding space of functions Z f 7→

f (h) · ν(h) dh H(k)\H(A)

gives a nonzero element in HomH(A) (π ⊗ ν¯, C) if and only if the central critical L-value L(π, R, 21 ) is nonzero. This first form of our global conjecture is given in §24, after which we examine the global restriction problem in the framework of Langlands-Arthur’s conjecture on the automorphic discrete spectrum, and formulate a more refined global conjecture in §26. For this purpose, we formulate an extension of Langlands’ multiplicity formula for metaplectic groups; see Conjecture 25.1. One case in which all of these conjectures are known to be true is when k = k0 × k0 is the split quadratic étale algebra over k0 , and V is a hermitian space over k of dimension n containing a codimension one nondegenerate subspace W . Then ∼ GLn (k0 ) × GLn−1 (k0 ) and H = ∼ GLn−1 (k0 ). G= Moreover, ν is the trivial representation. When k0 is local, and π is a generic representation of G = GLn (k0 ) × GLn−1 (k0 ), the local theory of Rankin-Selberg integrals [34], together with the multiplicity one theorems of [4], [3], [66], [67] and [76], shows that dim HomH (π, C) = 1. This agrees with our local conjecture, as the Vogan packets for G = GLn (k0 ) × GLn−1 (k0 ) are singletons. If k0 is global and π is a cuspidal representation of G(A), then π appears with multiplicity one in the discrete spectrum. The global theory of Rankin-Selberg integrals [34] implies that the period integrals over H(k)\H(A) give a nonzero linear form on π if and only if L(π, stdn ⊗ stdn−1 , 1/2) 6= 0, where L(π, stdn ⊗ stdn−1 , s) denotes the tensor product L-function. Again, this agrees with our global conjecture, since in this case, the local and global root numbers are all equal to 1, and ∨ R = stdn ⊗ stdn−1 + std∨ n ⊗ stdn−1 . In certain cases where the global root number  = −1, so that the central value is zero, we also make a prediction for the first derivative in §27. The cases we treat are certain orthogonal and hermitian cases, with dim W ⊥ = 1. We do not know if there is an analogous conjecture for the first derivative in the symplectic or skew-hermitian cases. In a sequel to this paper, we will present some evidence for our conjectures, for groups of small rank and for certain discrete L-packets where one can calculate the distinguished character explicitly. We should mention that in a series of amazing papers [77, 78, 74, 75] and [53], Waldspurger and Mœglin-Waldspurger have established the local conjectures for special orthogonal groups, assuming some natural properties

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

5

of the characters of representations in tempered L-packets. There is no doubt that their methods will extend to the case of unitary groups. Acknowledgments. — W. T. Gan is partially supported by NSF grant DMS-0801071. B. H. Gross is partially supported by NSF grant DMS 0901102. D. Prasad was partially supported by a Clay Math Institute fellowship during the course of this work. We thank P. Deligne, S. Kudla, M. Hopkins, M. Reeder, D. Rohrlich, and J.-L. Waldspurger for their help. We also thank the referee for his/her careful reading of the paper and for his/her numerous useful comments, corrections and suggestions. 2. Classical groups and restriction of representations Let k be a field, not of characteristic 2. Let σ be an involution of k having k0 as the fixed field. If σ = 1, then k0 = k. If σ 6= 1, k is a quadratic extension of k0 and σ is the nontrivial element in the Galois group Gal(k/k0 ). Let V be a finite dimensional vector space over k. Let h−, −i : V × V → k be a non-degenerate, σ-sesquilinear form on V , which is -symmetric (for  = ±1 in k × ): hαv + βw, ui = αhv, ui + βhw, ui hu, vi =  · hv, uiσ . Let G(V ) ⊂ GL(V ) be the algebraic subgroup of elements T in GL(V ) which preserve the form h−, −i: hT v, T wi = hv, wi. Then G(V ) is a classical group, defined over the field k0 . The different possibilities for G(V ) are given in the following table. (k, )

k = k0 ,  = 1

k = k0 ,  = −1

k/k0 quadratic,  = ±1

G(V )

orthogonal group O(V )

symplectic group Sp(V )

unitary group U(V )

In our formulation, a classical group will always be associated to a space V , so the hermitian and skew-hermitian cases are distinct. Moreover, the group G(V ) is connected except in the orthogonal case. In that case, we let SO(V ) denote the connected component, which consists of elements T of determinant +1, and shall refer to SO(V ) as a connected classical group. We will only work with connected classical groups in this paper. If one takes k to be the quadratic algebra k0 × k0 with involution σ(x, y) = (y, x) and V a free k-module, then a non-degenerate form h−, −i identifies the k = k0 × k0 module V with the sum V0 + V0∨ , where V0 is a finite dimensional vector space over k0 and V0∨ is its dual. In this case G(V ) is isomorphic to the general linear group GL(V0 ) over k0 .

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

6

W. T. GAN, B. H. GROSS & D. PRASAD

If G is a connected, reductive group over k0 , the pure inner forms of G are the groups G0 over k0 which are obtained by inner twisting by elements in the pointed set H 1 (k0 , G). If {gσ } is a one cocycle on the Galois group of the separable closure k0s with values in G(k0s ), the corresponding pure inner form G0 has points G0 (k0 ) = {a ∈ G(k0s ) : aσ = gσ agσ−1 }. The group G0 is well-defined up to inner automorphism over k0 by the cohomology class of gσ , so one can speak of a representation of G0 (k0 ). For connected, classical groups G(V ) ⊂ GL(V ), the pointed set H 1 (k0 , G) and the pure inner forms G0 correspond bijectively to forms V 0 of the space V with its sesquilinear form h, i (cf. [40, § 29D and § 29E]). Lemma 2.1. — 1. If G = GL(V ) or G = Sp(V ), then the pointed set H 1 (k0 , G) = 1 and there are no nontrivial pure inner forms of G. 2. If G = U(V ), then elements of the pointed set H 1 (k0 , G) correspond bijectively to the isomorphism classes of hermitian (or skew-hermitian) spaces V 0 over k with dim(V 0 ) = dim(V ). The corresponding pure inner form G0 of G is the unitary group U(V 0 ). 3. If G = SO(V ), then elements of the pointed set H 1 (k0 , G) correspond bijectively to the isomorphism classes of orthogonal spaces V 0 over k with dim(V 0 ) = dim(V ) and disc(V 0 ) = disc(V ). The corresponding pure inner form G0 of G is the special orthogonal group SO(V 0 ). Now let W ⊂ V be a subspace, which is non-degenerate for the form h−, −i. Then V = W + W ⊥ . We assume that 1) 2)

 · (−1)dim W W

⊥

⊥

= −1

is a split space.

When  = −1, so dim W ⊥ = 2n is even, condition 2) means that W ⊥ contains an isotropic subspace X of dimension n. It follows that W ⊥ is a direct sum W ⊥ = X + Y, with X and Y isotropic. The pairing h−, −i induces a natural map Y −→ Homk (X, k) = X ∨ which is a k0 -linear isomorphism (and k-anti-linear if k 6= k0 ). When  = +1, so dim W ⊥ = 2n + 1 is odd, condition 2) means that W ⊥ contains an isotropic subspace X of dimension n. It follows that W ⊥ = X + Y + E, where E is a non-isotropic line orthogonal to X + Y , and X and Y are isotropic. As above, one has a k0 -linear isomorphism Y ∼ = X ∨. Let G(W ) be the subgroup of G(V ) which acts trivially on W ⊥ . This is the classical group, of the same type as G(V ), associated to the space W . Choose an X ⊂ W ⊥ as above, and let P be the parabolic subgroup of G(V ) which stabilizes a complete flag

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

7

of (isotropic) subspaces in X. Then G(W ), which acts trivially on both X and X ∨ , is contained in a Levi subgroup of P , and acts by conjugation on the unipotent radical N of P . The semi-direct product H = N o G(W ) embeds as a subgroup of the product group G = G(V ) × G(W ) as follows. We use the defining inclusion H ⊂ P ⊂ G(V ) on the first factor, and the projection H → H/N = G(W ) on the second factor. When  = +1, the dimension of H is equal to the dimension of the complete flag variety of G. When  = −1, the dimension of H is equal to the sum of the dimension of the complete flag variety of G and half of the dimension of the vector space W over k0 . We call a pure inner form G0 = G(V 0 ) × G(W 0 ) of the group G relevant if the space W 0 embeds as a non-degenerate subspace of V 0 , with orthogonal complement isomorphic to W ⊥ . We note: Lemma 2.2. — Suppose k is non-archimedean. (i) In the orthogonal and hermitian cases, there are 4 pure inner forms of G = G(V ) × G(W ) and among these, exactly two are relevant. Moreover, among the two relevant pure inner forms, exactly one is a quasi-split group. (ii) In the symplectic case, there is exactly one pure inner form of G = G(V ) × G(W ), which is necessarily relevant. (iii) In the skew-hermitian case, there are 4 pure inner forms of G = G(V )×G(W ), exactly two of which are relevant. When dim V is odd, the two relevant pure inner forms are both quasi-split, and when dim V is even, exactly one of them is quasi-split. Proof. — The statement (i) follows from the fact that an odd dimensional split quadratic space is determined by its discriminant and that there is a unique split hermitian space of a given even dimension. The statements (ii) and (iii) are similarly treated. Given a relevant pure inner form G0 = G(V 0 ) × G(W 0 ) of G, one may define a subgroup H 0 ⊂ G0 as above. In this paper, we will study the restriction of irreducible complex representations of the groups G0 = G(V 0 ) × G(W 0 ) to the subgroups H 0 , when k is a local or a global field.

3. Selfdual and conjugate-dual representations Let k be a local field, and let k s be a separable closure of k. In this section, we will define selfdual and conjugate-dual representations of the Weil-Deligne group W D(k) of k. When k = R or C, we define W D(k) as the Weil group W (k) of k, which is an extension of Gal(k s /k) by C× , and has abelianization isomorphic to k × . A representation of W D(k) is, by definition, a completely reducible (or semisimple) continuous homomorphism ϕ : W D(k) → GL(M ),

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

8

W. T. GAN, B. H. GROSS & D. PRASAD

where M is a finite dimensional complex vector space. When k is non-archimedean, the Weil group W (k) is the dense subgroup I oF Z of Gal(k s /k), where I is the inertia group and F is a geometric Frobenius. We normalize the isomorphism W (k)ab → k × of local class field theory as in Deligne [13, 14, 15], taking F to a uniformizing element of k × . This defines the norm character | − | : W (k) → R× ,

with |F | = q −1 .

We define W D(k) as the product of W (k) with the group SL2 (C). A representation is a homomorphism ϕ : W D(k) → GL(M ) with (i) ϕ trivial on an open subgroup of I, (ii) ϕ(F ) semi-simple, (iii) ϕ : SL2 (C) → GL(M ) algebraic. The equivalence of this formulation of representations with that of Deligne [13, 14, 15], in which a representation is a homomorphism ρ : W (k) → GL(M ) and a nilpotent endomorphism N of M which satisfies Adρ(w)(N ) = |w| · N , is given in [26, § 2, Proposition 2.2]. We say two representations M and M 0 of W D(k) are isomorphic if there is a linear isomorphism f : M → M 0 which commutes with the action of W D(k). If M and M 0 are two representations of W D(k), we have the direct sum representation M ⊕M 0 and the tensor product representation M ⊗ M 0 . The dual representation M ∨ is defined by the natural action on Hom(M, C), and the determinant representation det(M ) is defined by the action on the top exterior power. Since GL1 (C) = C× is abelian, the representation det(M ) factors through the quotient W (k)ab → k × of W D(k). We now define certain selfdual representations of W D(k). We say the representation M is orthogonal if there is a non-degenerate bilinear form B :M ×M →C which satisfies

(

B(τ m, τ n) = B(m, n) B(n, m) = B(m, n),

for all τ in W D(k). We say M is symplectic if there is a non-degenerate bilinear form B on M which satisfies ( B(τ m, τ n) = B(m, n) B(n, m) = −B(m, n), for all τ in W D(k). In both cases, the form B gives an isomorphism of representations f : M → M ∨,

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

9

whose dual f ∨ : M = M ∨∨ → M ∨ satisfies f ∨ = b · f,

with b = the sign of B.

We now note: Lemma 3.1. — Given any two non-degenerate forms B and B 0 on M preserved by W D(k) with the same sign b = ±1, there is an automorphism T of M which commutes with W D(k) and such that B 0 (m, n) = B(T m, T n). Proof. — Since M is semisimple as a representation of W D(k), we may write M M= V i ⊗ Mi i

as a direct sum of irreducible representations with multiplicity spaces Vi . Each Mi is either selfdual or else Mi∨ ∼ = Mj for some i 6= j, in which case dim Vi = dim Vj . So we may write ! ! M M ∨ M= Vi ⊗ Mi ⊕ Vj ⊗ (Pj + Pj ) i

j

with Mi irreducible selfdual and Pj irreducible but Pj  Pj∨ . Since any non-degenerate form B remains non-degenerate on each summand above, we are reduced to the cases: (a) M = V ⊗ N with N irreducible and selfdual, in which case the centralizer of the action of W D(k) is GL(V ); (b) M = (V ⊗ P ) ⊕ (V ⊗ P ∨ ), with P irreducible and P  P ∨ , in which case the centralizer of the action of W D(k) is GL(V ) × GL(V ). In case (a), since N is irreducible and selfdual, there is a unique (up to scaling) W D(k)-invariant non-degenerate bilinear form on N ; such a form on N has some sign bN . Thus, giving a W D(k)-invariant non-degenerate bilinear form B on M of sign b is equivalent to giving a non-degenerate bilinear form on V of sign b · bN . But it is well-known that any two non-degenerate bilinear forms of a given sign are conjugate under GL(V ). This takes care of (a). In case (b), the subspaces V ⊗ P and V ⊗ P ∨ are necessarily totally isotropic. Moreover, there is a unique (up to scaling) W D(k)-invariant pairing on P × P ∨ . Thus to give a W D(k)-invariant non-degenerate bilinear form B on M of sign b is equivalent to giving a non-degenerate bilinear form on V . But any two such forms are conjugate under the action of GL(V ) × GL(V ) on V × V . This takes care of (b) and the lemma is proved. When M is symplectic, dim(M ) is even and det(M ) = 1. When M is orthogonal, det(M ) is an orthogonal representation of dimension 1. These representations correspond to the quadratic characters χ : k × → h±1i.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

10

W. T. GAN, B. H. GROSS & D. PRASAD

Since char(k) 6= 2, the Hilbert symbol gives a perfect pairing (−, −) : k × /k ×2 × k × /k ×2 → h±1i. We let C(d) be the one dimensional orthogonal representation given by the character χd (c) = (c, d). We also note the following elementary result: Lemma 3.2. — If Mi is selfdual with sign bi , for i = 1 or 2, then M1 ⊗ M2 is selfdual with sign b1 · b2 . Proof. — If Mi is selfdual with respect to a form Bi of sign bi , then M1 ⊗ M2 is selfdual with respect to the tensor product B1 ⊗ B2 which has sign b1 · b2 . Next, assume that σ is a nontrivial involution of k, with fixed field k0 . Let s be an element of W (k0 ) which generates the quotient group W (k0 )/W (k) = Gal(k/k0 ) = h1, σi. If M is a representation of W D(k), let M s denote the conjugate representation, with the same action of SL2 (C) and the action τs (m) = sτ s−1 (m) for τ in W (k). We say the representation M is conjugate-orthogonal if there is a non-degenerate bilinear form B : M × M → C which satisfies ( B(τ m, sτ s−1 n) = B(m, n) B(n, m) = B(m, s2 n), for all τ in W D(k). We say M is conjugate-symplectic if there is a non-degenerate bilinear form on M which satisfies ( B(τ m, sτ s−1 n) = B(m, n) B(n, m) = −B(m, s2 n), for all τ in W D(k). In both cases, the form B gives an isomorphism of representations f : M s → M ∨, whose conjugate-dual ϕ(s2 )

(f ∨ )s : M s −−−−→ ((M s )∨ )s −−−−→ M ∨ satisfies (f ∨ )s = b · f

with b = the sign of B.

We now note: Lemma 3.3. — Given two such non-degenerate forms B and B 0 on M with the same sign and preserved by W D(k), there is an automorphism of M which commutes with W D(k) and such that B 0 (m, n) = B(T m, T n). Proof. — The proof is similar to that of Lemma 3.1. As before, we may reduce to the following two cases:

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

11

(a) M = V ⊗N with N irreducible and conjugate-dual, in which case the centralizer of the action of W D(k) is GL(V ); (b) M = (V ⊗ P ) ⊕ (V ⊗ (P s )∨ ) with P irreducible and P  (P s )∨ , in which case the centralizer of the action of W D(k) is GL(V ) × GL(V ). In case (a), if the conjugate-duality of N has sign bN , then giving a W D(k)invariant non-degenerate bilinear form on M of sign b is equivalent to giving a nondegenerate bilinear form on V of sign b · bN , and all such are conjugate under GL(V ). Similarly, in case (b), giving a W D(k)-invariant nondegenerate bilinear form on M of sign b is equivalent to giving a non-degenerate bilinear form on V , and all such are conjugate under GL(V ) × GL(V ). The isomorphism class of the representation M s is independent of the choice of s in W (k0 ) − W . If s0 = ts is another choice, then the map f

0

: Ms → Ms m 7→ t(m)

is an isomorphism of representations of W D(k). We denote the isomorphism class of 0 M s and M s simply by M σ . If M is conjugate-orthogonal or conjugate-symplectic by the pairing B relative to s, then it is conjugate-orthogonal or conjugate-symplectic by the pairing B 0 (m, f (n)) = B(m, n) relative to s0 . In both cases, M σ is isomorphic to the dual representation M ∨ . If M is conjugate-dual via a pairing B with sign b = ±1, then det(M ) is conjugatedual with sign = (b)dim(M ) . Any conjugate-dual representation of W D(k) of dimension 1 gives a character χ : k × → C× which satisfies χ1+σ = 1. Hence χ is trivial on the subgroup Nk × , which has index 2 in k0× . We denote this 1-dimensional representation by C(χ). Lemma 3.4. — A representation C(χ) is conjugate-orthogonal if and only if χ is trivial on k0× , and conjugate-symplectic if and only if χ is nontrivial on k0× but trivial on Nk × . Proof. — : Since the action of WD(k) on C(χ) factors through the quotient W (k)ab , we may compute with the quotient W (k/k0 ) of W (k0 ). The Weil group W (k/k0 ) is isomorphic to the normalizer of k × in the multiplicative group of the quaternion division algebra over k0 [82, Appendix III, Theorem 2]. It is therefore generated by k × and s, with sα = ασ s for α ∈ k × , and s2 in k0× generating the quotient k0× /Nk × . If χ(s2 ) = +1, then the form B(z, w) = zw on C(χ) is conjugate-orthogonal. If χ(s2 ) = −1, then this form is conjugate-symplectic. We also note:

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

12

W. T. GAN, B. H. GROSS & D. PRASAD

Lemma 3.5. — (i) A representation M of W D(k) is conjugate-dual with sign b if and W D(k ) only if N = IndW D(k)0 M is selfdual with sign b and has maximal isotropic subspace M (which is naturally a W D(k)-submodule of N ). (ii) If Mi is conjugate-dual with sign bi , for i = 1 or 2, then M1 ⊗M2 is conjugate-dual with sign b1 · b2 . Proof. — For (i), suppose that M is conjugate-dual with respect to a form B. As a vector space, N = M ⊕s−1 ·M for s ∈ W D(k0 )rW D(k). We define a non-degenerate bilinear form BN on N by decreeing that M and s−1 · M are isotropic spaces and setting ( BN (m, s−1 · m0 ) = B(m, m0 ), BN (s−1 · m0 , m) = b · BN (m, s−1 · m0 ). It is easy to check that BN is preserved by W D(k0 ). Conversely, if the induced representation N is selfdual with respect to a form BN which has M as an isotropic subspace, then the pairing induced by BN on M ×s−1 ·M is necessarily nondegenerate. Thus we may define a nondegenerate form on M by B(m, m0 ) = BN (m, s−1 · m0 ). It is easy to check that B gives a conjugate-duality on M with the same sign as BN ; this proves (i). The assertion (ii) is also straightforward: if Mi is conjugate-dual with respect to the form Bi of sign bi , then M1 ⊗ M2 is conjugate-dual with respect to the tensor product B1 ⊗ B2 which has sign b1 · b2 . 3.0.1. Remark. — M. Weissman has pointed out that the representation M σ can be more canonically defined (without resorting to the choice of s ∈ W D(k) r W D(k0 )) W D(k ) in the following way. Consider the induced representation IndW D(k)0 M which can be realized on: {f : W D(k0 ) → M : f (τ · s) = τ (f (s)) for all τ ∈ W D(k) and s ∈ W D(k0 )}. Then the representation M σ of W D(k) can be realized on the subspace of such functions which are supported on W D(k0 ) r W D(k). We note: (i) Any W D(k)-equivariant map M → N induces a natural W D(k)-equivariant map M σ → N σ . (ii) There is a natural isomorphism (M σ )∨ → (M ∨ )σ , via the perfect duality on M σ × (M ∨ )σ defined by (f, f ∨ ) 7→ hf (s), f 0 (s)i,

for f ∈ M σ and f ∨ ∈ (M ∨ )σ ,

for any s ∈ W D(k0 ) r W D(k) and where h−, −i denotes the natural pairing on M × M ∨ . The above pairing is clearly independent of the choice of s. (iii) On this model of M σ , there is a canonical isomorphism (M σ )σ −→ M given by F 7→ F [s](s−1 )

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

13

for any s ∈ W D(k0 ) r W D(k). This isomorphism is independent of the choice of s. Thus a conjugate-duality with sign b is a W D(k)-equivariant isomorphism f : Mσ → M∨ whose conjugate-dual (f ∨ )σ : M σ → ((M σ )∨ )σ ∼ = ((M ∨ )σ )σ ∼ = M∨ satisfies (f ∨ )σ = b · f. This treatment allows one to suppress the somewhat mysterious looking identity B(n, m) = b · B(m, s2 n).

4. The centralizer and its group of components The centralizer C(M ) of a representation M of W D(k) is the subgroup of GL(M ) which centralizes the image. Write M M= mi Mi as a direct sum of irreducible representations Mi , with multiplicities mi ≥ 1. Then by Schur’s lemma Y C(M ) ' GL(mi , C). In particular, C(M ) is a connected reductive group. The situation is more interesting for representations M which are either selfdual or conjugate-dual, via a pairing B with sign b = ±1. We define C = C(M, B) as the subgroup of Aut(M, B) ⊂ GL(M ) which centralizes the image of W D(k). Up to isomorphism, the reductive group C depends only on the representation M and can be described more explicitly as follows. If we write M as a direct sum of irreducible representations Mi , with multiplicities mi , and consider their images in M ∨ under the isomorphism M σ → M ∨ provided by B, we find that there are three possibilities: 1. Miσ is isomorphic to Mi∨ , via a pairing Bi of the same sign b as B. 2. Miσ is isomorphic to Mi∨ , via pairing Bi of the opposite sign −b as B. In this case the multiplicity mi is even. 3. Miσ is isomorphic to Mj∨ , with j 6= i. In this case mi = mj . Hence, we have a decomposition M M M M= V i ⊗ Mi + W i ⊗ Ni + Ui ⊗ (Pi + (Piσ )∨ ), where (a) the Mi ’s are selfdual or conjugate-dual of the same sign b, (b) the Ni ’s are selfdual or conjugate-dual of the opposite sign −b;

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

14

W. T. GAN, B. H. GROSS & D. PRASAD

(c) Piσ is not isomorphic to Pi∨ , so that Pi and Pj = (Piσ )∨ are distinct irreducible summands. Moreover, the restriction of the form B to each summand in the above decomposition of M induces a nondegenerate pairing on the multiplicity space Vi , Wi or Ui . The induced pairing on Vi necessarily has sign +1, whereas the pairing on Wi necessarily has sign −1. On the other hand, the induced pairing on Ui need not have a sign. We can now determine the centralizer C. As in the proofs of Lemmas 3.1 and 3.3, giving an element T of C is equivalent to giving elements Ti in GL(Vi ), GL(Wi ) or GL(Ui ) × GL(Ui ), such that Ti preserves the induced nondegenerate pairing on Vi , Wi or Ui . Thus, we conclude that (cf. [24, § 6-7], [57] and [58]): Y Y Y C' O(Vi ) × Sp(Wi ) × GL(Ui ). In particular, the component group of C is A = π0 (C) ' (Z/2)k , where k is the number of irreducible summands Mi of the same type as M , or equivalently the number of Vi ’s in the above decomposition. For each such Mi , let ai be a simple reflection in the orthogonal group O(mi ). The images of the elements ai in A give a basis over Z/2Z. For any semisimple element a in C, we define M a = {m ∈ M : am = −m} to be the −1 eigenspace for a on M . This is a representation of W D(k), and the restricted pairing B : M a × M a → C is non-degenerate, of the same type as M . For the simple reflections ai in C, M ai = Mi are the irreducible summands of the same type as M . We can use these representations to define characters χ : A → h±1i. The basic idea is to define signed invariants d(M ) = ±1 of representations M of W D(k), which are either selfdual or conjugate-dual. Proposition 4.1. — Let d(M ) be an invariant of selfdual or conjugate-dual representations, taking values in ±1. Assume that 1. d(M + M 0 ) = d(M ) · d(M 0 ) 2. the value d(M a ) depends only on the image of a in the quotient group A = 0 CM /CM . Then the function χ(a) = d(M a ) defines a character of A. Indeed, the different classes in A are all represented by commuting involutions in C, and for two commuting involutions a and b we have the formula: M ab + 2(M a ∩ M b ) = M a + M b

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

15

as representations of W D(k). Hence χ(ab) = χ(a) · χ(b). The simplest example of such an invariant, which applies in both the conjugate-dual and the selfdual cases, is d(M ) = (−1)dim M . a 0 To see that dim M (mod 2) depends only on the coset of a (mod CM ), we recall that M M M M= Vi ⊗ Mi + Wi ⊗ Ni + Ui ⊗ (Pi + (Piσ )∨ ) Q and let a = i ai be a semisimple element of the product Y Y Y CM = O(Vi ) × Sp(Wi ) × GL(Ui ). i

i

i

Then −1 occurs with even multiplicity as an eigenvalue of ai ∈ Sp(Wi ). On the other hand, for ai ∈ O(Vi ), one has det ai = (−1)multiplicity of −1 as an eigenvalue of ai . Hence all the summands of M a have even dimension, except for the terms Viai ⊗ Mi which have odd dimension precisely when det(ai ) = −1 and dim Mi ≡ 1 (mod 2). 0 Thus it follows that the parity of dim M a depends only on the coset of a (mod CM ). In particular, one obtains a character of A: a

η(a) = (−1)dim M . Now assume M is selfdual. The character η is trivial when M is symplectic, as dim M a is even for all a in Sp(M ) = G(M, B). In the orthogonal case, dim M a is even precisely for elements a in the centralizer which lie in the subgroup SO(M, B) of index 2 in O(M, B). We denote this subgroup by C + . An element c of k × /k ×2 gives a character ηc (a) = (det M a )(c) of A. Indeed, the quadratic character det M a depends only on the coset of a (mod C 0 ). Since ηcd = ηc ηd , we get a pairing (c, a) : k × /k ×2 × A → h±1i which is trivial in the symplectic case. To construct other characters of A, we need more sophisticated signed invariants d(M ) of selfdual or conjugate-dual representations. We will obtain these from local root numbers, after recalling that theory in the next section. 5. Local root numbers Let M be a representation of the Weil-Deligne group W D(k) of a local field k, and let ψ be a nontrivial additive character of k. In this section, we define the local root number (M, ψ), following the articles of Tate [69] and Deligne [13]. We then study the properties of these constants for selfdual and conjugate-dual representations, and give explicit formulae in the orthogonal and conjugate-orthogonal cases. The local

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

16

W. T. GAN, B. H. GROSS & D. PRASAD

root numbers are more mysterious in the symplectic and conjugate-symplectic cases. Indeed, they form the basis of our conjectures on the restriction of representations of classical groups over local fields. Let dx be the unique Haar measure on k which is selfdual for Fourier transform with respect to ψ. For a representation M of the Weil group W (k), we define (M, ψ) = (M, ψ, dx, 1/2) in

C× ,

in the notation of [13, § 4-5]. This is the local constant L (M, ψ) in [69, 3.6.1]. In the non-archimedean case, if M is a representation of W D(k) = W (k) × SL2 (C), we may write X M= Mn ⊗ Symn n≥0

with each Mn a representation of the Weil group. We define (cf. [26, § 2]): Y (M, ψ) = (Mn , ψ)n+1 · det(−F |MnI )n . n≥0

This constant depends only on the isomorphism class of M . The following formulae involving (M, ψ) are well-known [69, 3.6], for representations M of the Weil group W (k). For a in k × , let ψa be the nontrivial additive character ψa (x) = ψ(ax). Then (M, ψa ) = det M (a) · (M, ψ), (M, ψ) · (M ∨ , ψ −1 ) = 1. Since ψ −1 = ψ−1 , we conclude that (M, ψ) · (M ∨ , ψ) = det M (−1). P For representations M = Mn ⊗ Symn of W D(k) in the non-archimedean case, we have X M∨ = Mn∨ ⊗ Symn , Y det(M ) = det(Mn )n+1 . n≥0

This allows us to extend the above formulas to the local root numbers (M, ψ) of representations of W D(k). Now let σ be an involution of k, and define ψ σ (x) = ψ(xσ ). Then (M σ , ψ σ ) = (M, ψ). (Indeed, this is true for any continuous isomorphism σ : k → k 0 . For M of dimension 1, this follows from Tate’s integral formula [69] for (M, ψ). It then follows from general M from the inductivity of -factors.) If we assume further that ψ σ = ψ −1 , then (M, ψ) · ((M ∨ )σ , ψ)

ASTÉRISQUE 346

= =

(M, ψ) · (M ∨ , ψ σ ) (M, ψ) · (M ∨ , ψ −1 )

=

1.

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

17

When we apply these formulas to selfdual and conjugate-dual representations, we obtain the following. Proposition 5.1. — 1. Assume that M is a selfdual representation of W D(k) with det(M ) = 1. Then (M ) = (M, ψ) is independent of the choice of ψ and satisfies (M )2 = 1. Furthermore, if M is of the form M = N + N ∨ , then (M ) = det N (−1). 2. Assume that M is a conjugate-dual representation of W D(k) and that the additive character ψ of k satisfies ψ σ = ψ −1 . Then (M, ψ)2 = 1. Furthermore, if M is of the form M = N + σ N ∨ , then (M, ψ) = 1. Since we are assuming that the characteristic of k is not equal to 2, the characters ψ of k which satisfy ψ σ = ψ −1 are precisely those characters which are trivial on k0 , the fixed field of σ. These characters form a principal homogeneous space for the group k0× , and the value (M, ψ) depends only on the Nk × -orbit of ψ. Indeed det M is conjugate-dual and hence trivial on Nk × ⊂ k0× . If det M is conjugate-orthogonal, the restriction of det M to k0× is trivial, and hence the value (M ) = (M, ψ) is independent of the choice of ψ. Following Deligne [14], we can say more when the selfduality or conjugate-duality of M is given by a pairing B with sign b = +1. Recall the spin covering of the special orthogonal group, which gives an exact sequence: 1 → Z/2 → Spin(M ) → SO(M ) → 1. Proposition 5.2. — 1. Assume that M is an orthogonal representation and that det(M ) = 1. Then the root number (M ) = (M, ψ) is independent of the choice of ψ and satisfies (M )2 = 1. Furthermore (M ) = +1 if and only if the representation ϕ : W D(k) → SO(M ) lifts to a homomorphism ϕ : W D(k) → Spin(M ). 2. Assume that M is a conjugate-orthogonal representation and that ψ σ = ψ −1 . Then the root number (M ) = (M, ψ) is independent of the choice of ψ and satisfies (M ) = +1. Proof. — The orthogonal case was proved by Deligne [14]; we note that in our case the characteristic of k is not equal to 2. We will deduce the second result for conjugateorthogonal representations of W (k) from Deligne’s formula, combined with the work of Frohlich and Queyrut [16]. The extension of the second result to conjugate-orthogonal representations of the Weil-Deligne group W D(k) is then an amusing exercise, which we leave to the reader (cf. [14, § 5]). If M is conjugate-orthogonal and dim(M ) = 1, we have seen that M corresponds to a complex character χ of the group k × /k0× . By [16, Thm 3], we have the formula (χ, ψ0 (Tr)) = χ(e)

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

18

W. T. GAN, B. H. GROSS & D. PRASAD

where ψ0 is any nontrivial additive character of k0 and e is any nonzero element of k with Tr(e) = 0 in k0 . The element e is well-defined up to multiplication by k0× , and e2 is an element of k0× . If we define the additive character ψ of k by ψ(x) = ψ0 (Tr(ex)) σ

then ψ = ψ

−1

, and

(M ) = (χ, ψ) = χ(e)2 = +1. This establishes the formula when M has dimension 1. Since the desired formula is additive in the representation M of W (k), and is true when dim(M ) = 1, we are reduced to the case of conjugate-orthogonal representations M of even dimension. Then N = Ind(M ) is an orthogonal representation of determinant 1. Let ψ0 be a nontrivial additive character of k0 ; by the inductivity of local epsilon factors in dimension zero [13]: (N, ψ0 )/(P, ψ0 )dim(M ) = (M, ψ0 (Tr))/(C, ψ0 (Tr))dim(M ) with C the trivial representation and P = Ind(C) the corresponding induced representation, which is orthogonal of dimension 2 and determinant ω. Since (C, ψ0 (Tr)) = 1 and (P, ψ0 )2 = ω(−1), we obtain the formula (N ) = (N, ψ0 ) = ω(−1)dim(M )/2 · (M, ψ0 (Tr)). On the other hand, we have (M ) = (M, ψ) = det(M )(e) · (M, ψ0 (Tr)), where e is a nonzero element of k with Tr(e) = 0 in k0 . Hence, to show that (M ) = +1, we are reduced to proving the formula (N ) = det(M )(e) · ω(−1)dim(M )/2 for the root number of the orthogonal induced representation N . To do this, we combine Deligne’s formula for the orthogonal root number with the following lemma. Lemma 5.3. — Let M be a conjugate-orthogonal representation of W (k) of even dimension. Then N = Ind(M ) is an orthogonal representation of W (k0 ) of determinant 1. The homomorphism ϕ : W (k0 ) → SO(N ) lifts to a homomorphism ϕ : W (k0 ) → Spin(N ) if and only if det M (e) · ω(−1)dim(M )/2 = +1. Proof. — Let T be the maximal torus in SO(N ) which consists of the rotations zi in n = dim(M ) orthogonal planes. The restriction of the spin covering Spin(N ) → SO(N ) to the torus T is the two-fold covering obtained by pulling back the spin Q covering z → z 2 of C× under the map F (z1 , · · · , zn ) = zi . The image of the map ϕ : W → SO(N ) lies in the normalizer of the Levi subgroup GL(M ) which fixes the decomposition N = M + M ∨ into maximal isotropic dual subspaces. There is an involution j of N which switches the subspaces M and M ∨ . Since det(j) = (−1)n and n = dim(M ) is even, this involution lies in SO(N ). The normalizer of the Levi is the semi-direct product GL(M ) · hji. We denote the resulting homomorphism of W to GL(M ) · hji also by ϕ.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

19

Since j has n eigenvalues which are +1, and n eigenvalues which are −1, if we view this involution as a product of rotations (z1 , · · · , zn ) in orthogonal planes, we get n/2 values zi = −1 and n/2 values zi = +1. Hence the involution j lifts to an element of order 2 in Spin(N ) if and only if n = dim(M ) is divisible by 4. Note that the quadratic extension k of k0 can be embedded in a cyclic quartic extension of k0 if and only if the character ω of k0× is a square, or equivalently, if and only if ω(−1) = 1. We therefore conclude that the natural homomorphism W → Z/2Z = hji (given by the quadratic extension k/k0 ) lifts to the restriction of the spin cover to hji = Z/2Z if and only if (5.4)

ω(−1)dim(M )/2 = +1.

We now consider the homomorphism ϕ

det

φ : W → GL(M ) · hji → C× · hji whose projection to the quotient hji is the quadratic character ω of Gal(k/k0 ). The resulting homomorphism φ : W → C× · hji is given by its restriction to W (k), which is nothing but the character χ = det(M ), a character of k × /k0× by the local class field theory. Hence the homomorphism φ : W → C× · hji lifts to a homomorphism from W to C× · hji: C× · hji : z2

φ:W

 / C× · hji

if and only if the character χ of k × /k0× has a square-root. Clearly, the character χ of k × /k0× is a square if and only if (5.5)

χ(e) = +1,

where e is a nonzero element of k with trace zero to k0 . Indeed, e generates the 2-torsion subgroup of the one dimensional torus k × /k0× . Since the subgroup SL(M ) is simply-connected, it always lifts to Spin(N ). Hence the restriction of the spin covering of SO(N ) to GL(M ) is obtained by taking the square root of the determinant of M , via the formula for the covering of T given above. It is easy to see that the 2-fold covering of W afforded by ϕ : W → GL(M ) · hji ⊂ SO(N ) is the sum of two coverings, one of which is the 2-fold cover of hji pulled back to give a 2-fold cover of GL(M ) · hji, and the other of which is the 2-fold cover z2

C× → C× pulled back to GL(M ) · hji via the determinant map from GL(M ) to C× . From our calculations above, these two 2-fold covers of W are respectively trivial if and only if we have the conditions as in (5.4) and (5.5).

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

20

W. T. GAN, B. H. GROSS & D. PRASAD

We now observe that H 2 (W, Z/2Z) classifying the 2-fold coverings of W is an abelian group under fiber product which, by local class field theory, is nothing but Z/2Z. Therefore the sum of two elements in H 2 (W, Z/2Z) is zero if either both of them are trivial, or both of them are nontrivial. Hence by (5.4) and (5.5), the parameter φ : W → GL(M ) · hji → SO(N ) lifts to Spin(N ) if and only if χ(e) · ω(−1)dim(M )/2 = +1. Together with the extension to representations of W D(k), this completes the proof of Proposition 5.2.

6. Characters of component groups In this section, we will use the results of the previous section on local root numbers, together with Proposition 4.1, to construct characters of the group A of components of the centralizer C of (M, B). First assume M and N are conjugate selfdual representations, with signs b(M ) and b(N ). Fix ψ with ψ σ = ψ −1 , and for semisimple a in CM ⊂ G(M, B), define χN (a) = (M a ⊗ N, ψ). Theorem 6.1. — 1. The value χN (a) depends only on the image of a in AM , and defines a character χN : AM → h±1i 2. If b(M ) · b(N ) = +1, then χN = 1 on AM . 3. If b(M ) · b(N ) = −1, let ψ 0 (x) = ψ(tx) with t the nontrivial class in k0× /Nk × , and define χ0N (a) = (M a ⊗ N, ψ 0 ). Then χ0N = χN · η dim(N ) ∈ Hom(AM , ±1), a

where the character η of AM is defined by η(a) = (−1)dim M . Proof. — (1) Write M M M M= Vi ⊗ Mi + Wi ⊗ Ni + Ui ⊗ (Pi + (Piσ )∨ ) as in §4., so that CM =

Y i

O(Vi ) ×

Y i

Sp(Wi ) ×

Y

GL(Ui ).

i

It suffices to check (1) for semisimple elements a which are nontrivial in exactly one of the factors in the above product expression for CM . Suppose, for example, that a = ai × 1 with ai ∈ O(Vi ). Then M a ⊗ N = dim Viai · (Mi ⊗ N ).

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

21

The parity of dim Viai depends only on the image of ai in O(Vi )/SO(Vi ), or equivalently only on the image of a in the component group AM . Since (Mi ⊗ N, ψ) = ±1, we see that ai

χ(a) = (M a ⊗ N, ψ) = (Mi ⊗ N, ψ)dim Vi

depends only on the image of a in AM . The other cases are similarly treated: when a = ai × 1 with ai ∈ Sp(Wi ) or ai ∈ GL(Ui ), one finds that (M a ⊗ N, ψ) = +1. For the details, see [24, § 10]. (2) When b(M ) · b(N ) = +1, the representations M a ⊗ N are all conjugate-orthogonal by Lemma 3.5, so χN = 1. (3) The final statement follows from the formula χ0N (a) = χN (a) · det(M a ⊗ N )(t) and the calculation of the sign of the conjugate-dual representation which is the determinant of the tensor product. We use this theorem to define the quadratic character χN (aM ) · χM (aN ) on elements (aM , aN ) in the component group AM × AN . Here M and N are two conjugate-dual representations, although by part 2 of Theorem 6.1 the character χN × χM can only be nontrivial when b(M ) · b(N ) = −1. The case when M and N are selfdual with signs b(M ) and b(N ) is more complicated. First, with ψ a nontrivial additive character of k, the function χN (a) = (M a ⊗ N, ψ) on CM need not take values in ±1. Indeed χN (a)2 = det(M a ⊗ N )(−1) = ±1. Even when det(M a ⊗ N ) = 1 for all a in CM , the value χN (a) = ±1 may not be 0 constant on the cosets of CM . For example, when M = P + P ∨ with P irreducible ∨ and not isomorphic to P and N is the trivial representation C of dimension 1, we have CM = GL(1, C). But χN (−1) = (M ⊗ N, ψ) = (M, ψ) = det P (−1), which need not be equal to χN (1) = 1. We will therefore only consider selfdual representations M and N of even dimen+ sion, and semisimple elements a in the subgroup CM of CM , where det(a|M ) = +1. a Then dim(M ) is also even, and det(M a ⊗ N ) = det(M a )dim(N ) · det(N )dim(M

a

)

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

22

W. T. GAN, B. H. GROSS & D. PRASAD

is clearly trivial. In particular, (M a ⊗ N, ψ) = (M a ⊗ N ) is independent of the choice of additive character ψ and satisfies (M a ⊗ N )2 = +1. We correct this sign + by another square root of det(M a ⊗ N )(−1), and define (for a ∈ CM ) χN (a) = (M a ⊗ N ) · det(M a )(−1)dim(N )/2 · det(N )(−1)dim(M

a

)/2

.

+ + 0 0 Now let A+ M be the image of CM in AM = CM /CM . Note that CM ⊂ CM ⊂ CM , so + the component group AM has index either 1 or 2 in AM . Then we have:

Theorem 6.2. — Assume that M and N are even dimensional selfdual representations of W D(k). 1. The value χN (a) depends only on the image of a in A+ M , and defines a character χ N : A+ M → h±1i. 2. If b(M ) · b(N ) = +1, then χN (a) = (det M a , det N ), where (−, −) is the Hilbert symbol. Proof. — (1) This follows from the method of [24, Prop. 10.5], analogous to the proof of Theorem 6.1(1). (2) When M and N are both symplectic, the tensor product representation M a ⊗N of the simply-connected group Sp(M a ) × Sp(N ) lifts to Spin(M a ⊗ N ) and (M a ⊗ N ) = +1. This proves (2) since det M a = det N = 1. When M and N are both orthogonal of even dimension, so are M a and N . Hence a M ⊗ N is orthogonal of dimension divisible by 4 and determinant 1. In this case, we have (M a ⊗ N ) = w2 (M a ⊗ N ) by Deligne [14], where w2 refers to the second Stiefel-Whitney class [48], which is valued in H 2 (k, Z/2Z) = {±1}. On the other hand, for two representations V and W of W D(k), w2 (V ⊗ W ) is given by [48, Problem 7-C, Pg. 87-88] Ç å dim W 2 w2 (V ⊗ W ) = w2 (V ) · dim W + dim(V ) · w2 (W ) + w1 (V ) · 2 Ç å dim V + · w1 (W )2 + w1 (V ) · w1 (W ) · (dim V · dim W + 1) 2 as elements of H 2 (k, Z/2Z) = {±1}. Here, w1 refers to the first Stiefel-Whitney class, which is valued in H 1 (k, Z/2Z) = k × /k ×2 , and the various operations refer to addition and cup product in the cohomology ring H ∗ (k, Z/2Z). In particular, if V and W are both even-dimensional, we have w2 (V ⊗ W ) =

dim W dim V · w1 (W )2 + · w1 (V )2 + w1 (V ) · w1 (W ) ∈ {±1}. 2 2

For the even dimensional orthogonal representations M a and N of interest, we have: w1 (N ) = det N

ASTÉRISQUE 346

and w1 (M a ) = det M a ,

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

23

and the cup product pairing H 1 (k, Z/2Z) × H 1 (k, Z/2Z) → H 2 (k, Z/2Z) is given by the Hilbert symbol. Hence, we have a

1

1

(M a ⊗ N ) = (det N, det N ) 2 dim M · (det M a , det M a ) 2 dim N · (det N, det M a ) 1

a

1

= (det N )(−1) 2 dim M · (det M a )(−1) 2 dim N · (det N, det M a ), as claimed. We use this theorem to define the quadratic character χN (aM ) · χM (aN ) + on elements (aM , aN ) in the component group A+ M × AN , for two selfdual representations M and N of even dimension. As in the conjugate-dual case, it follows from Theorem 6.2(2) that the character χN ×χM is only interesting when b(M )·b(N ) = −1. When the representations M and N are both symplectic, χ = 1. When M and N are both orthogonal, the character χ is given by a product of Hilbert symbols

χN (aM ) · χM (aN ) = (det M aM , det N ) · (det M, det N aN ).

7. L-groups of classical groups Having defined representations, selfdual representations, and conjugate-dual representations of the Weil-Deligne group W D(k) of k, our next goal is to relate these to the Langlands parameters of classical groups. Before doing that, we recall the L-group attached to each of the classical groups, with particular attention to the L-groups of unitary groups. If G is a connected reductive group over k0 , the L-group of G is a semi-direct product L “ o Gal(K/k0 ) G=G “ is the complex dual group and K is a splitting field for the quasi-split inner where G “ via pinned automorphisms. (Alternatively, form of G, with Gal(K/k0 ) acting on G “ through its quotient Gal(K/k0 )). one could use W (k0 ), acting on G Recall that in this paper, our classical group G = G(V ) comes equipped with an underlying space V , i.e. with a standard representation. We shall see that this extra “ with a standard representation. data equips the L-group L G or the dual group G “ For G = GL(V ), we have G = GL(M ) with dim M = dim V . If {e1 , · · · , en } is the basis of the character group of a maximal torus T ⊂ G given by the weights of V , ∨ then the weights of the dual torus Tb on M are the dual basis {e∨ 1 , · · · , en }. Now assume that G ⊂ GL(V /k) is a connected classical group, defined by a σ“ as sesquilinear form h, i : V × V → k of sign . The group G and its dual group G, well as the splitting field K of its quasi-split inner form, are given by the following table:

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

24

W. T. GAN, B. H. GROSS & D. PRASAD

(k, )

G

“ G

K

k = k0

SO(V ),

Sp2n (C)

k0

Sp2n (C)

=1

dim V = 2n + 1

k = k0

SO(V ),

SO2n (C)

p k0 ( disc(V ))

O2n (C) (disc(V ) ∈ / k ×2 )

=1

dim V = 2n

k = k0

Sp(V ),

 = −1

dim V = 2n

k 6= k0

U(V ),

 = ±1

dim V = n

L

G

SO2n (C) (disc(V ) ∈ k ×2 ) SO2n+1 (C)

k0

SO2n+1 (C)

GLn (C)

k

GLn (C) o Gal(k/k0 )

“ is a We make a few remarks on the table. Firstly, when k = k0 , the dual group G special orthogonal or symplectic group and thus has a unique standard representation, as indicated in the table. However, when V is an even dimensional quadratic space and disc(V ) ∈ / k ×2 , this standard representation has two extensions to L G. In the above table, we have identified L G with O2n (C) by regarding the nontrivial element in Gal(K/k0 ) as a simple reflection in O2n (C): this picks out one of these two extensions. Thus, when k = k0 , the L-group L G comes equipped with a standard self-dual representation M in each case. Secondly, when k 6= k0 , we fix a standard representation M for the identity com“ as follows. Extending scalars from k0 to k, one has ponent L G0 = G U(V ) ×k0 k ,→ Resk/k0 (GL(V )) ×k0 k ∼ = GL(V ) × GL(V σ ). Via the first projection, one has a k-isomorphism U(V ) ×k0 k ∼ = GL(V ). In the above, we have already fixed the L-group of GL(V ) with a standard representation M . Thus, we see that L G = GL(M ) o Gal(k/k0 ) and L G0 = GL(M ) comes equipped with a standard representation. In addition, we note that the L-group GL(M ) o Gal(k/k0 ) of U(V ) is isomorphic as a complex Lie group to the L-group of the anisotropic real group U(n), associated to a definite hermitian space of the same dimension as V over C. We will now use this fact to study the parity of selfdual complex representations of the L-group of U(V ). Let us make some general observations on the representation theory of the L-groups of anisotropic groups over R. In this case, we have L

“ o Gal(C/R), G=G

where the Galois group acts by a pinned involution (possibly trivial), which maps to “ and takes any representation V to its dual V ∨ . the opposition involution in Out(G) “ gives a principal SL2 in G, “ which is fixed by the action of Gal(C/R), The pinning of G so that we have “ o Gal(C/R). δ : SL2 × Gal(C/R) → G

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

25

“ Gal(C/R) be the image of −I in SL2 . Then 2 = 1 and  acts as a scalar Let  in Z(G) on every irreducible representation of L G. The following result is due to Deligne. Proposition 7.1. — Let G be an anisotropic group over R. Then every complex representation N of L G is selfdual. If N is irreducible, the L G-invariant pairing h, iN : N × N → C is unique up to scaling, and is (|N )-symmetric. Proof. — The proof is similar to Bourbaki [10, Ch. VIII, §7, Prop. 12 and Ex. 6], “ There one restricts the which treats the irreducible, selfdual representations M of G. pairing on M to the highest weight summand for the principal SL2 , which occurs with multiplicity one. In this case, we restrict the pairing on N to the subgroup SL2 × Gal(C/R), which again has an irreducible summand which occurs with multiplicity 1. The sign (|N ) is the sign of −I on this summand, which determines the sign of the pairing. “ which is selfdual, then M extends in two If M is an irreducible representation of G L ∨ ways to G. If M is not isomorphic to M , then the induced representation N = Ind(M ) “ it decomposes as a direct sum of L G is irreducible. If we restrict N to G, M + α · M ' M + M∨ where α generates Gal(C/R). Since M is not isomorphic to M ∨ , the subspaces M and α · M of N are isotropic for the pairing h, iN , which gives a non-degenerate pairing h, iM : M × M → C defined by (7.2)

hm, m0 iM = hm, αm0 iN .

“ we have This is a conjugate-duality on M : for g in G, ( hgm, αgα−1 m0 iM = hm, m0 iM hm0 , miM = (|M ) · hm, m0 iM We now specialize these arguments to the case of unitary groups. Since the representation of the principal SL2 → GL(M ) on M is irreducible and isomorphic to Symn−1 , we have (|M ) = (−1)n−1 . Hence the self-duality on N = IndM and the conjugate-duality on M are (−1)n−1 symmetric. In particular, we have Proposition 7.3. — If G = U(V ), with dimk V = n, then ( Sp(N ) = Sp2n (C), if n is even; L G ,→ O(N ) = O2n (C), if n is odd. In each case, L G is identified with the normalizer of a Levi subgroup of a Siegel parabolic subgroup in Sp2n (C) or O2n (C).

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

26

W. T. GAN, B. H. GROSS & D. PRASAD

Finally, it is instructive to describe the L-groups of the classical groups from the point of view of invariant theory. As we explain above, the L-groups of symplectic and special orthogonal groups G(V ) are themselves classical groups over C and have natural realizations as subgroups of GL(M ) for complex vector spaces M of appropriate dimensions. These subgroups can be described as follows. One has a decomposition M ⊗M ∼ = Sym2 M

2 M^

M

of GL(M )-modules. The action of GL(M ) on Sym2 M or ∧2 M has a unique open orbit consisting of nondegenerate symmetric or skew-symmetric forms on M ∨ . Then we note: (i) The stabilizer of a nondegenerate vector B in Sym2 M (resp. ∧2 M ) is the orthogonal group O(M, B) (resp. the symplectic group Sp(M, B)); these groups exhaust the L-groups of symplectic and orthogonal groups. (ii) The action of this stabilizer on the other representation ∧2 M (resp. Sym2 M ) is its adjoint representation. (iii) The two representations Sym2 M and ∧2 M are also useful for characterizing the selfdual representations of W D(k) introduced in §3: a representation M of W D(k) is orthogonal (resp. symplectic) if and only if W D(k) fixes a nondegenerate vector in Sym2 M (resp. ∧2 M ). These rather obvious remarks have analogs for the unitary group U(V ), which we now describe. Suppose that the L-group of U(V ) is GL(M ) o Gal(k/k0 ). Consider the semi-direct product H = (GL(M ) × GL(M )) o Z/2Z where Z/2Z acts by permuting the two factors of GL(M ); this is the L-group of Resk/k0 (GL(V /k)) with dimk V = dim M . The irreducible representation M  M of H 0 = GL(M ) × GL(M ) is invariant under Z/2Z and thus has two extensions to H. In one such extension, the group Z/2Z = S2 simply acts by permuting the two copies of M ; the other extension is then given by twisting by the nontrivial character of H/H 0 . In honor of Asai, we denote these two extensions by As+ (M ) and As− (M ) respectively. They can be distinguished by Trace(c|As+ (M )) = dim M

and

Trace(c|As− (M )) = − dim M,

where c is the nontrivial element in Z/2Z. One has M + IndH As− (M ). H 0 (M  M ) = As (M ) The action of H 0 on As± (M ) has an open dense orbit, consisting of isomorphisms M → M and whose elements we call nondegenerate. Now we have ∨

Proposition 7.4. — If dim M = n, then the stabilizer in H of a nondegenerate vector in n−1 As(−1) (M ) is isomorphic as a complex Lie group to the L-group of U(V ). Moreover, n the action of this stabilizer on the other representation As(−1) (M ) is the adjoint representation of L U (V ).

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

27

The representations As± (M ) are also useful for characterizing conjugate-dual representations of W D(k), which were discussed in §3. Indeed, given a representation ϕ : W D(k) → GL(M ), one obtains a map ϕ˜ : W D(k0 ) → H by setting ϕ(τ ˜ ) = (ϕ(τ ), ϕ(sτ s−1 )) ∈ GL(M ) × GL(M ), for τ ∈ W D(k), and ϕ(s) ˜ = (1, ϕ(s2 )) · c ∈ H r H 0 . The choice of s is unimportant, since the maps ϕ’s ˜ thus obtained for different choices of s are naturally conjugate under H 0 . Through this map, W D(k0 ) acts on As± (M ). In fact, the representation As+ (M ) of W D(k0 ) is obtained from M by the process of multiplicative induction [56] or twisted tensor product; it is an extension of the representation M ⊗ M σ of W D(k) to W D(k0 ), and As− (M ) is the twist of As+ (M ) by the quadratic character ωk/k0 associated to the quadratic extension k/k0 . Now we have: Proposition 7.5. — If M is a representation of W D(k), then (i) M is conjugate-orthogonal if and only if W D(k0 ) fixes a nondegenerate vector in As+ (M ). When M is irreducible, this is equivalent to As+ (M )W D(k0 ) 6= 0. (ii) M is conjugate-symplectic if and only if W D(k0 ) fixes a nondegenerate vector in As− (M ). When M is irreducible, this is equivalent to As− (M )W D(k0 ) 6= 0.

8. Langlands parameters for classical groups In this section, we discuss the Langlands parameters of classical groups. In particular, we show that these Langlands parameters can be understood in terms of selfdual or conjugate-dual representations M of W D(k). If G is a connected, reductive group over k0 , a Langlands parameter is a homomorphism “ o Gal(K/k0 ). ϕ : W D(k0 ) → L G = G This homomorphism is required to be continuous on W D(k0 ) = W (k0 ) when k0 = R or C. In the non-archimedean case, W D(k0 ) = W (k0 ) × SL2 (C) and ϕ is required to be trivial on an open subgroup of the inertia group in W (k0 ), the image of Frobenius is required to be semi-simple and the restriction of ϕ to SL2 (C) is required to be algebraic. In all cases, the projection onto Gal(K/k0 ) is the natural map W (k0 )/W (K) → Gal(K/k0 ). Finally, two Langlands parameters are considered “ equivalent if they are conjugate by an element in G. Associated to any Langlands parameter is the reductive group “ Cϕ ⊂ G

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

28

W. T. GAN, B. H. GROSS & D. PRASAD

which centralizes the image, and its component group Aϕ = Cϕ /Cϕ0 . The isomorphism class of both Cϕ and Aϕ are determined by the equivalence class of the parameter ϕ. “ = GL(M ) with dim M = dim V . If he1 , · · · , en i For G = GL(V /k0 ), we have G is the basis of the character group of a maximal torus T ⊂ G given by the weights ∨ of V , then the weights of the dual torus Tb on M are the dual basis he∨ 1 , · · · , en i. The Langlands parameters for G are simply equivalence classes of representations of W D(k0 ) on M . Now assume that G ⊂ GL(V /k) is a connected classical group, defined by a σsesquilinear form h−, −i : V × V → k of sign . Recall that the L-group of G or its “ comes equipped with a standard representation M . We will identity component G see that for each classical group G, a Langlands parameter ϕ for G corresponds to a natural complex representation W D(k) → GL(M ) with some additional structure, as given in the following theorem. Theorem 8.1. — (i) A Langlands parameter ϕ of the connected classical group G ⊂ GL(V /k) determines a selfdual or conjugate-dual representation M of W D(k), with the following structure: G Sp(V ) SO(V ) SO(V ) U(V ) U(V )

dim(V ) 2n 2n + 1 2n 2n + 1 2n

M Orthogonal Symplectic Orthogonal Conjugate-orthogonal Conjugate-symplectic

dim M 2n + 1 2n 2n 2n + 1 2n

det M = 1 det M = disc V

(ii) The isomorphism class of the representation M determines the equivalence class of the parameter ϕ, except in the case when M is orthogonal and every irreducible orthogonal summand Mi of M has even dimension. In the exceptional case, M and V have even dimension and there are two equivalence classes {ϕ, ϕ0 } of parameters for SO(V ) which give rise to the same orthogonal representation M . “ which centralizes the image of ϕ is (iii) In the unitary cases, the group Cϕ ⊂ G isomorphic to the group C of elements a in Aut(M, B) which centralize the image “ is W D(k) → GL(M ). In the orthogonal and symplectic cases, the group Cϕ ⊂ G + isomorphic to the subgroup C of C, consisting of those elements of Aut(M, B) which satisfy det(a|M ) = 1. Proof. — This is well-known if G is an orthogonal or symplectic group. Indeed, the L-group L G is essentially the automorphism group of a nondegenerate symmetric or skew-symmetric bilinear form B on a complex vector space M of appropriate dimension. So the theorem amounts to the assertion that if two homomorphisms

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

29

W D(k) → Aut(M, B) ⊂ GL(M ) are conjugate in GL(M ), then they are conjugate in Aut(M, B). This is the content of Lemma 3.1. Moreover, the description of the component group Cϕ follows directly from the results of §4. Henceforth we shall focus on the unitary case. For G = U(V ), a parameter is a homomorphism ϕ : W D(k0 ) → L G = GL(M ) o Gal(k/k0 ) with dim M = dim V = n. The restriction of ϕ to W D(k) gives the desired representation M . We must next show that M is conjugate-dual with sign (−1)n−1 . If s ∈ W (k) generates the quotient W D(k0 )/W D(k), then ϕ(s) = (A, α) = (A, 1) · (1, α) in L G with A in GL(M ). In the previous section (cf. equation (7.2)), we have seen that the standard representation M of GL(M ) has a conjugate-duality h−, −iM of sign (−1)n−1 with respect to the nontrivial element α ∈ Gal(k/k0 ) ⊂ L G. We define the bilinear form B(m, m0 ) = Bs (m, m0 ) = hm, A−1 m0 iM , Then the form B is non-degenerate on M and satisfies B(τ m, sτ s−1 m0 ) = B(m, m0 ) for all τ in W D(k), and B(m0 , m) = (−1)n−1 · B(m, s2 m0 ). Hence, as a representation of W D(k), M is conjugate-dual with sign = (−1)n−1 . It is clear that the conjugation of a parameter ϕ by an element of GL(M ) gives an isomorphism of the associated conjugate-dual representations. Hence we are reduced to showing that every conjugate-dual representation M of sign = (−1)n−1 extends to a Langlands parameter ϕ of W D(k0 ), and that the isomorphism class of M determines the equivalence class of ϕ. Suppose then that M is a conjugate-dual representation of W D(k) of sign = (−1)n−1 with n = dim M . To obtain an extension, consider the induced representation N = Ind(M ) of W D(k0 ). By Lemma 3.5(i), N is selfdual of sign = (−1)n−1 . Moreover, the proof of Lemma 3.5 shows that the image of W D(k0 ) in Sp(4d) or O(4d + 2) (depending on whether n = 2d or n = 2d + 1) is contained in the normalizer of a Levi subgroup in a Siegel parabolic subgroup. By Proposition 7.3, this normalizer is isomorphic to the L-group of U(V ): it splits as a semi-direct product GL(M ) o hαi, with det(α|N ) = (−1)n . Thus, we have produced an L-parameter for U(V ) whose restriction to W D(k) is the given M . Finally we need to show that the extension obtained above is unique, up to conju“ If ϕ and ϕ0 are two parameters extending ρ : W D(k) → GL(M ), we must gacy by G. show that the elements ϕ(s) = (A, α) ϕ0 (s)

=

(A0 , α) of

L

G

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

30

W. T. GAN, B. H. GROSS & D. PRASAD

are conjugate by an element of GL(M ) centralizing the image of ρ. The bilinear forms B(m, m0 ) 0

0

B (m, m )

= hm, A−1 m0 iM = hm, (A0 )−1 m0 iM

give two conjugate-dualities of M which are preserved by W D(k) and non-degenerate with sign (−1)n−1 . By Lemma 3.3, there is an element T in GL(M ) centralizing the image of ρ with B 0 (m, m0 ) = B(T m, T m0 ). This gives the identity hm, (A0 )−1 m0 i = hm, (T −1 )α A−1 T m0 i for all m and m0 . Hence A0 = T −1 AT α “ and the elements are conjugate by the element T ∈ GL(M ) = G. The argument identifying the group Cϕ with either the group C or C + associated to (M, B) is contained in §4. This completes the proof of the theorem. Corollary 8.2. — A representation M of W D(k) gives rise to a Langlands parameter for a quasi-split unitary group U(V ) if and only if W D(k0 ) fixes a non-degenerate n−1 vector in As(−1) (M ), with n = dim M . Proof. — This is a consequence of the theorem and Proposition 7.5. 8.0.2. Remark. — When M is orthogonal of even dimension, it is often convenient to view it as defining a unique Langlands parameter for the full orthogonal group O(V ) (which is not connected), with the equivalence being defined by O(M )-conjugacy; see [57]. We conclude this section by recalling certain simple invariants of the representation M of W D(k). For G = GL(V ), we have the character det M : k × → C× . For G = U(V ), we obtain the character det M : k × /k0× → C× as the sign of det M is (−1)n(n−1) = +1. Finally, for G = Sp(V ) or G = SO(V ) with dim(V ) even and disc(V ) = 1, the representation M is orthogonal with det(M ) = 1. Hence we have the root number (M ) = (M, ψ) independent of the additive character ψ of k, and (M ) = ±1. We will relate these invariants to the central characters of certain representations of G after introducing Vogan L-packets in the next section.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

31

9. Vogan L-packets - Desiderata Let G be a quasi-split, connected, reductive group over a local field k0 . In this section, we will discuss Langlands parameters ϕ as the conjectural parameters for the isomorphism classes of irreducible smooth admissible complex representations of the locally compact group G(k0 ). Before coming to that, we briefly recall the notions of smooth and admissible representations of G(k0 ) when k0 is a local field. When k0 is local and discretely valued, a smooth representation π is simply a homomorphism π : G(k0 ) −→ GL(E) for a complex vector space E (possibly infinite-dimensional) such that E = ∪K E K , where the union is over all open compact subgroups K of G(k0 ). Such a smooth representation is admissible if E K is finite dimensional for any open compact subgroup K. A homomorphism from (π, E) to (π 0 , E 0 ) is simply a linear map E −→ E 0 which commutes with the action of G(k0 ). When k0 = R or C, we will consider the category of smooth Frechet representations (π, E) of moderate growth, as introduced by Casselman [11] and Wallach [80]. An admissible representation is such a representation whose subspace of K-finite vectors (where K is a maximal compact subgroup of G(k0 )) is the direct sum of irreducible representations of K with finite multiplicities. A homomorphism (π, E) −→ (π 0 , E 0 ) is a continuous linear map E −→ E 0 which commutes with the action of G(k0 ). We come now to the local Langlands conjecture. We shall present this conjecture in a form proposed by Vogan [70], which treats representations π of all pure inner forms G0 of G simultaneously. Recall from Section 2 that the pure inner forms of G are the groups G0 over k0 which are obtained from G via inner twisting by elements in the finite pointed set H 1 (k0 , G). All of the pure inner forms G0 of G have the same center Z over k0 , and each irreducible representation π of G0 (k0 ) has a central character ωπ : Z(k0 ) → C× . The adjoint group G0ad (k0 ) acts on G0 (k0 ) by conjugation, and hence acts on the set of its irreducible complex representations. The quotient group G0ad (k0 )/Im G0 (k0 ) acts on the set of isomorphism classes of representations of G0 (k0 ). This quotient is abelian, and canonically isomorphic to the cohomology group E = G0ad (k0 )/Im G0 (k0 ) = ker(H 1 (k0 , Z) → H 1 (k0 , G0 )). It can be seen that the group E is independent of the choice of the inner form G0 of G. Let B be a Borel subgroup of G over k0 , with unipotent radical N . The quotient torus T = B/N acts on the group Hom(N, C× ). We call a character θ : N (k0 ) → C× generic if its stabilizer in T (k0 ) is equal to the center Z(k0 ). If π is an irreducible representation of G(k0 ) and θ is a generic character, then the complex vector space

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

32

W. T. GAN, B. H. GROSS & D. PRASAD

HomN (k0 ) (π, θ) has dimension ≤ 1. When the dimension is 1, we say π is θ-generic. This depends only on the T (k0 )-orbit of θ. When Z = 1, the group T (k0 ) acts simply-transitively on the set of generic characters. In general, the set D of T (k0 )-orbits on the set of all generic characters θ of N (k0 ) forms a principal homogeneous space for the abelian group E 0 : E 0 = Tad (k0 )/Im T (k0 ) = ker(H 1 (k0 , Z) → H 1 (k0 , T )). By Lemma 16.3.6(iii) of [63], Tad (k0 )/Im T (k0 ) = Gad (k0 )/Im G(k0 ), hence we have the equality E 0 = E We are now ready to describe the desiderata for Vogan L-packets, which will be assumed in the rest of this paper. These properties are known to hold for the groups GL(V ) and SL(V ), as well as for some classical groups of small rank. 1. Every irreducible representation π of G0 (k0 ) (up to isomorphism) determines a Langlands parameter “ o Gal(K/k0 ) ϕ : W D(k0 ) → G (up to equivalence). Each Langlands parameter ϕ for G corresponds to a finite set Πϕ of irreducible representations of G(k0 ) and its pure inner forms G0 (k0 ). Moreover, the cardinality of the finite set Πϕ is equal to the number of irreducible representations χ of the finite group Aϕ = π0 (Cϕ ). 2. Each choice of a T (k0 )-orbit θ of generic characters for G(k0 ) gives a bijection of finite sets J(θ) : Πϕ → Irr(Aϕ ). For archimedean v, this bijection was established by Vogan in [70, Thm. 6.3]. The L-packet Πϕ contains at most one θ0 -generic representation, for each T (k0 )-orbit of generic characters θ0 of G(k0 ). We conjectured in [24, Conjecture 2.6] that the L-packet Πϕ contains a generic representation if and only if the adjoint L-function L(ϕ, Ad, s) is regular at the point s = 1. In this case, we say the L-packet Πϕ is generic. Assume that the L-packet Πϕ is generic. In the bijection J(θ), the unique θ-generic representation π in Πϕ corresponds to the trivial representation of Aϕ . The θ0 -generic representations correspond to the one dimensional representations ηg described below. 3. The finite set Πϕ of irreducible representations of G(k0 ) is stable under the adjoint action of Gad (k0 ), which permutes the different generic representations for G(k0 ) in an L-packet transitively. In any of the bijections J(θ), the action of g ∈ G0ad (k0 ) on Irr(Aϕ ) is given by tensor product with the one-dimensional representation ηg of Aϕ alluded to above.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

33

More precisely, Tate local duality gives a perfect pairing “ → C× . H 1 (k0 , Z) × H 1 (K/k0 , π1 (G)) “ factors through the quotient Aϕ , The coboundary map Cϕ → H 1 (K/k0 , π1 (G)) and gives a pairing H 1 (k0 , Z) × Aϕ → C× . The adjoint action by the element g in G0ad (k0 ) → H 1 (k0 , Z), viewed as a one dimensional representation ηg of Aϕ , will take π(ϕ, χ) to the representation π(ϕ, χ ⊗ ηg ). 4. In any of the bijections J(θ), the pure inner form which acts on the representation with parameter (ϕ, χ) is constrained by the restriction of the irreducible “ Gal(K/k0 ) in Aϕ . representation χ to the image of the group π0 (Z(G)) More precisely, when k0 6= R, Kottwitz has identified the pointed set H 1 (k0 , G) with the group of characters of the component group of “ Gal(K/k0 ) . The inclusion Z(G) “ Gal(K/k0 ) → Cϕ Z(G) induces a map on component groups, whose image is central in Aϕ . Hence an “ Gal(K/k0 ) , irreducible representation χ of Aϕ has a central character on π0 (Z(G)) 1 and determines a class in H (k0 , G). This is the pure inner form G0 that acts on the representation π(ϕ, χ). 5. All of the irreducible representations π in Πϕ have the same central character ωπ . This character is determined by ϕ, using the recipe in [26, § 8]. 10. Vogan L-packets for the classical groups We now make the desiderata of Vogan L-packets completely explicit for the classical groups G ⊂ GL(V /k). We have already described the Langlands parameters ϕ for G explicitly, as certain representations M of W D(k), in Section 6. In all cases, the component group Aϕ is an elementary abelian 2-group, so Irr(Aϕ ) = Hom(Aϕ , ±1). We treat each family of groups in turn. The General Linear Group G = GL(V ) 1. A Langlands parameter is a representation M of W D(k), with dim(M ) = dim(V ). The group Cϕ = C(M ) is connected, so Aϕ = 1. Hence Πϕ consists of a single element. In this case, the full Langlands conjecture is known (by [29] and [31]). 2. There is a unique T -orbit on the generic characters, and the regularity of the adjoint L-function of ϕ at s = 1 detects generic L-packets Πϕ . 3. The adjoint action is trivial, as the center Z of G has trivial first cohomology. 4. The only pure inner form is G = GL(V ). 5. The center Z(k) = k × , and the central character of π(ϕ) has parameter det(M ).

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

34

W. T. GAN, B. H. GROSS & D. PRASAD

The Symplectic Group G = Sp(V ) 1. A Langlands parameter is an orthogonal representation M of W D(k), with dim(M ) = dim(V ) + 1

2.

3.

4. 5.

and

det(M ) = 1.

m−1 The group Aϕ = A+ , where m is the number of distinct irreM has order 2 ducible orthogonal summands Mi in M . The full Langlands conjecture is known when dim(V ) = 2 [46] or 4 [18]. The set D of T -orbits on generic characters is a principal homogeneous space for the group E = H 1 (k, Z) = k × /k ×2 . We will see in §12 that the choice of the symplectic space V identifies the set D with the set of k ×2 -orbits on the nontrivial additivecharacters ψ of k. The adjoint action is via elements c in the group E = k × /k ×2 . This acts on the irreducible representations of Aϕ via tensor product with the character ηc (a) = det(M a )(c), and on the set D of orbits of generic characters by mapping ψ(x) to ψ(cx). The only pure inner form is G = Sp(V ). The center Z(k) = h±1i, and the central character of π(ϕ) maps the element −1 in Z(k) to the local root number (M ).

The Odd Special Orthogonal Group G = SO(V ), dim(V ) = 2n + 1 1. A Langlands parameter is a symplectic representation M of W D(k), with dim(M ) = dim(V ) − 1. The group Aϕ = AM has order 2m , where m is the number of distinct irreducible symplectic summands Mi in M . The full Langlands conjecture is known when dim(V ) = 3 [45] or 5 [19]. 2. Since G is an adjoint group, there is a unique T -orbit on the set of generic characters, and hence a single natural bijection J : Πϕ → Hom(Aϕ , ±1). 3. The adjoint action on the L-packet is trivial. 4. The pure inner forms of G are the groups G0 = SO(V 0 ), where V 0 is an orthogonal space over k with dim(V 0 ) = dim(V ) and disc(V 0 ) = disc(V ) [40, (29.29)]. If k is non-archimedean and n ≥ 1, there is a unique non-split pure inner form 0 G , which has k-rank (n − 1). The representation π(ϕ, χ) is a representation of G if χ(−1) = +1 and a representation of G0 if χ(−1) = −1. If k = R and G = SO(p, q), then the pure inner forms are the groups G0 = SO(p0 , q 0 ) with q 0 ≡ q mod 2, and π(ϕ, χ) is a representation of one of the groups G0 with 0 (−1)(q−q )/2 = χ(−1). 5. The center Z of G is trivial. The Even Special Orthogonal Group G = SO(V ), dim(V ) = 2n, disc(V ) = d 1. A Langlands parameter determines an orthogonal representation M of W D(k), with dim(M ) = dim(V ) and det(M ) = C(d).

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

2.

3.

4.

5.

35

m The group Aϕ = A+ M has order 2 , where m is either the number of distinct irreducible orthogonal summands Mi in M , or the number of distinct orthogonal summands minus 1. The latter case occurs if some irreducible orthogonal summand Mi has odd dimension (in which case the orthogonal representation M determines the parameter ϕ). If every irreducible orthogonal summand Mi of M has even dimension, ∗ then A+ M = AM and there are two parameters {φ, φ } which determine the same orthogonal representation M . The representations π(φ, χ) and π(φ∗ , χ) are conjugate under the outer action of O(V ) on SO(V ). The full Langlands conjecture is known when dim(V ) = 2 or 4 or 6. The set D of T -orbits on generic characters is a principal homogeneous space for the group E = NK × /k ×2 , where K is the splitting field of G. We will see in §12 that the choice of the orthogonal space V identifies the set D with the set of G-orbits on the set of non-isotropic lines L ⊂ V , such that the space L⊥ is split. The adjoint action is via elements c in the group E = NK × /k ×2 . This acts on the irreducible representations of Aϕ via tensor product with the character ηc (a) = det(M a )(c), and on the set D of orbits of generic characters by mapping a line L = kv with hv, vi = α in k × to a line L0 = kv 0 with hv 0 , v 0 i = c · α. The pure inner forms of G are the groups G0 = SO(V 0 ), where V 0 is an orthogonal space over k with dim(V 0 ) = dim(V ) and disc(V 0 ) = disc(V ) [40, (29.29)]. If k is non-archimedean and V is not the split orthogonal space of dimension 2, there is a unique pure inner form G0 , such that the Hasse-Witt invariant of V 0 is distinct from the Hasse-Witt invariant of V . The representation π(ϕ, χ) is a representation of G if χ(−1) = +1 and a representation of G0 if χ(−1) = −1. If k = R and G = SO(p, q), then the pure inner forms are the groups G0 = SO(p0 , q 0 ) with q 0 ≡ q mod 2, and π(ϕ, χ) is a representation of one of the 0 groups G0 with (−1)(q−q )/2 = χ(−1). If dim(V ) = 2, then Z = G. If dim(V ) ≥ 4, then Z(k) = h±1i and the central character of π(ϕ) maps the element −1 in Z(k) to (M, ψ)/(det M, ψ).

The Odd Unitary Group G = U(V ), dim V = 2n + 1 1. A Langlands parameter is a conjugate-orthogonal representation M of W D(k), with dim(M ) = dim(V ). The group Aϕ = AM has order 2m , where m is the number of distinct irreducible conjugate-orthogonal summands Mi in M . The full Langlands conjecture is known when dim(V ) = 1 or 3 [60]. 2. There is a unique T -orbit on the set of generic characters, and hence a single natural isomorphism J : Πϕ → Hom(Aϕ , ±1). 3. The adjoint action on the L-packet is trivial. 4. The pure inner forms of G are the groups G0 = U(V 0 ), where V 0 is a hermitian (or skew-hermitian) space over k with dim(V 0 ) = dim(V ) [40, (29.19)].

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

36

W. T. GAN, B. H. GROSS & D. PRASAD

If k0 is non-archimedean, there is a unique pure inner form G0 such that the discriminant of V 0 is distinct from the discriminant of V . The representation π(ϕ, χ) is a representation of G if χ(−1) = +1 and a representation of G0 if χ(−1) = −1. If k0 = R and G = U(p, q), then the pure inner forms are the groups G0 = U(p0 , q 0 ), and π(ϕ, χ) is a representation of one of the groups G0 0 with (−1)q−q = χ(−1). 5. The center Z(k0 ) = k × /k0× = U(1), and the central character of π(ϕ, χ) has parameter det(M ). The Even Unitary Group G = U(V ), dim V = 2n 1. A Langlands parameter is a conjugate-symplectic representation M of W D(k), with dim(M ) = dim(V ). The group Aϕ = AM has order 2m , where m is the number of distinct irreducible conjugate-symplectic summands Mi in M . The full Langlands conjecture is known when dim(V ) = 2 [60]. 2. The set D of T -orbits on generic characters is a principal homogeneous space for the group E = H 1 (k, Z) = k0× /Nk × of order 2. We will see in §12 that the choice of a hermitian space V identifies the set D with the set of Nk × orbits on the nontrivial additive characters ψ of k/k0 . Similarly, the choice of a skew-hermitian space V identifies the set D with the set of Nk × -orbits on the nontrivial additive characters ψ0 of k0 . 3. The adjoint action is via elements c in the group E = k0× /Nk × . The nontrivial class c acts on the irreducible representations of Aϕ via tensor product with the a character η(a) = (−1)dim(M ) , and on the set D of orbits of generic characters by mapping ψ(x) to ψ(cx), or ψ0 (x) to ψ0 (cx). 4. The pure inner forms of G are the groups G0 = U(V 0 ), where V 0 is a hermitian (or skew-hermitian) space over k with dim(V 0 ) = dim(V ) [40, (29.19)]. If k0 is non-archimedean, there is a unique pure inner form G0 such that the discriminant of V 0 is distinct from the discriminant of V . The representation π(ϕ, χ) is a representation of G if χ(−1) = +1 and a representation of G0 if χ(−1) = −1. If k0 = R and G = U(p, q), then the pure inner forms are the groups G0 = U(p0 , q 0 ), and π(ϕ, χ) is a representation of one of the groups G0 0 with (−1)q−q = χ(−1). 5. The center Z(k0 ) = k × /k0× = U(1), and the central character of π(ϕ, χ) has parameter det(M ). The forthcoming book of Arthur [7] and the papers [49, 50] of Mœglin should establish most of the above expectations.

11. Vogan L-packets for the metaplectic group Let (W, h−, −iW ) be a symplectic space of dimension 2n ≥ 0 over the local field k. We assume, as usual, that char(k) 6= 2. In this section, we also assume that k 6= C.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

37

f Let Sp(W ) denote the nontrivial double cover of the symplectic group Sp(W )(k). We will use the Howe duality correspondence (also known as the theta corresponf dence) to describe the (genuine) representation theory of Sp(W ) in terms of the representation theory of the groups SO(V ) over k, with dim V = 2n + 1. Assuming the Langlands-Vogan parametrization of irreducible representations of SO(V ) over k, with dim V = 2n + 1, we then obtain a notion of Vogan L-packets for the genuine f irreducible representations π e of Sp(W ). More precisely, the Langlands parameter of f a genuine representation of Sp(W ) will be a symplectic representation ϕ : W D(k) → Sp(M ) with

dim M = 2n,

and the individual representations π e(ϕ, χ) in the Vogan packet Πϕ will be indexed by quadratic characters χ : Aϕ = AM → h±1i. f This parametrization of the irreducible genuine representations of Sp(W ) will depend on the choice of a nontrivial additive character ψ of k, up to multiplication by k ×2 . As we shall see in §12, such an orbit of additive characters ψ determines an orbit of generic characters θ : N → C× for Sp(W ). The character θ also determines ‹ ' N of Sp(W f a character θe of the unipotent radical N ). Our parametrization is normalized so that for generic parameters ϕ, the unique representation π e ∈ Πϕ which is e θ-generic corresponds to the trivial character χ = 1 of Aϕ . Such a dependence of the Langlands parametrization on the choice of an additive character ψ is already present in the case of the linear classical groups discussed in the previous section (through fixing of a generic character on a quasi-split form of the group). For the metaplectic groups, the dependence is more serious: even the Langlands parameter ϕ associated to π e depends on the choice of ψ. To define the parameters (ϕ, χ) of π e, we let (V, q) be a quadratic space over k with dim V = 2n + 1 and disc(V ) = (−1)n det(V ) ≡ 1 ∈ k × /k ×2 . Note that the discriminant above refers to the discriminant of the quadratic space (V, q). The quadratic form q on V gives rise to a symmetric bilinear form hx, yiV = q(x + y) − q(x) − q(y) so that hx, xiV = 2 · q(x), and disc(V, h−, −iV ) = 2 · disc(V, q) = 2 ∈ k × /k ×2 . The space W ⊗ V is a symplectic space over k with the skew-symmetric form h−, −iW ⊗ h−, −iV , and one has the associated Heisenberg group H(W ⊗ V ) = k ⊕ (W ⊗ V ),

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

38

W. T. GAN, B. H. GROSS & D. PRASAD

which has a one dimensional center k. Associated to ψ, H(W ⊗ V ) has a unique irreducible representation ωψ with central character ψ (by the Stone-von-Neumann theorem). Now Sp(W ⊗ V ) acts as automorphisms of H(W ⊗ V ) via its natural action on W ⊗V and the trivial action on k. Thus ωψ gives rise to a projective representation of Sp(W ⊗ V ) and it was shown by Weil that this projective representation is a linear f representation of Sp(W ⊗ V ). We thus have a representation ωψ of the semi-direct product f Sp(W ⊗ V ) n H(W ⊗ V ). This is the so-called Weil representation (associated to ψ). As a representation of f Sp(W ⊗V ), it is the direct sum of two irreducible representations, and its isomorphism class depends only on the k ×2 -orbit of ψ. Via a natural homomorphism f f Sp(W ) × O(V ) −→ Sp(W ⊗ V ), f we regard the Weil representation ωψ as a representation ωW,V,ψ of Sp(W ) × O(V ). The theory of Howe duality gives a correspondence between irreducible genuine repf resentations π e of Sp(W ) and certain irreducible representations σ of O(V ). More precisely, given an irreducible representation σ of O(V ), the maximal σisotypic quotient of ωW,V,ψ has the form σ  ΘW,V,ψ (σ) f for some smooth representation ΘW,V,ψ (σ) (the big theta lift of σ) of Sp(W ). It is known ([42] and [51]) that ΘW,V,ψ (σ) is either zero or has finite length. Let θW,V,ψ (σ) (the small theta lift of σ) denote the maximal semisimple quotient of ΘW,V,ψ (σ). It is known by Howe [32] and Waldspurger [72] that when the residue characteristic of k is different from 2, then θW,V,ψ (σ) is irreducible or zero; this is the so-called Howe’s conjecture. In the following, we will assume that the same holds when the residue characteristic of k is 2. f Analogously, if π e is an irreducible representation of Sp(W ), we have the representations ΘW,V,ψ (e π ) and θW,V,ψ (e π ) of O(V ). Now we have the following theorem, which is due to Adams-Barbasch [1] when k = R and follows from fundamental results of Kudla-Rallis [44] when k is nonarchimedean. Theorem 11.1. — Assume that the local field k is either real or non-archimedean with odd residual characteristic. Then corresponding to the choice of an additive character ψ of k, there is a natural bijection given by the theta correspondence: ¶ © f irreducible genuine representations π e of Sp(W ) O `

ASTÉRISQUE 346

 {irreducible representations σ 0 of SO(V 0 )}

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

39

where the union is disjoint, and taken over all the isomorphism classes of orthogonal spaces V 0 over k with dim V 0 = 2n + 1 and disc(V 0 ) = 1. f More precisely, given an irreducible representation π e of Sp(W ), there is a unique 0 V as above such that θW,V 0 ,ψ (e π ) is nonzero, in which case the image of π e under the above bijection is the restriction of θW,V 0 ,ψ (e π ) to SO(V 0 ) (note that this restriction is irreducible since O(V 0 ) = SO(V 0 ) × h±1i). Proof. — We give a sketch of the proof of Theorem 11.1 when k is non-archimedean; a detailed proof can be found in [17]. Let’s begin by noting that there are 2 isomorphism classes of quadratic space of dimension 2n+1 and trivial discriminant; we denote these by V and V 0 , and assume that V is split. To simplify notation, we shall write Θ in place of ΘW,V,ψ and Θ0 in place of ΘW,V 0 ,ψ . We now divide the proof into two steps: f (i) Given an irreducible representation π e of Sp(W ), exactly one of Θ(e π ) or Θ0 (e π ) is nonzero. This dichotomy was also shown in the recent paper of C. Zorn [87]. In any case, f [44, Thm. 3.8] shows that any irreducible representation π e of Sp(W ) participates in 0 theta correspondence with at most one of O(V ) or O(V ). We claim however that π e does have nonzero theta lift to O(V ) or O(V 0 ). To see this, note that [44, Prop. 4.1] shows that π e has nonzero theta lift to O(V ) if and only if HomS‹p(W )×S‹p(W ) (R(V ), π eπ e∨ ) 6= 0, f where R(V ) is the big theta lift of the trivial representation of O(V ) to Sp(W + W −) − (where W is the symplectic space obtained from W by scaling its form by −1). Similarly, one has the analogous statement for V 0 . On the other hand, if IP (s) denotes f the degenerate principal series representation of Sp(W + W − ) unitarily induced from s the character χψ · | det | of the Siegel parabolic subgroup stabilizing the maximal isotropic subspace ∆W , the diagonal W in W ⊕ W − , and χψ a genuine character of g GL(∆W ) defined in §16, then it was shown by Sweet [68] that IP (0) = R(V ) ⊕ R(V 0 ). It follows that R(V ) and R(V 0 ) are unitarizable and thus irreducible (since they have a unique irreducible quotient). In particular, we conclude that π e has nonzero theta lift to one of O(V ) or O(V 0 ) if and only if HomS‹p(W )×S‹p(W ) (IP (0), π eπ e∨ ) 6= 0. We thus need to show that this Hom space is nonzero. This can be achieved by the doubling method of Piatetski-Shapiro and Rallis [20] (cf. also [17] and [87]), which provides a zeta integral Z(s) : IP (s) ⊗ π e∨ ⊗ π e −→ C. The precise definition of Z(s) need not concern us here; it suffices to note that for a flat section Φ(s) ∈ IP (s) and f ⊗ f ∨ ∈ π e⊗π e∨ , Z(s, Φ(s), f ⊗ f ∨ ) is a meromorphic

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

40

W. T. GAN, B. H. GROSS & D. PRASAD

function in s. Moreover, at any s = s0 , the leading term of the Laurent expansion of Z(s) gives a nonzero element Z ∗ (s0 ) ∈ HomS‹p(W )×S‹p(W ) (IP (s0 ), π eπ e∨ ). For these basic properties of zeta integrals, see [17] or [87]. This proves our contention that π e participates in the theta correspondence with exactly one of O(V ) or O(V 0 ). By (i), one obtains a map ¶ © f irreducible genuine representations π e of Sp(W )

`

 {irreducible representations σ 0 of O(V 0 )}

Moreover, this map is injective by the theorem of Waldspurger [72] proving Howe’s conjecture. (ii) An irreducible representation π0 of SO(V ) has two extensions to O(V ) = SO(V )× h±1i, and exactly one of these extensions participates in the theta correspondence with f Sp(W ). The same assertion holds for representations of SO(V 0 ). Suppose on the contrary that π is an irreducible representation of O(V ) such that f both π and π ⊗ det participate in theta correspondence with Sp(W ), say π e = θW,V,ψ (π) and π e0 = θW,V,ψ (π ⊗ det). Now consider the seesaw diagram: f Sp(W + W −)

O(V ) × O(V )

f f Sp(W ) × Sp(W )

O(V ).

The seesaw identity implies that HomS‹p(W )×S‹p(W ) (ΘW +W − ,V,ψ (det), π e0  π e∨ ) ⊃ HomO(V ) ((π ⊗ det) ⊗ π ∨ , det) 6= 0. This implies that ΘW +W − ,V,ψ (det) 6= 0. However, a classical result of Rallis [59, Appendix] says that the determinant character f of O(V ) does not participate in the theta correspondence with Sp(4r) for r ≤ n. This gives the desired contradiction. We have thus shown that at most one of π or π ⊗ det could have nonzero theta f lift to Sp(W ). On the other hand, the analog of the zeta integral argument in (i) f shows that one of π or π ⊗ det does lift to Sp(2n); for this, one needs the structure

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

41

of the degenerate principal series representation IP (∆V ) (0) of O(V + V − ) which is determined in [9] (see also [83, Prop. 3.3]). This proves (ii). Putting (i) and (ii) together, we have established the theorem. The only reason for the assumption of odd residue characteristic in the theorem is that Howe’s conjecture for local theta correspondence is only known under this assumption. Since V is an odd dimensional quadratic space, SO(V ) is an adjoint group, there is a unique orbit of generic characters on it, and the Vogan parametrization of irreducible representations σ 0 of the groups SO(V 0 ) requires no further choices. So we label π e= π e(M, χ) using the Vogan parameters (M, χ) of the representation σ 0 = ΘW,V 0 ,ψ (e π ). The theorem thus gives the following corollary. Corollary 11.2. — Assume that the residue characteristic of k is odd. Suppose that the local Langlands-Vogan parametrization holds for SO(V 0 ). Then one has a parametrization (depending on ψ) of f {irreducible genuine representations π e of Sp(W )} by the set of isomorphism classes of pairs (ϕ, χ) such that ϕ : W D(k) −→ Sp(M ) is a symplectic representation of W D(k) and χ is an irreducible character of the component group Aϕ . f It follows that the various desiderata for the Vogan packets of Sp(W ) can be ob0 tained from those of SO(V ) if one understands the properties of the theta correspondence. For example, in the theta correspondence, generic representations of the split e e f group SO(V ) lift to θ-generic representations of Sp(W ). Hence the θ-generic element in the L-packet of M corresponds to the trivial character of the component group AM . Also, when k is non-archimedean, π e(M, χ) is lifted from the split group SO(V ) precisely when χ(−1) = 1. For these and other similar issues, see [17]. One difference between metaplectic and linear groups is in the description of the action of the adjoint group by outer automorphisms on the set of irreducible representations. The adjoint action of the symplectic similitude group GSp(W ) on the set f of genuine irreducible representations of Sp(W ) factors through the quotient k × /k ×2 = PGSp(W )(k)/Image Sp(W )(k). In the metaplectic case, this outer action does not permute the representations π e in an individual Vogan L-packet, and we predict a more complicated recipe, as follows. Conjecture 11.3. — If π e has ψ-parameter (M, χ) and c is a class in k × /k ×2 , the confc has ψ-parameter (M (c), χ · η[c]). Here M (c) is the twist of jugated representation π M by the one-dimensional orthogonal representation C(c) so that its component group AM (C) is canonically isomorphic to AM . The character η[c] is defined by η[c] = χN : AM → h±1i,

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

42

W. T. GAN, B. H. GROSS & D. PRASAD

where N is the two dimensional orthogonal representation N = C + C(c), so that 1

a

η[c](a) = (M a ) · (M (c)a ) · (c, −1) 2 dim M . This conjecture is known when dim W = 2, where it is a result of Waldspurger ([80] and [81]); our recipe above is suggested by his results. The above conjecture has the following consequence. If one replaces the character ψ by the character ψc : x 7→ ψ(cx) of k, then the new Vogan parameter (relative to ψc ) of π e will be (M (c), χ · η[c]). A consequence of this is the following. Suppose that π e is such that θW,V,ψ (e π ) 6= 0 and θW,V,ψc (e π ) 6= 0 as representations of SO(V ). Then, when dim W = 2, a basic result of Waldspurger says that θW,V,ψ (e π) ∼ π ) ⊗ χc , = θW,V,ψc (e where χc is the character spinor norm

(c,−)

SO(V )(k) −−−−−−−−→ k × /k ×2 −−−−→ h±1i. However, according to the conjecture above, if the Vogan parameter of θW,V ψ (e π ) is (M, χ), then that of θW,V,ψc (e π ) ⊗ χc is (M, χ · η[c]). So the two representations are equal if and only if the character η[c] is trivial. The assumption that π e has nonzero theta lift to SO(V ) with respect to both ψ and ψc implies that η[c](−1) = 1. When dim W = 2, this is equivalent to saying that η[c] is trivial. But when dim W > 2, this is no longer the case and one can construct such counterexamples already when dim W = 4.

12. The representation ν of H and generic data In this section, we shall describe the remaining ingredient in the restriction problem to be studied. Suppose as before that k is a local field with an involution σ (possibly trivial) and k0 is the fixed field of σ. Let V be a k-vector space endowed with a nondegenerate sesquilinear form h−, −i with sign . Moreover, suppose that W ⊂ V is a non-degenerate subspace satisfying: ⊥

1.  · (−1)dim W = −1 2. W ⊥ is a split space. So we have ( dim W

⊥

ASTÉRISQUE 346

=

odd, if  = 1, i.e. V is orthogonal or hermitian; even, if  = −1, i.e. V is symplectic or skew-hermitian.

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

43

Let G(V ) be the identity component of the automorphism group of V and G(W ) ⊂ G(V ) the subgroup which acts as identity on W ⊥ . Set G = G(V ) × G(W ). As explained in Section 2, G contains a subgroup H defined as follows. Since W ⊥ is split, we may write W ⊥ = X + X∨

or W ⊥ = X + X ∨ + E

depending on whether dim W ⊥ is even or odd, where in the latter case, E is a nonisotropic line. Let P be a parabolic subgroup which stabilizes a complete flag of (isotropic) subspaces in X. Then G(W ) is a subgroup of a Levi subgroup of P and thus acts by conjugation on the unipotent radical N of P . We set H = N o G(W ). Note that there is a natural embedding H ,→ G which is the natural inclusion H ⊂ P ⊂ G(V ) in the first factor and is given by the projection H −→ H/N = G(W ) in the second factor. When G0 = G(V 0 ) × G(W 0 ) is a relevant pure inner form of G, a similar construction gives a distinguished subgroup H 0 . The goal of this section is to describe a distinguished representation ν of H (and similarly H 0 ). It will turn out that dim ν = 1 if dim W ⊥ is odd (orthogonal and hermitian cases), whereas ν has Gelfand-Kirillov dimension 1/2 · dim(W/k0 ) when dim W ⊥ is even (symplectic and skew-hermitian cases). Because of this, we will treat the cases when dim W ⊥ is even or odd separately. Orthogonal and Hermitian Cases (Bessel Models) Assume that dim W ⊥ = 2n + 1 and write W ⊥ = X + X∨ + E

with E = hei,

where X and X ∨ are maximal isotropic subspaces which are in duality using the form h−, −i of V , and e is a non-isotropic vector. Let P (X) be the parabolic subgroup in G(V ) stabilizing the subspace X, and let M (X) be the Levi subgroup of P (X) which stabilizes both X and X ∨ , so that M (X) ∼ = GL(X) × G(W ⊕ E). We have P (X) = M (X) n N (X) where N (X) is the unipotent radical of P (X). The group N (X) sits in an exact sequence of M (X)-modules, 0 −−−−→ Z(X) −−−−→ N (X) −−−−→ N (X)/Z(X) −−−−→ 0,

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

44

W. T. GAN, B. H. GROSS & D. PRASAD

and using the form on V , one has natural isomorphisms Z(X) ∼ = {skew-hermitian forms on X ∨ }, and N (X)/Z(X) ∼ = Hom(W + E, X) ∼ = (W + E) ⊗ X. ∨ Here, when k = k0 , skew-hermitian forms on X simply mean symplectic forms. In particular, Z(X) is the center of N (X) unless k = k0 and dim X = 1, in which case Z(X) is trivial and N (X) is abelian. Now let `X : X → k be a nonzero k-linear homomorphism, and let `W : W ⊕ E −→ k be a nonzero k-linear homomorphism which is zero on the hyperplane W . Together, these give a map `X ⊗ `W : X ⊗ (W + E) −→ k, and one can consider the composite map ` ⊗`

X W `N (X) : N (X) −−−−→ N (X)/Z(X) ∼ = X ⊗ (W + E) −−−−−→ k.

Let UX be any maximal unipotent subgroup of GL(X) which stabilizes `X . Then the subgroup UX × G(W ) ⊂ M (X) fixes the homomorphism `N (X) . Now the subgroup H ⊂ G = G(V ) × G(W ) is given by H = (N (X) o (UX · G(W )) = N o G(W ). We may extend the map `N (X) of N (X) to H, by making it trivial on UX × G(W ). If ψ is a nontrivial additive character of k, and λX : UX −→ S1 is a generic character of UX , then the representation ν of H is defined by ν = (ψ ◦ `N (X) )  λX . The pair (H, ν) is uniquely determined up to conjugacy in the group G = G(V ) × G(W ) by the pair W ⊂ V . One can give a more explicit description of (H, ν), by explicating the choices of `X , UX and λX above. To do this, choose a basis {v1 , · · · , vn } of X, with dual basis {vi∨ } of X ∨ . Let P ⊂ G(V ) be the parabolic subgroup which stabilizes the flag 0 ⊂ hv1 i ⊂ hv1 , v2 i ⊂ · · · ⊂ hv1 , · · · , vn i = X, and let L = (k × )n × G(W + E)

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

45

be the Levi subgroup of P which stabilizes the lines hvi i as well as the subspace W +E. The torus T = (k × )n scales these lines: t(vi ) = ti vi , and G(W + E) acts trivially on X + X ∨ . Let N be the unipotent radical of P , so that N = UX n N (X) where UX is the unipotent radical of the Borel subgroup in GL(X) stabilizing the chosen flag above. Now define a homomorphism f : N → k n by f (u) xi z

=

(x1 , · · · , xn−1 , z),

= huvi+1 , vi∨ i,

i = 1, 2, · · · , n − 1

= hue, vn∨ i.

The subgroup of L which fixes f is G(W ), the subgroup of G(W +E) fixing the vector e. The torus acts on f by f (tut−1 ) = ((t1 /t2 )x1 , (t2 /t3 )x2 , · · · , tn z). Consider the subgroup H = N · G(W ) of G = G(V ) × G(W ). Then, for a nontrivial additive character ψ of k, the representation ν is given by: ν

: H → C× (u, g) 7→ ψ(

P

xi + z).

It is as regular as possible on N , among the characters fixed by G(W ). As noted above, up to G-conjugacy, the pair (H, ν) depends only on the initial data W ⊂ V , and not on the choices of ψ, {vi }, or e used to define it. A special case of (H, ν) is worth noting. If V is orthogonal of even dimension and W has dimension 1, then SO(W ) = 1 and H = N is the unipotent radical of a Borel subgroup P ⊂ G = SO(V ). In this case, ν is simply a generic character θW of N . By choosing different non-isotropic lines L in the 2-dimensional orthogonal space W + E, so that L⊥ = X + X ∨ + L0 , and using L in place of W in the above construction, the map L 7→ θL gives a bijection {T -orbits of generic characters on N } O  {SO(V )-orbits of non-isotropic lines L with L⊥ split} in the even orthogonal case. This bijection was described in [24, Prop. 7.8].

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

46

W. T. GAN, B. H. GROSS & D. PRASAD

Symplectic and Skew-Hermitian Cases (Fourier-Jacobi Models) We now treat the symplectic and skew-hermitian cases, so that W ⊥ is split of even dimension 2n and we may write W ⊥ = X + X ∨, where X and X ∨ are maximal isotropic subspaces which are in duality using the form on V . In this case, G(W ) is a subgroup of Sp(W/k0 ), preserving the form Trk/k0 ◦ h−, −i. It will turn out that the representation ν of H depends on some other auxiliary data besides the spaces W ⊂ V . As in the case of Bessel models, we include the case k = k0 × k0 in our discussion below. We assume first that dim X > 0. Let P (X) = M (X) · N (X) be the parabolic subgroup in G(V ) stabilizing the subspace X, with Levi subgroup ∼ GL(X) × G(W ) M (X) = stabilizing both X and X ∨ . Let Z(X) be the center of the unipotent radical N (X) of P (X), so that one has the exact sequence of M (X)-modules: 0 −−−−→ Z(X) −−−−→ N (X) −−−−→ N (X)/Z(X) −−−−→ 0, Using the form on V , one has natural isomorphisms Z(X) ∼ = {hermitian forms on X ∨ } and N (X)/Z(X) ∼ = Hom(W, X) ∼ = W ⊗ X. Here, if k = k0 , then hermitian forms on X ∨ simply mean symmetric bilinear forms. The commutator map [−, −] : N (X) × N (X) → N (X) factors through N (X)/Z(X) and takes value in Z(X). It thus gives rise to a skewsymmetric k0 -bilinear map Λ2k0 (X ⊗ W ) −→ Z(X) = {hermitian forms on X ∨ }, or equivalently by duality, a map {hermitian forms on X} −→ Λ2k0 (X ∨ ⊗W ) = {symplectic forms on Resk/k0 (X ⊗ W )}. Indeed, this last map is a reflection of the fact that, using the skew-hermitian structure on W , the space of hermitian forms on X can be naturally embedded in the space of skew-hermitian forms on X ⊗ W , and then by composition with the trace map if necessary, in the space of symplectic forms on Resk/k0 (X ⊗ W ). Now let `X : X → k be a nonzero homomorphism, and let UX ⊂ GL(X) be a maximal unipotent subgroup which fixes `X . Then the group H is defined by H = N (X) o (UX × G(W )) = N o G(W )

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

47

with N = N (X) o UX . If λX : UX −→ S1 is a generic character of UX , then by composing with the projection from H to UX , we may regard λX as a character of H. On the other hand, by pulling back, the homomorphism `X gives rise to a linear map k0 = {hermitian forms on k} −→ {hermitian forms on X}, and hence by duality `Z(X) : Z(X) = {hermitian forms on X ∨ } −→ k0 . Moreover, `X gives a k-linear map `W : X ⊗ W → W making the following diagram commute: [−,−]

Λ2k0 (X ⊗ W )   `W y

−−−−→

Λ2k0 (W )

−−− −−−−−−−−−−→ 2 [k:k0 ]

Z(X)  ` y Z(X)

·T rk/k0 (h−,−i)

k0 .

For example, when k = k0 and dim X = 1, then the commutator map [−, −] is given by the skew-symmetric form 2 · h−, −iW on N (X)/Z(X) = W . On the other hand, when k 6= k0 and dim X = 1, it is given by the skew-symmetric form T rk/k0 (h−, −iW ) on W/k0 . In any case, let us set ( the rank 1 quadratic space with discriminant 1 if k = k0 ; V1 = the rank 1 hermitian space with discriminant 1 if k 6= k0 , and let H(V1 ⊗ W ) be the Heisenberg group associated to the symplectic vector space V1 ⊗ W over k0 with form T rk/k0 (h−, −iV1 ⊗ h−, −iW ). Here, given a quadratic space (V, q) over k0 , the associated symmetric bilinear form is hv1 , v2 i = q(v1 + v2 ) − q(v1 ) − q(v2 ). Thus, when k = k0 , the form on V1 is such that hv, viV = 2, so that H(V1 ⊗ W ) is the Heisenberg group associated to the symplectic vector space (W, 2 · h−, −iW ). Now one has the following commutative diagram of algebraic groups over k0 : 0 −−−−→ Z(X) −−−−→   `0,X y 0 −−−−→

k0

N (X)   y

−−−−→ X ⊗ W −−−−→ 0   `W y

−−−−→ H(V1 ⊗ W ) −−−−→

W

−−−−→ 0.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

48

W. T. GAN, B. H. GROSS & D. PRASAD

Given a nontrivial character ψ0 : k0 → S1 , one may consider the unique irreducible unitarizable representation ωW,ψ0 of H(V1 ⊗ W ) of Gelfand-Kirillov dimension 21 · dimk0 W , on which the center of H(V1 ⊗ W ) acts by ψ0 . Pulling back by the above diagram, one obtains an irreducible representation ωψ0 of N (X) with central character ψ0 ◦ `Z(X) . Up to conjugation by M (X), the representation ωψ0 depends only on ψ0 up to multiplication by (k × )1+σ . The representation ωψ0 can be extended trivially to UX . Moreover, the group G(W ) acts as outer automorphisms of H(V1 ⊗ W ), so the theory of Weil representations furnishes us with a projective representation of G(W ) on ωψ0 . Thus, one has a projective representation ωψ0 of H. As in the orthogonal and hermitian cases, we can make the above discussion completely explicit by making specific choices of `X , UX and λX . Assuming that dim X = n > 0, choose a basis {vi } for X and let {vi∨ } be the dual basis of X ∨ . Let P ⊂ G(V ) be the subgroup stabilizing the flag 0 ⊂ hv1 i ⊂ · · · ⊂ hv1 , · · · , vn i = X, and let L = G(W ) × (k × )n be the Levi subgroup of P stabilizing the lines hvi i as well as the subspace W . Let N be the unipotent radical of P and define a homomorphism to a vector group f : N → k n−1 ⊕ W given by: f (u) xi y

=

(x1 , · · · , xn−1 , y)

= huvi+1 , vi∨ i P ∨ ∨ = j huwj , vn i.wj .

Here {w1 , · · · , wn } is a basis for W over k and hwi , wj∨ i = δij . Thus y is the unique vector in W with hw, yi = huw, vn∨ i for all w in W . The torus T = (k × )n acts on f by f (tut−1 ) = ((t1 /t2 )x1 , · · · , (tn−1 /tn )xn , tn · y) and an element g ∈ G(W ) acts by f (gug −1 ) = (x1 , · · · , xn , g(y)). Now the maps (x1 , · · · , xn−1 ) give a functional ` : N → k0 P n 7→ Tr( xi )

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

49

which is fixed by G(W ), and is as regular as possible subject to this condition. Choose a nontrivial additive character ψ0 of k0 . Then the character λ

: N → S1 P u 7→ ψ0 (`(u)) = ψ0 (Tr( xi ))

is regular, and up to conjugacy by the torus, independent of the choice of ψ0 . Since it is fixed by G(W ), we may extend it trivially to G(W ) and obtain a character λ of H. On the other hand, one may define a homomorphism of N to a Heisenberg group as follows. Let N0 C N be the kernel of the map N

→W

u

7→ y

and define a homomorphism f0

: N0 → k u0 7→ z = hu0 vn∨ , vn∨ i.

Note that the element z lies in the subfield k0 of k, since u0 vn∨ is isotropic =⇒ z − z σ = 0. Hence, we have f0 : N0 −→ k0 . The torus act by f0 (tut−1 ) = t1+σ ·z n and G(W ) acts trivially. The above two maps combine to give a homomorphism from N to the Heisenberg group H(W/k0 ): 0 −−−−→ N0 −−−−→ N −−−−→ W −−−−→ 0       f0 y y yid 0 −−−−→ k0 −−−−→ H(V1 ⊗ W ) −−−−→ W −−−−→ 0 which is equivariant for the action of G(W ) on N and G(W ) ⊂ Sp(V1 ⊗ W ) on H(V1 ⊗ W ). The nontrivial additive character ψ0 then gives rise to the projective representation ωψ of H as above. It is now more convenient to consider the symplectic and skew-hermitian cases separately. (i) (symplectic case) When k = k0 , we have G(W ) = Sp(W ). In this case, for each character ψ of k = k0 , it is known that the projective representation ωψ of ‹ ) = Sp(W f G(W ) lifts to a linear representation of the double cover G(W ), the metaplectic group. Recalling that H = N o G(W ), we thus obtain a unitary representation νψ = ωψ ⊗ λ

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

50

W. T. GAN, B. H. GROSS & D. PRASAD

of

‹ = N o G(W ‹ ) H in the case when dim W > 0. When W ⊥ = 0, so that W = V , we have ⊥

N = {1} and H = G(W ) = G(V ). In this case, we simply set νψ = ωψ , ‹ which is a representation of H. In each case, the representation νψ has Gelfand-Kirillov dimension 1/2 · dimk (W ). Up to conjugation by the normalizer of H in G, νψ depends only on ψ up to the action of (k × )2 . A particular case of this is worth noting. When W = 0, so that G(W ) is trivial, the group H is simply the unipotent radical N of the Borel subgroup P and νψ is simply a generic character of N . This gives a bijection {T -orbits of generic characters of N } O 

 k ×2 -orbits of nontrivial characters ψ of k

in the symplectic case. (ii) (skew-hermitian case) When k 6= k0 , G(W ) = U(W ). In this case, the projective representation ωψ0 of G(W ) = U(W ) lifts to a linear representation of G(W ), but when dim W > 0, the lifting is not unique: it requires the choice of a character µ : k × → C× whose restriction to k0× is the quadratic character ωk/k0 associated to k/k0 [28]. Equivalently, when k is a field, it requires the choice of a 1-dimensional, conjugate-dual representation of W D(k) with sign c = −1. Given such a µ, we let ωψ0 ,µ be the corresponding representation of G(W ) and set νψ0 ,µ = ωψ0 ,µ ⊗ λ. Hence, we have defined an irreducible unitary representation νψ0 ,µ of H = N.U(W ) when dim W ⊥ > 0. When W ⊥ = 0, so that W = V , we have N = {1} and H = G(W ) = G(V ). In this case, we simply set νψ0 ,µ = ωψ0 ,µ . In each case, the representation νψ0 ,µ has Gelfand-Kirillov dimension 1/2 · dimk0 (W ). It depends, up to conjugation by the normalizer of H in G = U(V )× U(W ), on ψ0 up to the action of Nk × (as well as the choice of µ).

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

51

A particular case of this is noteworthy. When W = 0 (and V is even dimensional), so that G(W ) is trivial, the group H is simply the unipotent radical N of the Borel subgroup P and there is no need to choose µ. Hence, νψ0 ,µ = νψ0 is simply a generic character of N . This gives a bijection {T -orbits of generic characters of N } O  {Nk × -orbits of nontrivial characters ψ0 of k0 } in the even skew-hermitian case. Remarks. — Note that when W = 0, there is no need to invoke the Weil representation at all. Hence, the above description of generic characters could be carried out for hermitian spaces (of even dimension 2n) as well. One would consider the situation W =0⊂V

of hermitian spaces.

and note that the homomorphisms f : N −→ k n−1 and f0 : N = N0 −→ k can still be defined by the same formulas. But now the image of f0 lies in the subspace of trace zero elements of k (as opposed to the subfield k0 when V is skew-hermitian). The torus actions on f and f0 are again given by the same formulas. Thus, giving a T -orbit of generic characters of N in the even hermitian case amounts to giving a nontrivial character of k trivial on k0 , up to the action of Nk × . This completes our definition of the representation ν of H. As we noted in the course of the discussion above, special cases of the pair (H, ν) give the determination of T -orbits of generic characters. Recall that if G = G(V ) is quasi-split, with Borel subgroup B = T · N , then the set D of T (k0 )-orbits of generic characters of N is a principal homogeneous space for the abelian group E = T ad (k0 )/Im T (k0 ) = ker(H 1 (k0 , Z) → H 1 (k0 , T )). When G = GL(V ) or U(V ) with dim V odd or SO(V ) with dim V odd or dim V = 2, the group E is trivial. In the remaining cases, E is a finite elementary abelian 2-group. In Section 10, we have described the E-torsor D explicitly for the various classical groups G(V ), but did not say how this was done. Our discussion of (H, ν) above has thus filled this gap, and we record the result in the following proposition for ease of reference.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

52

W. T. GAN, B. H. GROSS & D. PRASAD

Proposition 12.1. — 1. If V is symplectic, E = k × /k ×2 , and we have constructed an explicit bijection of E-spaces D ←→ k ×2 -orbits on nontrivial ψ : k → C× . 2. If V is hermitian of even dimension, E = k0× /Nk × , and we have constructed an explicit bijection of E-spaces D ←→ Nk × -orbits on nontrivial ψ : k/k0 → C 3. If V is skew-hermitian of even dimension, E = k0× /Nk × , and we have constructed an explicit bijection of E-spaces D ←→ Nk × -orbits on nontrivial ψ0 : k0 → C× 4. If V is orthogonal of even dimension and split by the quadratic algebra K, then E = NK × /k ×2 , and we have constructed an explicit bijection of E-spaces D ←→ SO(V )-orbits on non-isotropic lines L ⊂ V , with L⊥ split. We stress that the bijections constructed in Proposition 12.1 depend crucially on the form h−, −i on V .

13. Bessel and Fourier-Jacobi models for GL(n) The construction of the pair (H, ν) given in the previous section includes the case when k = k0 × k0 is the split quadratic algebra. In this case, the groups G(V ) and G(W ) are general linear groups, and it is useful to give a direct construction of (H, ν) in the context of general linear groups, rather than regarding them as unitary groups of hermitian or skew-hermitian spaces over k. We describe this direct construction in this section. We first give a brief explanation of how one translates from the context of unitary groups to that of general linear groups. The hermitian or skew-hermitian space V has the form V = V0 × V0∨ for a vector space V0 over k0 . Moreover, up to isomorphism, the hermitian form on V can be taken to be h(x, x∨ ), (y, y ∨ )i = (hx, y ∨ i, hy, x∨ i) ∈ k, whereas the skew-hermitian form on V can be taken to be h(x, x∨ ), (y, y ∨ )i = (hx, y ∨ i, −hy, x∨ i) ∈ k. Then, by restriction to V0 , one has an isomorphism G(V ) ∼ = GL(V0 ) of linear algebraic groups over k0 .

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

53

If W ⊂ V is a nondegenerate subspace, then W = W0 × W0∨ gives rise to W0 ⊂ V0 . If, further, W ⊥ is split, and X ⊂ W ⊥ is a maximal isotropic subspace, then X has the form X = X0 × Y0∨ ⊂ V0 × V0∨ with the natural pairing of X0 and Y0∨ equal to zero, so that X0 is contained in the kernel of Y0∨ . Writing the kernel of Y0∨ as X0 + W0 , we see that the isotropic space X determines a decomposition V0 = X0 + W0 + Y0 , with a natural perfect pairing between Y0 and Y0∨ . Then the parabolic subgroup P (X) stabilizing X in G(V ) is isomorphic to the parabolic subgroup of GL(V0 ) stabilizing the flag X0 ⊂ X0 + W0 ⊂ V0 . It is now easy to translate the construction of (H, ν) given in the previous section to the setting of W0 ⊂ V0 , and we simply describe the answer below. Bessel Models for GL(n) In this case, we start with a vector space V0 over k0 with a decomposition V0 = X0 + W0 + E0 + X0∨ , where E0 = hei is a line. Consider the (non-maximal) parabolic subgroup Q stabilizing the flag X0 ⊂ X0 + W0 + E0 ⊂ V0 . It has Levi subgroup L = GL(X0 ) × GL(W0 + E0 ) × GL(X0∨ ) and unipotent radical U sitting in the exact sequence:

0

/ Hom(X0∨ , X0 )

/U

/ Hom(X0∨ , W0 + E0 ) + Hom(W0 + E0 , X0 )

/ 0.

We may write the above exact sequence as:

0

/ X0 ⊗ X0

/U

/ X0 ⊗ (W0 + E0 ) + (W0∨ + E0∨ ) ⊗ X0

/ 0,

where E0∨ = hf i is the dual of E0 . Let `X0 : X0 −→ k0 be any nontrivial homomorphism, and let UX0 × UX0∨ be a maximal unipotent subgroup of GL(X0 ) × GL(X0∨ ) which fixes `X0 . On the other hand, let `W0 : (W0 + E0 ) + (W0∨ + E0∨ ) −→ k0 be a linear form which is trivial on W0 + W0∨ but nontrivial on E0 and E0∨ . Together, the homomorphisms `X0 and `W0 give a map ` = `X0 ⊗ `W0 : U −→ X0 ⊗ (W0 + E0 ) + (W0∨ + E0∨ ) ⊗ X0 −→ k0 .

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

54

W. T. GAN, B. H. GROSS & D. PRASAD

Since ` is fixed by UX0 × UX0∨ × GL(W0 ), we may extend ` trivially to this group. Thus, we may regard ` as a map on H = U o ((UX0 × UX0∨ ) × GL(W0 )). Choose any nontrivial additive character ψ0 of k0 and any generic character λ : UX0 × UX0∨ −→ S1 , which we may regard as a character of H. Then the representation ν of H is defined by ν = (ψ0 ◦ `) ⊗ λ. The pair (H, ν) depends only on the spaces W0 ⊂ V0 , up to conjugacy by GL(V0 ). This completes the construction of (H, ν) in the case when codimension of W0 in V0 is odd. Fourier-Jacobi models for GL(n) In this case, we consider a vector space V0 over k0 , together with a decomposition V0 = X0 + W0 + X0∨ . As before, let Q be the parabolic subgroup stabilizing the flag X0 ⊂ X0 + W0 ⊂ V0 . Thus Q has Levi subgroup L = GL(X0 ) × GL(W0 ) × GL(X0∨ ), and its unipotent radical U sits in the exact sequence, 0 → Hom(X0∨ , X0 ) → U → Hom(X0∨ , W0 ) + Hom(W0 , X0 ) → 0, in which Hom(X0∨ , X0 ) is central. The group U is completely described by the natural bilinear map Hom(X0∨ , W0 ) × Hom(W0 , X0 ) → Hom(X0∨ , X0 ). Indeed, given a bilinear map h−, −i : B × C → A, of vector groups, there is a natural central extension of B × C by A defined by a group structure on A × B × C given by (a1 , b1 , c1 )(a2 , b2 , c2 ) = (a1 + a2 + hb1 , c2 i, b1 + b2 , c1 + c2 ). Given a linear map `X0 : X0 → k0 , let UX0 × UX0∨ ⊂ GL(X0 ) × GL(X0∨ ), be a maximal unipotent subgroup fixing `X0 . Let λ : UX0 × UX0∨ −→ S1

ASTÉRISQUE 346

55

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

be a generic character, which we may regard as a character of H = U o (UX0 × UX0∨ × GL(W0 )) via projection onto UX0 × UX0∨ . On the other hand, the homomorphism `X0 allows one to define a homomorphism from U to the Heisenberg group H(W0 + W0∨ ): 0 −−−−→ X0 ⊗ X0 −−−−→   y 0 −−−−→

k0

U   y

−−−−→ W0∨ ⊗ X0 + X0 ⊗ W0 −−−−→ 0   y

−−−−→ H(W0∨ + W0 ) −−−−→

W0∨ + W0

−−−−→ 0,

which is clearly equivariant under the action of UX0 × UX0∨ × GL(W0 ). Thus, given any nontrivial additive character ψ0 of k0 , we may consider the unique irreducible representation of H(W0∨ + W0 ) with central character ψ0 , and regard it as a representation of U using the above diagram. This representation can be extended trivially to UX0 × UX0∨ , and is realized naturally on the space S (W0 ) of SchwarzBruhat functions on W0 . For any character µ : k0× −→ C× , one then obtains a Weil representation ωψ0 ,µ of H = (GL(W0 ) × UX0 × UX0∨ ) n U. on S (W0 ), where the action of GL(W0 ) is given by (g · f )(w) = µ(det(g)) · f (g −1 · w). When dim X0 > 0, the representation νψ,µ of H is then given by νψ0 ,µ = ωψ0 ,µ ⊗ λ. When dim X0 = 0, we have W0 = V0 and we take the representation νψ0 ,µ of H = GL(W0 ) to be the representation ωψ0 ,µ of GL(W0 ) on S (W0 ) defined above. In either case, the isomorphism class of ωψ0 ,µ is independent of ψ0 and the pair (H, νψ0 ,µ ) is independent of ψ0 , up to conjugacy in GL(V0 ). This completes the definition of (H, ν) when the codimension of W0 in V0 is even. This concludes our direct construction of the pair (H, ν) for general linear groups. 14. Restriction Problems and Multiplicity One Theorems We are now ready to formulate the local restriction problems studied in this paper. Let W ⊂ V be as in §12, so that G = G(V ) × G(W ) contains the subgroup H = N o G(W ). We have defined a representation ν of H (or its double cover), which may depend on some auxiliary data such as ψ or µ. Let π = πV πW be an irreducible representation of G (or an appropriate double cover). Then the restriction problem of interest is to determine dimC HomH (π ⊗ ν, C). More precisely, we have:

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

56

W. T. GAN, B. H. GROSS & D. PRASAD

1. In the orthogonal or hermitian cases, the representation ν of H depends only on W ⊂ V and so we set d(π) = dimC HomH (π ⊗ ν, C) = dimC HomH (π, ν). In the literature, this restriction problem is usually referred to as a problem about the existence of Bessel models, and for an irreducible representation π = ∨ πV  πW of G, the space HomH (π, ν) usually called the space of πW -Bessel models of πV . 2. In the symplectic case, the representation νψ is a representation of the double ‹ = N o Sp(W f cover H ) and depends on a nontrivial additive character ψ of k = k0 up to the action of k ×2 . In this case, for the above Hom space to be nonzero, the representation π = πV  πW must be a genuine representation ‹ Hence, we have to take an irreducible representation when restricted to H. π e = πV  π eW

of

f Sp(V ) × Sp(W )

π e=π eV  πW In this case, we set

of

f ) × Sp(W ). Sp(V

or

d(˜ π , ψ) = dimC HomH π ⊗ ν ψ , C). e (e In the literature, this restriction problem is usually referred to as one about Fourier-Jacobi models, and the space HomH π ⊗ ν ψ , C) usually called the e (e ∨ (πW , ψ)-Fourier-Jacobi models of πV . 3. In the skew-hermitian case, the representation νψ0 ,µ of H depends on a nontrivial additive character ψ0 of k0 , up to the action of Nk × , and also on the choice of a character µ of k × whose restriction to k0× is the quadratic character associated to k/k0 . In this case, we set d(π, µ, ψ0 ) = dimC HomH (π ⊗ ν ψ0 ,µ , C). In the literature, this restriction problem is usually referred to as a problem about the existence of Fourier-Jacobi models in the context of unitary groups, ∨ , ψ0 , µ)-Fourier-Jacobi and the space HomH (e π ⊗ ν ψ0 ,µ , C) usually called the (πW models of πV . We remark that in the orthogonal and hermitian cases, since ν is 1-dimensional and unitary, one has: HomH (π ⊗ ν, C) ∼ = HomH (π ⊗ ν ∨ , C) ∼ = HomH (π, ν). In the symplectic and skew-hermitian cases over non-archimedean fields, the same assertion holds, even though ν is infinite-dimensional. However, over archimedean fields, it is only clear to us that: HomH (π, ν) ⊆ HomH (π ⊗ ν ∨ , C) ∼ = HomH (π ⊗ ν, C). The difficulty arises in the subtlety of duality in the theory of topological vector spaces. In any case, we work with HomH (π ⊗ ν, C) since this is the space which naturally

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

57

arises in the global setting. We should also mention that, over archimedean fields, the tensor product π ⊗ ν refers to the natural completed tensor product of the two spaces (which are nuclear Fréchet spaces) and Hom(−, C) refers to continuous linear functionals. For a discussion of these archimedean issues, see [3, Appendix A]. A basic conjecture in the subject is the assertion that d(π) ≤ 1 in the various cases. Recently, there has been much progress in the most basic cases where dim W ⊥ = 0 or 1. We describe these in the following theorem. Theorem 14.1. — Assume that k has characteristic zero and dim W ⊥ = 0 or 1. (i) In the orthogonal case, with G = O(V ) × O(W ) or SO(V ) × SO(W ), we have d(π) ≤ 1. (ii) In the hermitian case (including the case when k = k0 × k0 ), we have d(π) ≤ 1. (iii) In the symplectic case, suppose that k is non-archimedean. Then we have d(π, ψ) ≤ 1. (iv) In the skew-hermitian case (including the case k = k0 × k0 ), suppose that k is non-archimedean. Then we have d(π, µ, ψ0 ) ≤ 1. Proof. — For the groups O(V ) × O(W ) in (i) and U(V ) × U(W ) in (ii), the results are due to Aizenbud-Gourevitch-Rallis-Schiffmann [4] in the p-adic case and to SunZhu [66] and Aizenbud-Gourevitch [3] in the archimedean case. This is extended to G = SO(V ) × SO(W ) by Waldspurger [76] in the non-archimedean case and by Sun-Zhu [67] in the archimedean case. The cases (iii) and (iv) are due to Sun [64] in the non-archimedean case (the archimedean case seems to be still open). Remarks. — For the group GLn × GLn−1 , the above multiplicity one result has been extended to the case when k has characteristic p by Aizenbud-Avni-Gourevitch [2].

15. Uniqueness of Bessel Models In this section, we show that if k is non-archimedean, the multiplicity one theorem for the general Bessel models can be deduced from Theorem 14.1(i) and (ii) in the orthogonal and hermitian cases. We remind the reader that the case k = k0 × k0 is included in our discussion. In particular, the results we state below are valid in this case as well, though we frequently write our proofs only for k a field, and leave the adaptation to the case k = k0 × k0 to the reader.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

58

W. T. GAN, B. H. GROSS & D. PRASAD

Thus, we consider the case when W ⊂ V are orthogonal or hermitian spaces of odd codimension. Then we have W ⊥ = X + X∨ + E where E = k · e is a non-isotropic line and X = hv1 , v2 , · · · , vn i is an isotropic subspace with dim X = n > 0 and dual basis {vi∨ } of X ∨ . With G = G(V ) × G(W ) and H = N · G(W ), we would like to show that dim HomH (πV ⊗ πW , ν) ≤ 1 for any irreducible representation πV  πW of G. Let E− = k · f denote the rank 1 space equipped with a form which is the negative of that on E, so that E + E − is a split rank 2 space. The two isotropic lines in E + E − are spanned by 1 ∨ · (e − f ). vn+1 = e + f and vn+1 = 2 · he, ei Now consider the space W 0 = V ⊕ E− which contains V with codimension 1 and isotropic subspaces Y = X + k · vn+1 = hv1 , · · · , vn+1 i and ∨ ∨ Y ∨ = X ∨ + k · vn+1 = hv1∨ , · · · , vn+1 i.

Hence we have W 0 = Y + Y ∨ + W. Let P = P (Y ) be the parabolic subgroup of G(W 0 ) stabilizing Y and let M be its Levi subgroup stabilizing Y and Y ∨ , so that M∼ = GL(Y ) × G(W ). Let τ be an irreducible supercuspidal representation of GL(Y ) and πW an irreducible smooth representation of G(W ) and let G(W 0 )

I(τ, πW ) = IndP

(τ  πW )

be the (unnormalized) induced representation of G(W 0 ) from the representation τ  πW of P . Our goal is to prove the following theorem:

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

59

Theorem 15.1. — Assume that k is non-archimedean. With the notations as above, we have HomG(V ) (I(τ, πW ) ⊗ πV , C) = HomH (πV ⊗ πW , ν) as long as πV∨ does not belong to the Bernstein component of G(V ) associated to (GL(Y 0 )×M 0 , τ ⊗µ0 ), where Y 0 ⊂ V is isotropic of dimension equal to dim Y with V = ∨ Y 0 + Y 0 + V 0 , M 0 is a Levi subgroup of G(V 0 ) and µ0 is any irreducible supercuspidal representation of M 0 . Proof. — We assume that k is a field in the proof and calculate the restriction of Π := I(τ, πW ) to G(V ) by Mackey’s orbit method. For this, we begin by observing that G(V ) has at most two orbits on the flag variety G(W 0 )/P (Y ) consisting of: 1. (n + 1)-dimensional isotropic subspaces of W 0 which are contained in V ; these exist if and only if W is isotropic, in which case if Y 0 is a representative of this closed orbit, then its stabilizer in G(V ) is the parabolic subgroup PV (Y 0 ) = P (Y 0 ) ∩ G(V ); 2. (n + 1)-dimensional isotropic subspaces of W 0 which are not contained in V ; a representative of this open orbit is the space Y and its stabilizer in G(V ) is the subgroup Q = P (Y ) ∩ G(V ). By Mackey theory, this gives a filtration on the restriction of Π to G(V ) as follows: G(V )

0 −−−−→ indQ

G(V )

(τ ⊗ πW )|Q −−−−→ Π|G(V ) −−−−→ IndPV (Y 0 ) τ ⊗ πW |G(V 0 ) −−−−→ 0,

where the induction functors here are unnormalized and where ind denotes the induction with compact support. Denoting the above short exact sequence by 0 −−−−→ A −−−−→ B −−−−→ C −−−−→ 0, for simplicity, we have an exact sequence: 0 −−−−→ HomG(V ) (C, πV∨ ) −−−−→ HomG(V ) (B, πV∨ ) −−−−→ HomG(V ) (A, πV∨ ) −−−−→ Ext1G(V ) (C, πV∨ ). By our assumption on πV∨ , we have: G(V )

HomG(V ) (IndPV (Y 0 ) (τ ⊗ µ), πV∨ ) = 0 and G(V )

Ext1G(V ) (IndPV (Y 0 ) (τ ⊗ µ), πV∨ ) = 0 for any smooth (not necessarily of finite length) representation µ of G(V 0 ). Thus, HomG(V ) (C, πV∨ ) = 0 = Ext1 (C, πV∨ ) and we have G(V )

HomG(V ) (indQ

(τ ⊗ πW )|Q , πV∨ ) = HomG(V ) (Π, πV∨ ).

It thus suffices to analyze the representations of G(V ) which appear on the open orbit. For this, we need to determine the group Q = P (Y ) ∩ G(V ) as a subgroup of G(V ) and P (Y ).

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

60

W. T. GAN, B. H. GROSS & D. PRASAD

Recall that W 0 = Y ⊕W ⊕Y ∨ , and V is the codimension 1 subspace X ⊕W ⊕X ∨ ⊕E which is the orthogonal complement of f . It is not difficult to see that as a subgroup of G(V ), Q = G(V ) ∩ P (Y ) ⊂ PV (X). Indeed, if g ∈ Q, then g fixes f and stabilizes Y , and we need to show that it stabilizes X. If x ∈ X, it suffices to show that hg · x , e − f i = 0. But hg · x , e − f i = hx , g −1 · (e − f )i = hx , g −1 · vn+1 − 2f i = 0, as desired. Now we claim that as a subgroup of PV (X), Q = (GL(X) × G(W )) n NV (X), where NV (X) is the unipotent radical of PV (X). To see this, given an element h ∈ PV (X), note that h ∈ Q if and only if h · vn+1 ∈ Y,

or equivalently h · e − e ∈ Y.

We may write h · e = λ · e + w + x,

with w ∈ W and x ∈ X.

Then we see that h · e − e ∈ Y if and only if λ = 1 and w = 0, so that h fixes e modulo X and hence stabilizes W modulo X, in which case h ∈ (GL(X) × G(W )) n NV (X), as desired. Since we are restricting the representation τ  πW of P (Y ) to the subgroup Q, we also need to know how Q sits in P (Y ). For this, note the following lemma. Lemma 15.2. — The natural projection pr : P (Y )  GL(Y ) × G(W ) induces the following commutative diagram with exact rows, where the vertical arrows are inclusions: 0 −−−−→

N (Y ) x  

pr

−−−−→ P (Y ) −−−−→ GL(Y ) × G(W ) −−−−→ 0 x x    

0 −−−−→ N (Y ) ∩ Q −−−−→



Q x  

−−−−→

R × G(W ) x  

−−−−→ 0

0 −−−−→ N (Y ) ∩ Q −−−−→ NV (X) −−−−→ Hom(k · vn+1 , X) −−−−→ 0. Here R ⊂ GL(Y ) is the mirabolic subgroup which stabilizes the codimension one subspace X ⊂ Y and fixes vn+1 modulo X and Hom(k · vn+1 , X) is the unipotent radical of R. Proof. — The projection pr is given by the action of P (Y ) on Y × (Y + W )/Y . Consider the restriction of pr to the subgroup Q = G(V ) ∩ P (Y ) = (GL(X) × G(W )) n NV (X) of P (Y ). We note:

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

61

(i) the subgroup GL(X) × G(W ) maps isomorphically to its image in GL(Y ) × G(W ), and its image is precisely the Levi subgroup GL(X) × G(W ) of R × G(W ). This is clear. (ii) the kernel N (Y ) ∩ Q of pr|Q is contained in NV (X). To see this, suppose that n ∈ N (Y ) ∩ Q. Then n ∈ G(V ) (since Q ⊂ G(V )), so that n · f = f . To show that n ∈ NV (X), we need to show that n acts trivially on X and acts trivially on W ⊕ E modulo X. Now, as an element of N (Y ), n acts trivially on Y and acts trivially on W modulo Y . Thus, n certainly acts trivially on X ⊂ Y , and for w ∈ W , n · w − w ∈ Y ∩ V = X. It remains to show that n acts trivially on E modulo X, i.e. that n·e−e ∈ X = Y ∩V . Since n · e − e lies in V , it suffices to show that n · e − e lies in Y . But we have: n · e − e = n · (e + f ) − (ef ) = n · vn+1 − vn+1 ∈ Y. This proves the N (Y ) ∩ Q ⊂ NV (X). (iii) the projection pr induces an isomorphism: ∼ Hom(k · vn+1 , X). NV (X)/(N (Y ) ∩ Q) = Indeed, if n ∈ NV (X), then n fixes X and fixes W modulo X. Moreover, since n fixes f (as it is in G(V )) and fixes e modulo X, we have n · vn+1 − vn+1 = n · (e + f ) − (e + f ) ∈ X. This shows that pr(n) lies in the unipotent radical Hom(k · vn+1 , X) of R. Indeed, pr(n)(vn+1 ) = n · e − e. It remains to show that pr|NV (X) is surjective onto Hom(kvn+1 , X). For any x ∈ X, let nx ∈ NV (X) be the element which fixes X and W and such that nx (e) = e + x. Then pr(nx )(vn+1 ) = x, as desired. In view of the above, we see that the image of Q under pr is precisely R × G(W ) and the image of NV (X) is the unipotent radical of R. The lemma is proved. By the lemma, one has: (τ  πW )|Q = τ |R  πW . By a well-known result of Gelfand-Kazhdan, since τ is supercuspidal, one knows that τ |R ∼ = indR χ U

where U is the unipotent radical of the Borel subgroup of GL(Y ) stabilizing the flag hv1 i ⊂ hv1 , v2 i ⊂ · · · ⊂ hv1 , · · · , vn+1 i = Y, and χ is any generic character of U . Now it is clear from the lemma that the pre-image of U × G(W ) in Q is precisely the subgroup H = (UX × G(W )) n NV (X) ⊂ G(V ), where UX is the unipotent subgroup of GL(X) stabilizing the flag hv1 i ⊂ hv1 , v2 i ⊂ · · · ⊂ hv1 , · · · , vn i = X.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

62

W. T. GAN, B. H. GROSS & D. PRASAD

Further, the representation χ  πW of U × G(W ) pulls back to the representation ν ∨ ⊗ πW of H. Indeed, the pre-image of U in Q is the subgroup UX n NV (X), and the pullback of χ is in general position when restricted to UX . Moreover, when restricted to NV (X), the pullback of χ is nontrivial and fixed by UX × G(W ). Hence, by induction in stages, we conclude that G(V )

indQ

G(V ) (τ ⊗ πW )|Q ∼ = indH πW ⊗ ν ∨ .

Thus, by dualizing and Frobenius reciprocity, one has HomG(V ) (I(τ, πW ), πV∨ ) ∼ = HomH (πV ⊗ πW , ν). This completes the proof of the theorem. Corollary 15.3. — In the orthogonal or hermitian cases over a non-archimedean k, with W ⊂ V of odd codimension, we have dimC HomH (π, ν) ≤ 1 for any irreducible representation π of G = G(V ) × G(W ). Proof. — To apply Theorem 15.1, choose a supercuspidal representation τ which does not belong to the Bernstein components in Theorem 15.1. Then, replacing τ by its twist by an unramified character, we may assume that the associated induced representation I(τ, πW ) is irreducible; this is possible by a result of Waldspurger [61]. Then, by Theorem 15.1, the corollary is reduced to Theorem 14.1. Remarks. — In the archimedean case, a recent paper of Jiang-Sun-Zhu [37] adapted the proof of Theorem 15.1 to show the containment HomH (πV ⊗ πW , ν) ⊂ HomG(V ) (I(τ, πW ) ⊗ πV , C). Namely, to each element on the left hand side, [37] constructs an associated element on the right hand side, using an explicit integral. This is enough to deduce the multiplicity one result of Corollary 15.3 from the results of [66], [67] and [3]. 16. Uniqueness of Fourier-Jacobi Models In this section, we continue with the assumption that k is non-archimedean and our goal is to establish the analog of Theorem 15.1 in the symplectic and skew-hermitian cases, which will imply that d(π, ψ) ≤ 1. Before coming to the analogous result, which is given in Theorem 16.1, we need to recall certain structural results about parabolic induction for the metaplectic groups. ‹ in Sp(W f Recall that if W is a symplectic space, then a parabolic subgroup P ) is nothing but the inverse image of a parabolic P in Sp(W ). It is known that the metaplectic covering splits (uniquely) over unipotent subgroups, so for a Levi decomposition P = M · N , it makes sense to speak of the corresponding Levi decomposition ‹= M f · N in Sp(W f P ).

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

63

Furthermore, we note that for a maximal parabolic subgroup P (X) of Sp(W ) with Levi subgroup of the form M = GL(X) × Sp(W0 ) in Sp(W ), Ä ä f = GL(X) › f 0 ) /∆µ2 M × Sp(W › where GL(X) is a certain two-fold cover of GL(X) defined as follows. As a set, we write › GL(X) = GL(X) × {±1}, and the multiplication is given by (g1 , 1 ) · (g2 , 2 ) = (g1 g2 , 1 2 · (det g1 , det g2 )), where (−, −) denotes the Hilbert symbol on k × with values in {±1}. › The two-fold cover GL(X) has a natural genuine 1-dimensional character › χψ : GL(X) −→ C× defined as follows. The determinant map gives rise to a natural group homomorphism › › top X) = GL(1). › det : GL(X) −→ GL(∧ › On the other hand, one has a genuine character on GL(1) defined by (a, ) 7→  · γ(a, ψ)−1 , where γ(a, ψ) = γ(ψa )/γ(ψ) and γ(ψ) is an 8-th root of unity associated to ψ by Weil. Composing this character › with det gives the desired genuine character χψ on GL(X), which satisfies: χ2ψ (g, ) = (det(g), −1). Thus, there is a bijection between the set of irreducible representations of GL(X) and › the set of genuine representations of GL(X), given simply by τ 7→ τ˜ψ = τ ⊗ χψ . Note that this bijection depends on the additive character ψ of k. Now associated to f 0 ), one has the representation a representation τ of GL(X) and π0 of Sp(W τ˜ψ  π0

f. of M

Then one can consider the (unnormalized) induced representation Iψ (τ, π0 ) = Ind

‹p(W ) S e P

(˜ τψ  π0 ).

Here is the analog of Theorem 15.1 in the symplectic case. Theorem 16.1. — Consider W = X ⊕ W0 ⊕ X ∨ with X 6= 0 and fix the additive character ψ of the non-archimedean local field k. Let – τ be a supercuspidal representation of GL(X); f 0 ); – π0 be a genuine representation of Sp(W – π be an irreducible representation of Sp(W ),

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

64

W. T. GAN, B. H. GROSS & D. PRASAD

f and consider the (unnormalized) induced representation Iψ (τ, π0 ) of Sp(W ). Assume that π ∨ does not belong to the Bernstein component associated to (GL(X)×M, τ µ), where M is any Levi subgroup of Sp(W0 ) and µ is any supercuspidal representation of M . Then HomS‹p(W ) (Iψ (τ, π0 ) ⊗ π, ωW,ψ ) ∼ = HomH (π ⊗ π0 , νW,W0 ,ψ ). Proof. — We shall compute ∨ HomS‹p(W ) (Iψ (τ, π0 ) ⊗ ωW,ψ , π ∨ ).

Let P (X) = M (X) · N (X) be the parabolic subgroup in Sp(W ) stabilizing the subspace X, so that Ä ä f(X) ∼ › f 0 ) /∆µ2 . M × Sp(W = GL(X) ∨ ‹(X)-module; The Weil representation ωW,ψ = ωW,ψ has a convenient description as a P ∨ this is the so-called mixed model of the Weil representation. This model of ωW,ψ is realized on the space ∨ S(X ∨ ) ⊗ ωW 0 ,ψ ∨ of Schwartz-Bruhat functions on X ∨ valued in ωW . In particular, evaluation at 0 0 ,ψ ‹ gives a P (X)-equivariant map ∨ ∨ ev : ωW,ψ −→ χψ |detX |1/2  ωW , 0 ,ψ

where N (X) acts trivially on the target space. In fact, this map is the projection of ∨ onto its space of N (X)-coinvariants. ωW,ψ On the other hand, to determine the kernel of the map ev, note that GL(X) acts transitively on the nonzero elements of X ∨ . Recall that we have fixed a basis {v1 , · · · , vn } of X in the definition of the data (H, νψ ), with dual basis {v1∨ , · · · , vn∨ } of X ∨ . Let R be the stabilizer of vn∨ in GL(X), so that R is a mirabolic subgroup of GL(X). Let Q = (R × Sp(W0 )) · N (X) ⊂ P (X) ‹(X) is so that its inverse image in P Ä ä ‹ = (R e × Sp(W f 0 ))/∆µ2 · N (X) ⊂ P ‹(X). Q ‹(X)-modules: Then one deduces the following short exact sequence of P

0

/ indPe(X) χ |detX |1/2  ω ∨ W0 ,ψ ψ e Q

/ ω∨ W,ψ

ev

/ χ |detX |1/2  ω ∨ W0 ,ψ ψ

/ 0,

‹ on ω ∨ where the compact induction functor ind is unnormalized and the action of Q W0 ,ψ f 0 ) · N (X) with respect to the character ψ. is via the Weil representation of Sp(W

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

65

f Tensoring the above short exact sequence by τ˜ψ  π0 and then inducing to Sp(W ), one gets a short exact sequence of Sp(W )-modules: 0   y Sp(W )

indQ

∨ (|detX |1/2 · τ |R ⊗ (π0 ⊗ ωW )) = A 0 ,ψ   y ∨ Iψ (τ, π0 ) ⊗ ωW,ψ =B   y

Sp(W )

∨ IndP (X) (τ · |detX |1/2 ⊗ (π0 ⊗ ωW )) = C 0 ,ψ   y

0. By our assumption on π, HomSp(W ) (C, π ∨ ) = 0

and HomSp(W ) (B, π ∨ ) = HomSp(W ) (A, π ∨ ).

Moreover since τ is supercuspidal, by a well-known result of Gelfand-Kazhdan, one has τ |R ∼ = indR U χ, where U is the unipotent radical of the Borel subgroup of GL(X) stabilizing the flag hv1 i ⊂ hv1 , v2 i ⊂ · · · ⊂ hv1 , · · · , vn i = X and χ is any generic character of U . Observing that H = (U × Sp(W0 )) · N (X), we conclude that Sp(W )

A = indH

(π0 ⊗ νψ∨ ).

Therefore, the desired result follows by Frobenius reciprocity. Corollary 16.2. — In the symplectic case over a non-archimedean local field, we have dimC HomH (π ⊗ ν ψ , C) ≤ 1 for any irreducible representation π of G = G(V ) × G(W ). One can prove an analog of Theorem 16.1 in the skew-hermitian case, including the case when k = k0 × k0 , and deduce the following corollary; we omit the details. Corollary 16.3. — In the skew-hermitian case over a non-archimedean k, with W ⊂ V of even codimension, we have dimC HomH (π ⊗ ν ψ0 ,µ , C) ≤ 1 for any irreducible representation π of G = G(V ) × G(W ).

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

66

W. T. GAN, B. H. GROSS & D. PRASAD

17. Local Conjectures In this section, we propose a conjecture for the restriction problem formulated in Section 14. Recall that we have a pair of spaces W ⊂ V and we are considering the restriction of irreducible representations π = πV  πW of G = G(V ) × G(W ) to the subgroup H = N · G(W ) ⊂ G. Recall also that, with auxiliary data if necessary, we ‹ which have defined a unitary representation ν of H (or sometimes its double cover H), has dimension 1 when W ⊂ V are orthogonal or hermitian, and has Gelfand-Kirillov dimensional 1/2 · dim(W/k0 ) when W ⊂ V are symplectic or skew-hermitian. Then we are interested in d(π) = dimC HomH (π ⊗ ν, C), which is known to be ≤ 1 in almost all cases. In this section, we shall give precise criterion for this Hom space to be nonzero, in terms of the Langlands-Vogan parametrization of irreducible representations of G. We first note the following conjecture, which has been called multiplicity one in L-packets. Recall that a pure inner form G0 = G(V 0 ) × G(W 0 ) of the group G is relevant if the space W 0 embeds as a non-degenerate subspace of V 0 , with orthogonal complement isomorphic to W ⊥ . In this case, one can define a subgroup H 0 = G(W 0 ) n N 0 ⊂ G0 . Conjecture 17.1. — There is a unique representation π of a relevant pure inner form G0 = G(V 0 ) × G(W 0 ) in each generic Vogan L-packet Πϕ of G which satisfies HomH 0 (π ⊗ ν, C) 6= 0. In the papers [74, 75] and [53], Waldspurger and Mœglin-Waldspurger have made substantial progress towards this conjecture. Namely, assuming certain natural and expected properties of the characters of representations in a Vogan L-packet, they have shown that the above conjecture holds in the special orthogonal case. There is no doubt that these methods will give the same result in the hermitian case. We also note that when k = k0 × k0 , we have G ∼ = GL(V0 ) × GL(W0 ) so that the Vogan packets are all singletons. In this case, the above conjecture simply asserts that HomH (π ⊗ ν, C) 6= 0 for any irreducible generic representation π of G. In this case, we have: Theorem 17.2. — (i) If k = k0 × k0 , then Conjecture 17.1 holds when dim W ⊥ = 0 or 1. (ii) If k = k0 × k0 is non-archimedean, then Conjecture 17.1 holds in general. Proof. — When dim W ⊥ = 0 or 1, Conjecture 17.1 is an immediate consequence of the local Rankin-Selberg theory of Jacquet, Piatetski-Shapiro and Shalika ([34] and [36]). Indeed, the local Rankin-Selberg integral gives a nonzero element of HomH (π, ν). When k is non-archimedean, the general case then follows from Theorems 15.1 and Theorem 16.1. In each of the remaining cases, we will make the above conjecture more precise by specifying a canonical character χ of the component group Aϕ . The character χ

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

67

depends on the choice of a generic character θ of G (used to normalize the LanglandsVogan parametrization) and on the additional data needed to define the representation ν when  = −1. We then conjecture that the representation π in Conjecture 17.1 has parameter π = π(ϕ, χ) in the Vogan correspondence J(θ). We treat the various cases separately. G = SO(V ) × SO(W ), dim W ⊥ odd Here the character θ is determined by the pair of orthogonal spaces W ⊂ V . In view of Proposition 12.1, specifying θ amounts to giving a non-isotropic line L in the even orthogonal space (with L⊥ split), and we simply take the line L to have discriminant equal to the discriminant of the odd space. The representation ν is also canonical. The L-packet Πϕ is determined by a parameter ϕ : W D(k) −→ Sp(M ) × O(N ) with dim N even. We define χ = χN × χM : AM × A+ N −→ h±1i, where the characters χN and χM were defined in §6. G = U(V ) × U(W ), dim W ⊥ odd Here, in view of Proposition 12.1, we need to choose a nontrivial character ψ : k/k0 −→ S1 up to the action of Nk × in order to define a generic character θ0 of the even unitary group. If δ is the discriminant of the odd hermitian space, then we define θ(x) = θ0 (−2 · δ · x) and use θ to fix the Vogan parametrization for the even unitary group. Note that θ is simply the generic character of the even unitary group determined by the additive character ψ−2·δ (x) = ψ(−2 · δ · x). The representation ν is canonical. The L-packet is determined by a parameter ϕ : W D(k) −→ GL(M ) × GL(N ) with M conjugate-symplectic of even dimension and N conjugate-orthogonal of odd dimension. We define: χ = χN × χM : AM × AN −→ h±1i, using the character ψ to calculate the local epsilon factors which intervene in the definition of χ.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

68

W. T. GAN, B. H. GROSS & D. PRASAD

f ) × Sp(W ) or Sp(V ) × Sp(W f G = Sp(V ), dim W ⊥ even Here we need to choose a nontrivial additive character ψ : k → S1 to define a generic character θ of the symplectic group, the notion of Vogan parameters for the metaplectic group and the representation νψ of H. The L-packet is determined by a parameter ϕ : W D(k) −→ Sp(M ) × SO(N ) with dim N odd. Let N1 = N ⊕ C be the corresponding orthogonal representation of even dimension and define χ = χN1 × χM : AM × A+ N1 −→ h±1i. The group Aϕ is a subgroup of index 1 or 2 in AM × A+ N1 and we take the restriction of χ to this subgroup. G = U(V ) × U(W ), W ⊂ V skew-hermitian and dim W ≡ dim V ≡ 1 mod 2 Here there is a unique orbit of generic character θ on the quasi-split group U(V ) × U(W ). On the other hand, we need to choose ψ0 : k0 → S1 up to Nk × and µ : k × /Nk × −→ C× , nontrivial on k0× , to define the representation νψ0 ,µ of H. Let e be the discriminant of V and W which is a nonzero element of trace 0 in k, well-defined up to Nk × . Let ψ(x) = ψ0 (Tr(ex)) which is a nontrivial character of k/k0 , well-defined up to Nk × . The L-packet has parameter ϕ : W D(k) −→ GL(M ) × GL(N ) with M and N conjugate-orthogonal representations of odd dimension. We define: χ = χN × χM (µ−1 ) = χN (µ−1 ) × χM : AM × AN −→ h±1i, using ψ to calculate the local epsilon actors which intervene in the definition of χ. Here, M (µ−1 ) and N (µ−1 ) are the twist of M and N by the character µ−1 . Note that the representations M (µ−1 ) and N (µ−1 ) are conjugate-symplectic.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

69

G = U(V ) × U(W ), W ⊂ V skew-hermitian, dim W ≡ dim V ≡ 0 mod 2 In this case, we must choose ψ0 : k0 → S1 to define θ for both groups, and µ : k × /Nk × → C× , nontrivial on k0× , to define ν = νψ0 ,µ . The parameter of an L-packet is ϕ : W D(k) −→ GL(M ) × GL(N ) with M and N conjugate-symplectic representations of even dimension. We define χ = χN × χM (µ−1 ) = χN (µ−1 ) × χM : AM × AN −→ h±1i. Here, the twisted representations M (µ−1 ) and N (µ−1 ) are conjugate-orthogonal. Since the representations both have even dimension, the values of χ are independent of the choice of ψ used to define the epsilon factors. Now we have: Conjecture 17.3. — Having fixed the Langlands-Vogan parametrization for the group G and its pure inner forms in the various cases above, the unique representation π in a generic Vogan packet Πϕ which satisfies HomH (π ⊗ ν, C) 6= 0 has parameters π = π(ϕ, χ) where χ is as defined above. Note that the character χ defined above satisfies: χ(−1, −1) = 1, so that π(ϕ, χ) is a representation of a relevant pure inner form G0 of G. In the non-archimedean case, the sign χ(−1, 1) = χ(1, −1) determines which relevant pure inner form acts on π(ϕ, χ). 18. Compatibilities of local conjectures In this section, we verify that the precise conjecture 17.3 is independent of: 1. the bijection J(θ) : Πϕ ↔ Hom(Aϕ , ±1) given by the choice of a generic character θ of G; 2. the scaling of the form h−, −i on V and hence W , which does not change the groups G and H; 3. the data needed to define the representation ν of H. This serves as a check on the internal consistency of the conjecture. Again, we consider the various cases separately. In the orthogonal case, the generic character θ and the representation ν of H are determined by the pair of spaces W ⊂ V and are unchanged if the bilinear form on V is scaled by k × . The character χ = χN × χM depends only on the Langlands

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

70

W. T. GAN, B. H. GROSS & D. PRASAD

parameter ϕ : W D(k) −→ Sp(M ) × O(N ). So our conjecture is internally consistent in this case, as there is nothing to check. In the hermitian case, both the generic character θ and the character χ = χN × χM depend on a choice of nontrivial ψ : k/k0 → S1 , up to multiplication by Nk × , while the representation ν is determined by the spaces W ⊂ V . If we scale the hermitian form on W ⊂ V by an element of k0× , the generic character θ, the representation ν of H and the character χ of the component group are unchanged. To see the dependence of our conjecture on the choice of ψ, suppose that t represents the nontrivial coset of k0 /Nk × and let θt be the generic character associated to ψt (x) = ψ(tx). For (a, b) ∈ AM × AN , we have χt (a, b) = (M a ⊗ N, ψ0t ) · (M ⊗ N b , ψt ) = det M a (t) · (M a ⊗ N, ψ0 ) · (M ⊗ N b , ψ) a

= (−1)dim M · χ(a, b) = η(a) · χ(a, b) Here we have used the facts that M is conjugate-symplectic of even dimension and N is conjugate-orthogonal of odd dimension. Now if the parameter of π under J(θ) is (ϕ, χ), then its parameter under J(θt ) is (ϕ, χ · η) = (ϕ, χt ), according to the desiderata in §10. Hence our conjecture is independent of the choice of ψ in the hermitian case. f In the symplectic case, we will discuss representations of G = Sp(W ) × Sp(V ); the f case of representations of Sp(W ) × Sp(V ) is similar. In this case, we used the choice of an additive character ψ : k → S1 , up to multiplication by k ×2 , to f (i) define the notion of L-parameters M for representations of Sp(W ); e f (ii) define a generic character θ for Sp(V ) and a generic character θ for Sp(W ); (iii) define the representation ν = νψ for H. Note, however, that the character χ of the component group AM is independent of the choice of ψ. Suppose that under the ψ-parametrization, the parameter (M, N, χ) corresponds to the representation π e of G, so that our conjecture predicts that HomH (e π , νψ ) 6= 0. Now replace the character ψ by ψc for c ∈ k × /k ×2 and let π e0 be the representation of G corresponding to (M, N, χ) under the ψc -parametrization. By our construction of the Vogan parametrization for metaplectic groups, it is easy to see that π e0 is isomorphic to c the conjugated representation π e . Thus, our conjecture for the character ψc predicts that HomH (e π c , νψc ) 6= 0 and hence HomH (e π , νψc c ) 6= 0. Since νψc = νψc , our conjecture is internally consistent with respect to changing ψ. Note that the use of ψ in (i) and (ii) above concerns the Vogan parametrization, whereas its use in (iii) concerns the restriction problem in representation theory. Hence

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

71

there is no reason why one needs to use the same character ψ for these two different purposes. Suppose that one continues to use ψ for the Vogan parametrization in (i) and (ii), but uses the character ψc (x) = ψ(cx) to define the representation νψc of H. Then for a given Vogan packet of G with ψ-parameter ϕ, one can ask which representation π ∈ Πϕ satisfies HomH (π, νψc ) 6= 0. This can be answered using Conjecture 17.3, together with Conjecture 11.3 from §11. We have: Proposition 18.1. — Assume the conjectures 11.3 and 17.3. Let ϕ : W D(k) −→ Sp(M ) × SO(N ) f be a generic Langlands parameter for Sp(W )×Sp(V ) relative to the nontrivial additive character ψ of k. Let Nc = N (c) ⊕ C for c ∈ k × /k ×2 Then the unique representation π in Πϕ with HomH (π, νψc ) 6= 0 corresponds under the bijection J(θe × θ) to the restriction of the character χNc × χM : AM × A+ Nc −→ h±1i to the subgroup Aϕ = AM × A+ N , multiplied by the character ηc (a) = det N a (c)

of A+ N.

Proof. — Let π be the representation whose ψ-parameter is (M, N, χ), where χ is as given in the proposition: χ = χNc × χM · ηc : AM × A+ N −→ h±1i. We want to show that HomH (π, νψc ) 6= 0. By Conjecture 11.3, the ψc -parameters of π are (M (c), N, χ0 ) with χ0 = χNc · η[c] × χM · ηc2 . Hence, it suffices to show that χ0 is equal to the character predicted by Conjecture 17.3 relative to ψc . More precisely, we need to show that χ0 = χN1 × χM (c) . We now calculate this character on an element + (a0 , a) ∈ AM × A+ N = AM (c) × AN , + + using the fact that for a in CN → CN , c

Nca = N (c)a . Since this space has even dimension, det Nca = det N a = det N1a .

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

72

W. T. GAN, B. H. GROSS & D. PRASAD

Hence we have χ0 (a0 , a) = (χNc · η[c])(a0 ) · χM (a) 0

0

1

= (M a ⊗ Nc ) · (M a ⊗ (C ⊕ C(c))) · (M ⊗ Nca ) · det(Nca )(−1) 2 ·dim M 0

1

= (M a ⊗ (N (c) + C(c)) · (M ⊗ N a (c)) · det(N a (c))(−1) 2 ·dim M 0

1

= (M (c)a ⊗ N1 ) · (M (c) ⊗ N1a ) · det(N1a )(−1) 2 ·dim M = (χN1 × χM (c) )(a0 , a). This proves the proposition. Finally, we consider the skew-hermitian case with W ⊂ V of even codimension. We consider the two cases: (i) dim W ≡ dim V ≡ 1 mod 2; (ii) dim W ≡ dim V ≡ 0 mod 2 in turn. Assume first that dim W ≡ dim V ≡ 1 mod 2 and the discriminant of W and V is a trace zero element e ∈ k. In this case, the Vogan parametrization is completely canonical, given the spaces W ⊂ V , with the trivial character of Aϕ corresponding to a generic representation of G(V ) × G(W ). However, the representation ν of H depends not only on the spaces W ⊂ V but also on the choice of an additive character ψ0 : k0 → S1 and on the choice of a multiplicative character µ : k × → C× which is trivial on N(k × ) but nontrivial on k0× . Thus, to be completely precise, we denote the group H by HW,V and the representation ν by νW,V,ψ0 ,µ . Finally, the character χ of the component group is defined using µ and the additive character ψ(x) = ψ0 (Tr(ex)) of k and depends on ψ0 up to multiplication by Nk × . Suppose without loss of generality that the representation π with parameter (ϕ, χ) is one for the group G(W ) × G(V ), so that our conjecture (for the character ψ0 ) predicts that HomHW,V (π, νW,V,ψ0 ,µ ) 6= 0. If t represents the nontrivial coset of k0× /Nk × , let χt be the character of Aϕ defined using the character ψ0t (x) = ψ0 (tx). Then we have χt (a, b) = (M a ⊗ N (µ−1 ), ψt ) · (M ⊗ N (µ−1 )b , ψt ) a

b

= (−1)dim M · (−1)dim N · χ(a, b) = η(a, b) · χ(a, b). Thus, the representation πt indexed by the character χt is one for the pure inner form G(W 0 ) × G(V 0 ). Moreover, the spaces W 0 ⊂ V 0 are simply the spaces tW ⊂ tV obtained from W ⊂ V by scaling the skew-hermitian forms by t. Thus, our conjecture

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

73

(for the character ψ0t ) predicts that HomHtW,tV (πt , νtW,tV,ψ0t ,ν ) 6= 0. To see that this is equivalent to the prediction of our conjecture for the character ψ0 , note that G(W 0 ) × G(V 0 ) is canonically identified with G(W ) × G(V ) as a subgroup of GL(W ) × GL(V ) and under this identification, one has πt = π. Moreover, we also have HW 0 ,V 0 = HW,V as subgroups of G(W ) × G(V ) and νtW,tV,ψ0t ,µ = νW,V,ψ0 ,µ . This proves that our conjecture is internally consistent with changing ψ0 . On the other hand, if we replace µ by µ0 , then µ0 = µ · µ0 for some character µ0 : k × /k0× → C× . Moreover, we have [28] νµ0 ,ψ0 ∼ = µ0 · νµ,ψ0 . Hence HomH (π ⊗ ν µ0 ,ψ0 , C) ∼ = HomH ((π · µ−1 0 ) ⊗ ν µ,ψ0 , C). Now our conjecture for µ0 says that the left hand side of the above is nonzero if and only if π has Vogan parameter (M, N, χM,N,µ0 ) with χM,N,µ0 (a, b) = (M a ⊗ N (µ0

−1

), ψ) · (M ⊗ N (µ0

−1 b

) , ψ)

−1 −1 b = (M a ⊗ (N · µ−1 ), ψ) · (M ⊗ (N · µ−1 ) , ψ) 0 )(µ 0 )(µ

= χM,N (µ−1 ),µ (a, b). 0

On the other hand, our conjecture for µ says that the right hand side is nonzero if and −1 only if π · µ−1 , χM,N (µ−1 ),µ ). Thus, our conjecture 0 has Vogan parameter (M, N (µ0 ) 0 0 for µ is equivalent to that for µ. Finally, consider the case when dim W ≡ dim V ≡ 0 mod 2. In this case, we need the additive character ψ0 : k0 → S1 to specify the Vogan parametrization, and both ψ0 and µ to define the representation νW,V,ψ0 ,µ . The character χ, on the other hand, is independent of ψ0 but depends on µ. Suppose that under the ψ0 -Vogan parametrization, the representation π corresponding to the character χ is one for the group G(V ) × G(W ), so that our conjecture for ψ0 predicts that HomHW,V (π ⊗ ν W,V,ψ0 ,µ , C) 6= 0. If we replace the additive character ψ0 by ψ0t with t ∈ k0× but t ∈ / Nk × , then under t the ψ0 -Vogan parametrization, the character χ corresponds to the conjugated representation π t (using an element in the similitude group with similitude t). So our conjecture for ψ0t predicts that HomHW,V (π t ⊗ ν W,V,ψ0t ,µ , C) 6= 0. But one may check that t νW,V,ψ0t ,µ ∼ , = νW,V,ψ 0 ,µ

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

74

W. T. GAN, B. H. GROSS & D. PRASAD

so that the two predictions (for ψ0 and ψ0t ) are consistent with each other. The consistency check when changing µ is similar to the analogous situation treated above; so we omit the details.

19. Reduction to basic cases In this section, we shall show: Theorem 19.1. — Assume that k is a non-archimedean local field. Then Conjectures 17.1 and 17.3 follow from the basic cases where dim W ⊥ = 0 or 1. Proof. — As we shall explain, this is a simple consequence of Theorems 15.1 and 16.1. We treat the two cases separately. We first consider the orthogonal and hermitian cases. Suppose that W ⊂ V with W ⊥ = X + X∨ + E where X = hv1 , · · · , vn i is nonzero isotropic and E is a non-isotropic line. Let M and N be L-parameters for G(V ) and G(W ) respectively. We would like to verify Conjectures 17.1 and 17.3 for the associated Vogan packet ΠM × ΠN of G = G(V ) × G(W ). We shall exploit Theorem 15.1 for this purpose. Recall the setting of Theorem 15.1, where we have set W 0 = V ⊕ (−E) = V ⊕ k · f and Y = hv1 , · · · , vn , vn+1 i with vn+1 = e + f . Then we have W 0 = Y + Y ∨ + W. Let τ be an irreducible supercuspidal representation of GL(Y ) with L-parameter Nτ . We may assume that for any πW ∈ ΠN , the induced representation G(W 0 )

I(τ, πW ) = IndP (Y ) (τ  πW ) is irreducible. Then the set {I(τ, πW ) : πW ∈ ΠN } is simply the Vogan packet associated to the parameter N 0 = Nτ + N + (Nτσ )∨ . Moreover, there is a canonical isomorphism AN ∼ = AN 0 ∼ and the representations πW and I(τ, πW ) are indexed by the same character of A+ N = + AN 0 .

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

75

We may further assume that τ is chosen so that the conditions of Theorem 15.1 are met. Then by Theorem 15.1, we see that for any πV ∈ ΠM , HomG(V ) (I(τ, πW ) ⊗ πV , C) = HomH (πV ⊗ πW , ν). Thus, Conjecture 17.1 holds for ΠM × ΠN if it holds for ΠN 0 × ΠM . To see that the same implication holds for Conjecture 17.3, it suffices to check that the character χN × χM

+ of A+ M × AN

agrees with the character χN 0 × χM

+ of A+ M × AN 0 .

For a ∈ A+ M , we see from definition that χN 0 (a) = χN (a) · χNτ +(Nτσ )∨ (a), and it follows by Proposition 5.1 χNτ +(Nτσ )∨ (a) = 1 for any a ∈ A+ M. This establishes Theorem 19.1 in the orthogonal and hermitian cases. The symplectic and skew-hermitian cases are handled in a similar way, using Theorem 16.1; we omit the details.

20. Variant of the local conjecture In this section, we give a variant of the local conjecture 17.3. This variant does not require the precise parametrization of the members of a Vogan L-packet by the characters of the component group, which can be a very delicate issue. This conjecture is typically what is checked in practice. Suppose that W ⊂ V and we are given an L-parameter M of G(V ), so that M is a selfdual or conjugate-dual representation of the Weil-Deligne group W D(k) of k with a given sign. As described in §4, the component group AM is an elementary abelian 2-group with a canonical basis {ai }, indexed by the distinct isomorphism classes of irreducible summands Mi of M which are of the same type as M . Hence, we have a canonical isomorphism AM ∼ = Z/2Z · a1 × · · · × Z/2Z · ak . Here, the elements ai are such that M ai ∼ = Mi . We note that if V is a quadratic space of even dimension, then the relevant component group needed to describe the representation theory of pure inner forms of SO(V ), is a subgroup A+ M of AM of index atmost 2; the group AM itself should parametrize L-packets of representations of O(V ) as suggested in [57].

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

76

W. T. GAN, B. H. GROSS & D. PRASAD

Now consider a representation π of G(V ) in the Vogan packet ΠM . Let LW (π) denote the set of generic L-parameters N for G(W ) such that X d(π, N ) := dim HomH (π  π 0 , ν) 6= 0. π 0 ∈ΠN

According to our Conjecture 17.1, one has a partition {generic L-parameters of G(W )} =

[

LW (π).

π∈ΠM

In this context, we have the following variant of Conjecture 17.3. Conjecture 20.1. — (1) Let π be an irreducible admissible representation of G(V ) belonging to the Vogan packet ΠM . For any L-parameter N in LW (π), define a character χπ,N on AM , by 1

1

χπ,N (ai ) = (Mi ⊗ N, ψ) · det Mi (−1) 2 dim N · det N (−1) 2 dim Mi for a fixed ψ appropriate for each of the cases studied in this paper, viz. any (nontrivial) character of k in the orthogonal or the symplectic case, any character of k/k0 in 1 1 the hermitian or skew-hermitian case; the factor det Mi (−1) 2 dim N ·det N (−1) 2 dim Mi arises only in the orthogonal case, where Mi is orthogonal and N is symplectic, or the other way around; the factor corresponding to the determinant of a symplectic representation is taken to be 1. Then for any two L-parameters N and N 0 in LW (π), we have: χπ,N (ai ) = χπ,N 0 (ai ), on the subgroup A+ M of AM in the even orthogonal case, and on AM in all the other cases. In particular, π determines a character χπ on AM (or A+ M ), defined by χπ (ai ) = χπ,N (ai ) for any N ∈ LW (π). (2) The map π 7→ χπ gives a bijection (depending on the choice of a character ψ) ΠM ←→ Irr(AM ). (3) One has the analogs of (1) and (2) above with the roles of V and W exchanged. In effect, the above conjecture says that one can exploit the restriction problem for W ⊂ V and use the collection of epsilon factors described above to serve as parameters for elements of ΠM . In the hermitian and skew-hermitian cases, the character χπ associated to a given π ∈ ΠM is equal to the character χN of AM defined in §6, for any N ∈ LW (π), provided the additive character ψ is appropriately chosen. In the orthogonal and symplectic cases, these two characters may differ.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

77

21. Unramified parameters In this section, we assume that k is a non-archimedean local field with ring of integers A, uniformizing element $ and finite residue field A/$A. We will also assume that A/$A has characteristic p > 2, so that the group A× /A×2 has order 2. In the case when k has a nontrivial involution σ, we will assume that the action of σ on A/$A is also nontrivial. Then k is unramified over k0 and every unit in the subring A0 of A fixed by σ is the norm of a unit of A. In addition, we will only consider additive characters ψ of k which are trivial on A but not on $−1 A. Then ψ is determined up to translation by a unit in A. If we insist that ψ σ = ψ ±1 , then ψ is determined up to translation by a unit in A0 . We call such additive characters of k unramified. Let W D(k) = W (k) × SL2 (C) be the Weil-Deligne group of k. A representation ϕ : W D(k) −→ GL(M ) is unramified if ϕ is trivial on SL2 and on the inertia subgroup I of W (k). An unramified representation is determined by the semisimple conjugacy class ϕ(F ) in GL(M ). Let C(s) denote the one dimensional unramified representation of W D(k) with ϕ(F ) = s ∈ C× . Then any unramified representation M of W D(k) is isomorphic to a direct sum of the form n M M= C(si ), with n = dim M . i=1

We now determine which unramified representations of W D(k) are selfdual or conjugate-dual (with respect to the unramified involution σ of k). Proposition 21.1. — Assume that M is an unramified representation of W D(k) and is either selfdual or conjugate-dual. Then M is isomorphic to a direct sum of the form: M∼ = ⊕i (C(si ) + C(s−1 )) ⊕ m · C(−1) ⊕ n · C(1), i

with si 6= s−1 in C× and m, n ≥ 0 in Z. i If M has this form, then we have the following cases: (i) M is orthogonal and its centralizer in SO(M ) has component group ( Z/2Z, if both m, n > 0, + AM = 1, otherwise. (ii) M is symplectic if and only if m ≡ n ≡ 0 mod 2, in which case its centralizer in Sp(M ) has component group AM = 1. (iii) M is conjugate-orthogonal if and only if m ≡ 0 mod 2, in which case its centralizer in Aut(M, B) has component group ( Z/2Z, if n > 0, AM = 1, otherwise.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

78

W. T. GAN, B. H. GROSS & D. PRASAD

(iv) M is conjugate-symplectic if and only if n ≡ 0 mod 2, in which case its centralizer in Aut(M, B) has component group ( Z/2Z, if m > 0, AM = 1, otherwise. Proof. — Since C(s)∨ ∼ = C(s−1 ) and C(s)σ ∼ = C(s), the one dimensional representation C(s) is selfdual or conjugate-dual if and only if s2 = 1. In the selfdual case, both C(−1) and C(1) are orthogonal. In the conjugate-dual case, C(1) is conjugateorthogonal and C(−1) is conjugate-symplectic. Indeed, the unramified character µ : k × /Nk × −→ h±1i defined by is nontrivial on

k0× .

µ(α) = (−1)ord$ (α) The proposition follows easily.

Proposition 21.2. — (i) If M and N are two selfdual unramified representations of W D(k) of even dimension, with signs cM and cN respectively, then the character χN : A+ M −→ h±1i defined by 1

1

χN (a) = (M a ⊗ N, ψ) · det M a (−1) 2 dim N · det N (−1) 2 dim M

a

is trivial. (ii) If M and N are two conjugate-dual unramified representations with signs cM and cN respectively and ψ σ = ψ −1 , then the character χN : A+ M −→ h±1i defined by χN (a) = (M a ⊗ N, ψ) is trivial. Proof. — If M is any unramified representation of W D(k) and ψ is an unramified additive character, then we have the formulae: (M, ψ) = 1 and

det M (−1) = 1.

The proposition follows easily from these facts. We now turn to the restriction conjectures for unramified generic parameters ϕ. Since χN × χM is the trivial character of Aϕ , the unique representation in the associated Vogan packet which supports a nonzero Hom space should be the one indexed by the trivial character. In that case, for the purpose of global applications, we can make our conjectures more refined. Recall that W ⊂ V is a pair of nondegenerate spaces for the sesquilinear form h−, −i. and that W ⊥ is split. We say that an A-lattice L ⊂ V is nondegenerate if (1) h−, −i : L × L → A;

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

79

(2) the map L → Hom(L, A) defined by mapping w to fw (v) = hv, wi is an isomorphism of A0 -modules. We assume henceforth that there is a nondegenerate A-lattice L ⊂ V with the additional property that LW = L ∩ W is a nondegenerate A-lattice in W . Then the ⊥ ⊥ orthogonal complement L⊥ W of LW in L is a nondegenerate lattice in W , so that LW has the form Y + Y ∨ or Y + Y ∨ + Ae with Y isotropic and Y ∨ ∼ = Hom(Y, A). Moreover, L = LW + L⊥ . W

Under this assumption, the group G = G(V ) × G(W ) is quasi-split and split by an unramified extension of k0 . Indeed, the subgroup J = Aut(L) × Aut(LW ) is a hyperspecial maximal compact subgroup of G. We now construct the subgroup H of G and the unitary representation ν of H using this unramified data. Write L = LW + L⊥ W , and define the parabolic subgroup PA and its unipotent radical NA using a complete A-flag in the isotropic subspace Y ⊂ L⊥ W . Then HA = NA · Aut(LW ) gives a model of H over A0 and HA = J ∩ HA (k0 ) = J ∩ H. In the orthogonal and hermitian cases, the one dimensional representation ν of H associated to the decomposition ⊥ L⊥ + Ae W =Y +Y

and a suitable unramified additive character ψ has trivial restriction to the subgroup J ∩ H. In the metaplectic case, we can define ν = νψ using an unramified additive character ψ (there are two choices, up to translation by A×2 ). In the skew-hermitian case, we define ν = νψ,µ using an unramified character ψ with ψ σ = ψ (which is unique up to translation by NA× ) and the unramified symplectic character µ associated to the representation C(−1). Then in all cases, the representation ν of H is J ∩ H-spherical; it has a unique line fixed by the compact open subgroup J ∩ H. Since the group G is quasi-split over k0 , we can also define unramified generic characters θ of the unipotent radical U of a Borel subgroup, using the pair LW ⊂ L of nondegenerate lattices and a suitable unramified additive character ψ. Again, the T -orbit of θ is unique except in the metaplectic case when there are two unramified orbits. In all cases, the restriction of θ to the compact open subgroup J ∩ U is trivial. To summarize, if we use unramified data to define the representations θ of U and ν of H, then the complex vector spaces HomJ∩U (C, θ) and HomJ∩H (C, ν) both have dimension equal to 1. Now let ϕ be an unramified generic parameter and let π be the unique θ-generic element in the Vogan packet πϕ . Then the formula of Casselman and Shalika [12] shows that

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

80

W. T. GAN, B. H. GROSS & D. PRASAD

(i) HomJ (C, π) has dimension 1; (ii) the pairing of one-dimensional complex vector spaces HomJ (C, π) ⊗ HomU (π, θ) −→ HomJ∩U (C, θ) = C is nondegenerate. We conjecture that the same is true for the representation ν of H. Conjecture 21.3. — Let π be the unique J-spherical representation in the Vogan packet Πϕ . Then (i) HomH (π, ν) has dimension 1; (ii) the pairing of one-dimensional complex vector spaces HomJ (C, π) ⊗ HomH (π, ν) −→ HomJ∩H (C, ν) is nondegenerate. Besides the cases treated by Casselman-Shalika [12], this conjecture has been verified in a large number of cases, which we summarize below. Theorem 21.4. — Conjecture 21.3 is known in the following cases: (i) the special orthogonal and hermitian cases; (ii) the general linear case, with dim W ⊥ = 1; (iii) the symplectic case, with dim W ⊥ = 2. Proof. — The orthogonal case is due to Kato-Murase-Sugano [38]. Their proof is extended to the unitary case by Khouri in his Ohio-State PhD thesis [39]. Parts (ii) and (iii) are both due to Murase-Sugano [54, 55].

22. Automorphic forms and L-functions The remainder of this paper is devoted to formulating global analogs of our local conjectures. Let F be a global field with ring of adèles A and let G be a reductive algebraic group over F . Then G(F ) is a discrete subgroup of the locally compact group G(A). For simplicity, we shall further assume that the identity component of the center of G is anisotropic, so that the quotient space G(F )\G(A) has finite measure. We shall consider the space A (G) of automorphic forms on G, which consists of smooth functions f : G(F )\G(A) −→ C satisfying the usual finiteness conditions [52, I.2.17, Pg. 37], except that we do not impose the condition of K∞ -finiteness at the archimedean places. For each open compact Kf ⊂ G(Af ), the space A (G)Kf has a natural topology, giving it the structure of an LF-space (see [81]) with respect to which the action of G(F ⊗ R) is smooth.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

81

Let A0 (G) ⊂ A (G) denote the subspace of cusp forms. An irreducible admissible representation π = π∞ ⊗ πf of G(A) is cuspidal if it admits a continuous embedding π ,→ A0 (G). The multiplicity of π in A0 (G) is the dimension of the space HomG(A) (π, A0 (G)), which is necessarily finite. Suppose now that G is quasi-split, with a Borel subgroup B = T · U defined over F . A homomorphism λ : U −→ Ga is generic if its centralizer in T is equal to the center of G. Composing λA with a nontrivial additive character ψ of A/F gives an automorphic generic character θ = ψ ◦ λA . Now one may consider the map F (θ) : A (G) −→ C(θ) defined by Z f 7→

f (u) · θ(u) du. U (F )\U (A)

The map F (θ) is a nonzero continuous homomorphism of U (A)-modules, which is known as the θ-Fourier coefficient. If F (θ) is nonzero when restricted to the π-isotypic component in A (G), we say that π is globally generic with respect to θ. The notion of automorphic forms can also be defined for nonlinear finite covers ‹ G(A) of G(A), which are split over the discrete subgroup G(F ); see [53]. For the purpose of this paper, we only need to consider this in the context of the metaplectic double cover of Sp(W )(A) and so we give a brief description in this case. Assume that the characteristic of F is not two. For each place v of F , we have a f unique nonlinear double cover Sp(W )(Fv ) of Sp(W )(Fv ). If the residual characteristic of Fv is odd, then this cover splits uniquely over a hyperspecial maximal compact subgroup Fv of Sp(W )(Fv ). Hence, one may form the restricted direct product Y f Sp(W )(Fv ), Kv

which contains a central subgroup Z = ⊕v µ2,v . If Z + denotes the index two subgroup of Z consisting of elements with an even number of components equal to −1, then the group ! Y f f Sp(W )(A) := Sp(W )(Fv ) /Z + Kv

is a nonlinear double cover of Sp(W )(A). It is a result of Weil that this double cover splits uniquely over the subgroup Sp(W )(F ), so that we may speak of automorphic f f forms on Sp(W )(A). An automorphic form f on Sp(W )(A) is said to be genuine if it satisfies f ( · g) =  · f (g) for  ∈ µ2 . f f We denote the space of genuine automorphic forms on Sp(W )(A) by A (Sp(W )). If B = T · U is a Borel subgroup of Sp(W ), then the double covering splits uniquely over U (Fv ) for each v. Hence, in the adelic setting, there is a unique splitting of the

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

82

W. T. GAN, B. H. GROSS & D. PRASAD

double cover over U (A), and more generally over the adelic group of the unipotent radical of any parabolic subgroup of Sp(W ). As a result, one can define the notion of cusp forms as in the linear case, and we denote the space of such cusp forms by f A0 (Sp(W )). Moreover, if θ is a generic automorphic character of U , then one can define the θ-Fourier coefficient of f in the same way as before. For the global analog of our restriction problems, we also need to discuss the notion of automorphic forms on the non-reductive group JW = Sp(W ) n H(W ) where H(W ) = W ⊕ F is the Heisenberg group associated to W . The group JW is called the Jacobi group associated to W and we shall consider its double cover f JeW (A) = Sp(W )(A) · H(W )(A). For a given additive character ψ of F \A, one has the the space of automorphic forms Aψ (JeW ) on JeW (A), which consists of certain smooth functions on JW (F )\JeW (A) with central character ψ and is usually called the space of Jacobi forms. For our applications, we are interested in a particular automorphic representation of JeW (A), namely the automorphic realization of the global Weil representation Q associated to ψ = v ψv . Recall that for each place v, the group f JeW (Fv ) = Sp(W )(Fv ) · H(W )(Fv ) has a local Weil representation ωψv whose restriction to H(W )(Fv ) is the unique irreducible representation with central character ψv . The restricted tensor product ωψ = “ ⊗v ωψv is the global Weil representation associated to ψ. One of the main results of Weil [82] is that there is a unique (up to scaling) continuous embedding θψ : ωψ ,→ Aψ (JeW ). f Composing θψ with the restriction of functions from JeW (A) to Sp(W )(A) gives a f Sp(W )(A)-equivariant (but not injective) map f ωψ −→ A (Sp(W )). We now come to the global L-functions and epsilon factors associated to an irreducible cuspidal representation π, following Langlands. To define an L-function or epsilon factor, one needs the extra data of a finite dimensional representation R of the L-group L G. If π = ⊗v πv is an irreducible automorphic representation and we assume the local Langlands-Vogan correspondence for G(Fv ), then each πv determines a local L-parameter φv : W D(Fv ) −→ L G. Hence, one has the local L-factors L(R ◦ φv , s) and one defines the global L-function Y L(π, R, s) = L(R ◦ φv , s), v

ASTÉRISQUE 346

83

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

which converges when Re(s) is sufficiently large. Similarly, one has the local epsilon factors v (π, R, ψ, s) = (R ◦ φv , ψv , s), and one defines the global epsilon factor by Y (π, R, s) = v (π, R, ψ, s). v

It is a finite product independent of the additive character ψ of A/F . One expects that the L-function above has meromorphic continuation to the whole complex plane and satisfies a functional equation of a standard type, taking s to 1 − s, so that the center of the critical strip is s = 1/2. The following table gives some examples of R and their associated L-functions which appear in this paper. When the cuspidal representation π is globally generic, the meromorphic continuation of these L-functions are known. L

G GL(V ) GL(V ) GL(V /E), E/F quadratic GL(V ) × GL(W ) SO(W ) × SO(V ), dim W ⊥ odd f Sp(W ) × S p(V )

G GL(M ) GL(M ) (GL(M ) × GL(M )) · Gal(E/F ) GL(M ) × GL(N ) O(M ) × Sp(N ), dim M even SO(M ) × Sp(N ), dim M odd

U(W ) × U(V )

(GL(M ) × GL(N )) · Gal(E/F )

R Sym2 M ∧2 M As± (M ) M ⊗N M ⊗N M ⊗N L

G (M G

Ind

⊗ N)

b

23. Global Restriction Problems We are now ready to formulate the global restriction problems. We shall change notations slightly from the earlier part of the paper, by replacing the pair of fields k0 ⊂ k in the local setting by F ⊂ E in the global setting, with the characteristic of F different from 2. Hence σ is an involution (possibly trivial) on E with E σ = F , and V is a vector space over E equipped with a sesquilinear form h−, −i of the relevant type. The group G = G(V ) is then an algebraic group over F . Also, we shall include the case E = F × F in our discussion. Suppose that we have a pair of vector spaces W ⊂ V over E equipped with a ⊥ sesquilinear form of sign , such that W ⊥ is split and  · (−1)dim W = −1. Then we have the groups ( G = G(V ) × G(W ); H = N · G(W ) over F , as defined in §2. The groups of F -points G(F ) and H(F ) are discrete subgroups of the locally compact adelic groups G(A) and H(A) respectively, where A is the ring of adèles of F . In the orthogonal case, we assume that if V or W has dimension 2, then it is not split.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

84

W. T. GAN, B. H. GROSS & D. PRASAD

Then the quotient spaces G(F )\G(A) and H(F )\H(A) both have finite measure. We may then consider the space of automorphic forms and cusp forms for the group G, as in §22. In this section, we will consider irreducible tempered representations π of G(A) which occur in the space of cusp forms A0 (G) on G(F )\G(A) and study their restriction to H(A). As in the local setting, when G is quasi-split, we need to introduce an automorphic generic character θ : U (F )\U (A) −→ S1 for the group G; this serves to fix the local Langlands-Vogan parametrization at all places v of F . In addition, we need to construct an automorphic representation ν on H(F )\H(A) for the restriction problem. Assume in this paragraph that G = G(V ) × G(W ) is quasi-split. In the orthogonal or symplectic case, we use the spaces W ⊂ V to naturally define a generic F -homomorphism λ : U −→ Ga as in §12. This defines λA : U (A) −→ A; now composing λA with a nontrivial additive character ψ : A/F −→ S1 gives an automorphic generic character of U (A): θ = ψ ◦ λA . In the hermitian or skew-hermitian case, we use the spaces W ⊂ V to construct a generic homomorphism λ : U −→ ResE/F (Ga ). Then, in the hermitian case, we compose λA with a fixed nontrivial additive character ψ : AE /(E + A) −→ S1 to obtain an automorphic generic character θ0 = ψ ◦ λA . We then set θ(x) = θ0 (−2 · δ · λA (x)) where δ is the discriminant of the odd hermitian space. In the skew-hermitian case, we take a nontrivial additive character ψ : A/F −→ S1 and set θ(x) = ψ(2 · T rE/F (λA (x)). We stress that these definitions are global analogs of our definitions in the local setting. Next, we need to define an automorphic version of the representation ν of H(A) = N (A)·G(W )(A). (The group G(W ) is not assumed to be quasi-split.) In the orthogonal and hermitian cases, we define ν by composing the generic G(W )-invariant map l : N −→ ResE/F (Ga ),

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

85

constructed in §12 using the spaces W ⊂ V , with a nontrivial additive character ψ : AE /E → S1 and then extending this trivially on G(W )(A): ν = ψ ◦ lA . As in the local case, the choice of ψ is unimportant. Then we define: F (ν) : A0 (G) −→ C(ν) by Z f 7→

f (h) · ν(h) · dh. H(F )\H(A)

The map F (ν) is called a Bessel coefficient. In the symplectic and skew-hermitian cases, the representation ν is infinitedimensional; so the situation is slightly more involved. Recall from §12 that, using the spaces W ⊂ V , we have defined a G(W )-invariant generic linear form l : N −→ Ga , which gives rise to a continuous linear map lA : N (A) −→ A. Composing this with a nontrivial additive character ψ : A/F → S1 , and extending trivially to G(W )(A), we obtain an automorphic character Λ = ψ ◦ lA of H(A). On the other hand, we have also defined a homomorphism N −→ H(W ) where H(W ) = F ⊕ ResE/F W is the Heisenberg group associated to ResE/F (W ). Thus, we have a homomorphism H = G(W ) · N −→ JW := Sp(ResE/F (W )) · H(W ). As discussed in §22, the group JeW (A) has a global Weil representation ωψ with central character ψ, and one has a canonical automorphic realization θψ : ωψ ,→ A (JeW ). It will now be convenient to consider the symplectic and skew-hermitian cases separately. In the symplectic case, the map H −→ JW defined above gives rise to a map ‹ H(A) −→ JeW (A). ‹ By pulling back, one can thus regard ωψ as a representation of H(A). Moreover, the above map gives rise to a natural inclusion ‹ A (JeW ) ,→ A (H).

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

86

W. T. GAN, B. H. GROSS & D. PRASAD

Composing the automorphic realization θψ with this inclusion realizes ωψ as a sub‹ Multiplying by the automorphic character Λ of H, one obtains an module in A (H). automorphic realization ‹ θψ : νψ = ωψ ⊗ Λ ,→ A (H). Now we can define the map F (νψ ) : A0 (G) ⊗ νψ −→ C by Z f ⊗ φ 7→

f (h) · θψ (φ)(h) dh. H(F )\H(A)

The map F (νψ ) is called a Fourier-Jacobi coefficient. In the skew-hermitian case, we choose an automorphic character × × µ : A× E /E −→ C

satisfying µ|A× = ωE/F . Then one obtains a splitting homomorphism (see [43] and [28]) sψ,µ : H(A) −→ JeW (A). Using sψ,µ , one may pull back the global Weil representation ωψ to obtain a representation ωψ,µ of H(A). As above, one also obtains an automorphic realization θψ,µ : νψ,µ = ωψ,µ ⊗ Λ −→ A (H). Thus, we can define the map F (νψ,µ ) : A0 (G) ⊗ νψ,µ −→ C by Z f ⊗ φ 7→

f (h) · θψ,µ (φ)(h) dh. H(F )\H(A)

The map F (νψ,µ ) is called a Fourier-Jacobi coefficient in the context of unitary groups. Now the global restriction problem is: Determine whether the map F (ν) defined in the various cases above is nonzero when restricted to a tempered cuspidal representation π of G(A).

24. Global conjectures: central values of L-functions To formulate our global conjectures to the restriction problem of the previous section, we need to introduce a distinguished symplectic representation R of the Lgroup L G over F . Recall from §7 that either the L-group of a classical group or its identity component comes equipped with a standard representation. Thus, with G =

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

87

“ has a standard representation G(V ) × G(W ), either L G or its identity component G M ⊗ N . We set  M ⊗ N in the orthogonal and symplectic cases;      LG R = IndG b (M ⊗ N ), in the hermitian case;     boW (F ) IndG ((M ⊗ N )  µ−1 ), in the skew-hermitian case. b×W (E) G This representation R was already introduced in the table at the end of §22, except for the skew-hermitian case. In the skew-hermitian case, we have incorporated the character µ used in the definition of ν. In doing so, we need to work with the L-group “ o W (F ) rather than the version G “ o Gal(E/F ) which we have been using in the G rest of the paper. It is this twist by µ which makes R a symplectic representation “ o W (F ) (by Lemmas 3.4 and 3.5). As explained in §22, we can then speak of of G the L-function L(π, R, s) and the global epsilon factor (π, R, s) for any automorphic representation π of G. We recall that these L-functions are normalized so that the functional equation takes s to 1 − s and the center of the critical strip is s = 1/2. The first form of our global conjecture is: Conjecture 24.1. — Let π be an irreducible tempered representation of G(A) which occurs with multiplicity one in the space A0 (G) of cusp forms on G(F )\G(A). Let ν be the automorphic representation of H(A) introduced in §23. Then the following are equivalent: (i) the restriction of the linear form F (ν) to π is nonzero; (ii) the complex vector space HomH(A) (π, ν) is nonzero and the L-function L(π, R, s) does not vanish at s = 1/2, which is the center of the critical strip; (iii) the complex vector spaces HomH(Fv ) (πv , νv ) are nonzero for all places v of F and L(π, R, 1/2) 6= 0. Let us make some remarks about this conjecture. Remarks. — (i) The equivalence of (ii) and (iii) is clear, provided one knows Conjecture 21.3. (ii) When E = F × F , with G ∼ = GL(V0 ) × GL(W0 ), then the L-function L(π, R, s) is the product ∨ L(π, R, s) = L(πV ⊗ πW , s) · L(πV∨ × πW , s) of two Rankin-Selberg L-functions, so that L(π, R, 1/2) = |L(πv × πW , 1/2)|2 . In this case, the conjecture is known when dim W ⊥ = 1. Indeed, this is an immediate consequence of the integral representation of the global Rankin-Selberg L-function L(πV × πW , 1/2) [34]. The general case has been addressed in the recent preprint of Y. F. Liu [47]. (iii) More generally, under the assumption that π is globally generic and has a cuspidal functorial lift to the appropriate general linear group, the implication (i) =⇒

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

88

W. T. GAN, B. H. GROSS & D. PRASAD

(ii) has been shown by Ginzburg-Jiang-Rallis in a series of papers for the various cases [21, 22, 23]. Moreover, in the hermitian case with dim W ⊥ = 1, an approach to this conjecture via the relative trace formula has been developed by Jacquet and Rallis [35]. The recent preprint [86] of Wei Zhang successfully realizes this strategy, and establishes the global conjecture in the Hermitian case under some simplifying local assumptions. (iv) One expects a refinement of Conjecture 24.1 in the form of an exact formula relating |F (ν)|2 with the central value L(π, R, 1/2). Such a refinement has been formulated by Ichino-Ikeda [33] in the orthogonal case, with dim W ⊥ = 1. In the analogous setting for the hermitian case, the formulation of this refined conjecture is the UCSD thesis of N. Harris [30]. As formulated, the global conjecture 24.1 is essentially independent of the local conjectures 17.1 and 17.3. Rather, they complement each other, since the local nonvanishing in Conjecture 24.1 is governed by our local conjectures. From this point of view, the appearance of the particular central L-value L(π, R, 1/2) may not seem very well-motivated. However, as we shall explain in the next two sections, if we examine the implications of our local conjectures in the framework of the Langlands-Arthur conjecture on the automorphic discrete spectrum, the appearance of L(π, R, 1/2) is very natural. For example, observe that the global conjecture 24.1 in the symplectic/metaplectic case involves the central L-value of the symplectic representation R = M ⊗ N with M symplectic and N odd orthogonal, whereas in the local conjecture 17.3, it is the epsilon factor associated to M ⊗ (N ⊕ C) which appears. So in some sense, the global conjecture is less subtle than the local one. The explanation of this can be found in §26, as a consequence of the Langlands-Arthur conjecture (or rather its extension to the metaplectic case). Finally, we highlight a particular case of the conjecture. As we explained in §12, special cases of the data (H, ν) are automorphic generic characters ν = θ on H = U . These cases are highlighted in the following table, and arise when the smaller space W is either 0 or 1-dimensional. G(V ) odd orthogonal even orthogonal symplectic metaplectic odd hermitian even skew hermitian

dim W 0 1 0 0 0 0

’) G(W SO(0) Sp(0) Sp(0) SO(1) GL(0) GL(0)

N 0 0 0 C 0 0

As one sees from the table, in all except the metaplectic case, N = 0 so that R = 0 and L(π, R, s) is identically 1. In the metaplectic case, N = C so that R = M and L(π, R, s) is the standard L-function L(π, s). Hence Conjecture 24.1 specializes to the following two conjectures in these degenerate cases.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

89

Conjecture 24.2. — Let π be an irreducible tempered representation of G(A) which occurs with multiplicity one in the space A0 (G) of cusp forms on G(F )\G(A) and let θ be an automorphic generic character for G. Then, when G is a linear group, the following are equivalent: (i) the restriction of the map F (θ) to π is nonzero; (ii) the complex vector spaces HomU (Fv ) (πv , θv ) are nonzero for all places v of F . ‹ f )(A) is metaplectic, we fix an additive character ψ of A/F When G(A) = Sp(V which determines an automorphic generic character θ and also gives the notion of Langlands-Vogan parameters. For any element c ∈ F × /F ×2 , let χc be the associated quadratic character of A× /F × and let θc denote the generic character associated to the additive character ψc (x) = ψ(cx). Then we have: ‹ Conjecture 24.3. — Let π be an irreducible tempered representation of G(A) = f )(A) which occurs with multiplicity one in the space A0 (G) of cusp forms on Sp(V ‹ G(F )\G(A) and let θ be an automorphic generic character for G. Then the following are equivalent: (a) the restriction of the map F (θc ) to π is nonzero; (b) the complex vector spaces HomU (Fv ) (πv , θc,v ) are nonzero for all places v of F and L(π ⊗ χc , 1/2) 6= 0. In the metaplectic case, with dim V = 2, the above conjecture is known by the work of Waldspurger [71, 73]. Note that if the conditions in the above two conjectures hold, the space HomU (A) (π, θ) has dimension 1 and F (θ) is a basis. Moreover, since π is tempered in the conjecture, the adjoint L-function L(π, Ad, s) of π is expected to be regular and nonzero at s = 1 (which is the edge of the critical strip). Indeed, just as the holomorphy of the local adjoint L-factor at s = 1 characterizes the generic L-packets in the local part of this paper, the tempered cuspidal representations considered in the global conjectures of this section should be characterized by the analytic properties of their global adjoint L-function. More precisely, we have: Conjecture 24.4. — Let G be a connected reductive group over F (or the metaplecf )(A)) and let Ad be the adjoint representation of the L-group L G on tic group Sp(V “ “ Lie(G/Z(G)). (a) Let π be a tempered automorphic representation of G(A). Then the following are equivalent: (i) The representation π is cuspidal; (ii) The adjoint L-function L(π, Ad, s) is holomorphic at s = 1. (b) Let π be a cuspidal representation of G(A). Then the following are equivalent: (i) The representation π is tempered; (ii) The partial adjoint L-function LS (π, Ad, s) is holomorphic in Re(s) ≥ 1 (for S a finite set of places containing all archimedean places and finite places v where G×F Fv or πv is ramified).

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

90

W. T. GAN, B. H. GROSS & D. PRASAD

The rationale for part (a) of the conjecture is that one expects the conjectural “ L-parameter of a tempered representation π to have finite centralizer modulo Z(G) if and only if π is cuspidal, and further, the holomorphy of the adjoint L-function “ of the centralizer. of a tempered π at s = 1 detects the finiteness (modulo Z(G)) The rationale for part (b) is similar, taking into account the conjecture of Arthur [5, 6] which describes the non-tempered part of the cuspidal spectrum in terms of A-parameters. The implication (i) =⇒ (ii) (in both (a) and (b)) will follow from known analytic properties of Rankin-Selberg L-functions of GLn once the functorial lifting from classical groups to GLn is established. In general, this conjecture should be a consequence of the Ramanujan conjecture and the Arthur conjecture [5, 6]. 25. Global L-parameters and Multiplicity Formula In this section, we review the notion of global L-parameters in the context of a fundamental conjecture of Langlands and Arthur [5, 6], concerning multiplicities of representations in the automorphic discrete spectrum. We will only present this conjecture for tempered representations, and will also discuss its simplification for the classical groups considered in this paper. In the next section, we shall re-examine the global conjecture 24.1 in the framework of the Langlands-Arthur conjecture. We henceforth assume that the F -algebra E is a field. Let G be a connected reductive group over the global field F , and assume that the quotient space G(F )\G(A) has finite volume. The Langlands-Arthur conjecture gives a description of the decomposition of the discrete spectrum L2disc (G(F )\G(A)) or equivalently the space A 2 (G) of square-integrable automorphic forms. We shall only describe this conjecture for the tempered part of the discrete spectrum, which we denote by L2disc,temp (G). Note that L2disc,temp (G) is necessarily contained in the cuspidal spectrum by a result of Wallach [79]. Suppose that G0 is the quasi-split inner form of G over F , with a Borel-subgroup B = T · U . Fix an automorphic generic character θ = ⊗v θv of U as in the previous section. We fix an integral structure on G, which determines a hyperspecial maximal compact subgroup Jv ⊂ G0 (Fv ) for almost all finite places v, as in §21. If G = G(V ) is a classical group, such an integral structure is given by fixing a lattice L ⊂ V . The Langlands-Arthur Conjecture (1) For any pure inner form G of G0 , there is a decomposition M d 2 L2disc,temp (G) = Lφ (G), φ

where φ runs over the discrete global L-parameters and each L2φ is a G(A)-submodule. The precise definitions of these objects are given as follows. By definition, a discrete global L-parameter is a homomorphism “ o WF φ : LF −→ L G = L G0 = G

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

91

“ is finite. These parameters are taken up such that its centralizer in the dual group G “ to conjugacy by the dual group G. Moreover, LF is the hypothetical Langlands group of F – the global analog of the Weil-Deligne group W D(Fv ) – whose existence is only conjectural at this point. One postulates however that there is a natural surjective map LF −→ WF (the Weil group of F ), and the projection of φ to the second factor WF in L G is required to be this natural surjection. Moreover, one postulates that for each place v of F , there is a natural conjugacy class of embedding W D(Fv ) −→ LF . Assuming the above, one may attach the following data to a given discrete global L-parameter φ: (i) a global component group Aφ = ZG b(Im(φ)), which is finite by assumption. (ii) for each place v of F , a local L-parameter φ

φv : W D(Fv ) −−−−→ LF −−−−→

L

G0

for the local group G0,v , such that for almost all v, φv is unramified. This gives rise to a natural map of component groups: Aφ −→ Aφv . One thus has a diagonal map ∆ : Aφ −→

Y

Aφ v .

v

(iii) for each place v, the local Vogan packet Πφv of irreducible representations of the pure inner forms G(Fv ), together with a bijection J(θv ) : Πφv ↔ Irr(Aφv ) specified by the local component θv of the automorphic generic character θ. For an irreducible character ηv of Aφv , we denote the corresponding representation in Πφv by πηv . In particular, the representation corresponding to the trivial character of Aφv is a representation of G0 (Fv ) and for almost all v, it is spherical with respect to the hyperspecial maximal compact subgroup Jv . (iv) a global Vogan packet O Πφ = {πη = πηv : πηv ∈ Πφv and ηv is trivial for almost all v}. v

In particular, the representations in the global packet are indexed by irreducible characters Y η = ⊗v ηv of Aφ v . v

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

92

W. T. GAN, B. H. GROSS & D. PRASAD

If πηv is a representation of Gηv (Fv ), then πη is a representation of the restricted direct product Y Gη := Gηv (Fv ). Jv

Note, however, that the group Gη need not be the adelic group of a pure inner form of G0 . For example, in the classical group case, Gη need not be associated to a space V equipped with a relevant sesquilinear form over F . If Gη = G(A) for a pure inner form of G0 , we shall call the representation Q η = v ηv coherent. This notion can be explicated as follows. It was shown by Kottwitz [41, Cor. 2.5 and Prop. 2.6] that one has a natural map L L 1 ” W (Fv ) ), C× ) v H (Fv , G0 ) −−−−→ v Hom(π0 (Z(G0 )   y ”0 )W (F ) ), C× ) Hom(π0 (Z(G and the kernel of this map is the image of the natural map M H 1 (F, G0 ) −→ H 1 (Fv , G0 ). v

Now the character η = v ηv gives rise to an element in ⊕v H 1 (Fv , G0 ), and η is coherent if and only if this element is in the image of H 1 (F, G0 ). Thus, we see ”0 )W (F ) ). that η is coherent if and only if η is trivial when restricted to π0 (Z(G (v) for each πη ∈ Πφ , a non-negative integer Q

mη = h∆∗ (η), 1iAφ , where the expression on the right denotes the inner product of the two characters of the finite group Aφ . Thus mη is the multiplicity of the trivial character of Aφ , in the representation obtained by restriction of the tensor product of the representations ηv to the diagonal. If η is not coherent, then one sees that mη ”0 )W (F ) ) and is equal to zero, since η is nontrivial when restricted to π0 (Z(G hence on Aφ . When η is coherent, so the adelic group Gη is defined over F , the Langlands-Arthur conjecture for tempered representations predicts that mη is the multiplicity of the representation πη in the discrete spectrum of Gη With the above data, we have: (2) As G runs over all pure inner forms of G0 over F , there is an equivariant decomposition: M M L2φ (G) = m η · πη . G

η

We denote the representation in Πφ associated to the trivial character by π0 . It is a representation of G0 (A) and is the unique representation in Πφ which is abstractly θ-generic. According to the multiplicity formula in (v), its multiplicity in L2φ (G0 ) is 1, and the conjecture 24.2 then says that π0 has a nonzero θ-Fourier coefficient.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

93

Though the above conjecture of Langlands and Arthur is extremely elegant, it has a serious drawback: the group LF is not known to exist. However, in the case of the classical groups considered in this paper, one can present the conjecture on multiplicities in a way that avoids mentioning the group LF . We do this below. In the case of classical groups, there is a further simplification, as the component groups Aφv are all elementary abelian 2-groups. In particular, the representations ηv are all 1-dimensional, so their restricted tensor product η also has dimension 1. Hence the predicted multiplicity mη is either zero or one, the latter case occurring when η has trivial restriction to the diagonal. In the general case, the groups Aφv can be nonabelian, and both the dimension of the representation η and the dimension mη of its Aφ -invariants can be arbitrarily large. We now specialize to the case where G = G(V ) is a classical group. Let G0 = G(V0 ) be the quasi-split inner form. Arguing exactly as we did in the local case, one sees that giving a global L-parameter for G φ : LF −→ L G0 is equivalent to giving a representation ϕ : LF −→ GL(M ) which is selfdual or conjugate-dual with a specific sign b. The requirement that φ is discrete then translates to the requirement that as a representation of LE , M M∼ Mi = i

where each Mi is selfdual or conjugate-dual with the same sign b as M and Mi  Mj if i 6= j. In this case, the global component group is the 2-group Y Aφ = A+ (Z/2Z)Mi )+ , ϕ =( i

where the superscript + is needed only when ϕ is selfdual. Now to remove the mention of the hypothetical group LE , observe that when specialized to the case G = GL(V ), with dim V = n, the Langlands-Arthur conjecture simply says that there is a natural bijection {irreducible cuspidal representations of GL(V )} O  {irreducible n-dimensional representations of LF } . Thus, one may suppress the mention of LF by replacing the latter set with the former. Hence, in the context of the classical groups G(V ), one replaces the data of each Mi by an irreducible cuspidal representation πi of GLni (AE ), with ni = dim Mi . Moreover, in view of Proposition 7.5 and its analog for symplectic and orthogonal groups, the selfduality or conjugate-duality of Mi with sign b can be described invariant theoretically and hence can be captured by the following L-function condition:

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

94

W. T. GAN, B. H. GROSS & D. PRASAD

(a) an irreducible cuspidal representation π of GLn (A) is selfdual of sign ( +1 if its symmetric square L-function L(π, Sym2 , s) has a pole at s = 1; −1 if its exterior square L-function LS (π, ∧2 , s) has a pole at s = 1. (b) an irreducible cuspidal representation π of GLn (AE ) is conjugate-dual of sign ( +1, if the Asai L-function L(π, As+ , s) has a pole at s = 1; −1, if the Asai L-function L(π, As− , s) has a pole at s = 1. To summarize, a discrete global L-parameter ϕ for G0 = G(V0 ) is the data of a P number of inequivalent cuspidal representations πi of GLni (AE ), with i ni = dim M , satisfying the above L-function conditions for each i. The point of this reformulation is that given such a global L-parameter, one still has the data given in (i)–(v) above. More precisely, one has: Q (i) The global component group Aϕ is simply the 2-group i (Z/2Z)πi with a canonical basis indexed by the πi ’s. (ii) For each v, the associated local L-parameter is the representation M ϕv = ϕi,v i

of W D(kv ), where ϕi,v is the local L-parameter of the local component πi,v of πi . The L-function condition presumably forces each ϕi,v to be selfdual or conjugate-dual with the given sign b. Moreover, one has a natural homomorphism Y Aϕ −→ Aϕv = Aϕi,v i

arising from the natural map (Z/2Z)πi → Cϕi,v → Aϕi,v , obtained by sending (Z/2Z)πi to the central subgroup h±1i in the centralizer Cϕi,v . Thus, one continues to have the diagonal map ∆. (iii) For each place v, the local parameter thus gives rise to a local Vogan packet Πϕv as before. (iv) One can now define the global Vogan packet as before. (v) The formula for mη is as given above. The formulation of the Langlands-Arthur conjecture given above amounts to a description of the discrete spectrum of classical groups in terms of the automorphic representations of GLn . The proof has been promised in a forthcoming book [7]. In the remainder of this section, we formulate an extension of the Langlands-Arthur f conjecture to the case of the metaplectic groups Sp(W ). Motivated by Theorem 11.1, one expects that discrete global L-parameters for f Sp(W ) should be discrete global L-parameters for SO(2n+1) with 2n = dim W . Thus,

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

95

f a discrete global L-parameter of Sp(W ) should be a multiplicity free 2n-dimensional symplectic representation of LF : M = M 1 ⊕ · · · ⊕ Mr with each irreducible summand Mi also symplectic. In the reformulation of Lparameters given above, it is thus given by the data of a collection of pairwise inequivalent cuspidal representations πi of GL2ni (A) with P (a) i ni = n and (b) L(πi , ∧2 , s) having a pole at s = 1 for each i. Using Theorem 11.1 and Corollary 11.2, one sees that such a global L-parameter ϕ f continues to give rise to the data (i)–(iv) above in the context of Sp(W ). In particular, one obtains a global Vogan packet Πϕ of irreducible genuine representations of f Sp(W )(A) with a bijection Ç å Y Πϕ ←→ Irr Aϕv . v

However, the multiplicity formula given in (v) above needs to be modified. Motivated by results of Waldspurger in the case when dim W = 2, we make the following conjecture. f Conjecture 25.1. — Let (ϕ, M ) be a discrete global L-parameter for Sp(W ) with associated global Vogan packet Πϕ . Let χϕ be the character on the global component group Aϕ defined by χϕ (a) = (M a , 1/2). More concretely, if ai ∈ Aϕ is the basis element associated to the factor (Mi , πi ) in M , then χϕ (ai ) = (πi , 1/2). Then M f L2ϕ (Sp(W )) ∼ mη πη = η

where mη = h∆∗ (η), χϕ i. We note that Arthur has also introduced nontrivial quadratic characters of the global component group Aϕ in his conjectures for the multiplicities of non-tempered representations of linear groups. We conclude this section with some ramifications of Conjecture 25.1. Given a disf crete global L-parameter (ϕ, M ) (relative to a fixed additive character ψ) for Sp(W ), with dim W = 2n, note that M is also a discrete global L-parameter for SO(V ) with dim V = 2n + 1. For each place v, the elements in the associated Vogan packets ( Πϕv (V ) of SO(V ) f Πϕ (W ) of Sp(W ) v

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

96

W. T. GAN, B. H. GROSS & D. PRASAD

are both indexed by Irr(Aϕv ). For a character ηv of Aϕv , let πηv ∈ Πϕv (V ) and σηv ∈ Πϕv (W ) be the corresponding representations. By construction, πηv and σηv are local theta lifts (with respect to ψ) of each other. One might expect that, globally, the submodule L 2 0 f L2ϕ (Sp(W )) of the discrete spectrum can be obtained from V 0 Lϕ (SO(V )) using global theta correspondence. As we explain below, this is not always the case. Q More precisely, if η = ⊗v ηv is a character of v Aϕv , then the corresponding representations πη ∈ Πϕ (V ) and ση ∈ Πϕ (W ) may or may not be global theta lifts of each other. Indeed, πη occurs in the discrete spectrum of some SO(V 0 ) ⇐⇒ ∆∗ (η) = 1 whereas f ση occurs in the discrete spectrum of Sp(W ) ⇐⇒ ∆∗ (η) = χϕ . Thus, if χϕ is nontrivial, then the subset of η’s which indexes automorphic representations for SO(V ) will be disjoint from that which indexes automorphic representations f of Sp(W ). In such cases, there is clearly no way of obtaining the automorphic elements in Πϕ (W ) from those of Πϕ (V ) via global theta correspondence (with respect to ψ). Suppose, on the other hand, that χϕ is the trivial character, so that (Mi , 1/2) = 1 for all the irreducible symplectic summands Mi of M . Then the automorphic elements in Πϕ (W ) and Πϕ (V ) are indexed by the same subset of η’s and are abstract theta f )) from L2ϕ (SO(V )) via global lifts of each other. However, to construct L2ϕ (Sp(W theta correspondence, there is still an issue with the non-vanishing of global theta liftings. In this case, the non-vanishing of the global theta lifting is controlled by the non-vanishing of the central L-value Y L(M, 1/2) = L(Mi , 1/2). i

f Only when L(M, 1/2) is nonzero does one know that L2ϕ (Sp(W )) can be obtained L from V 0 L2ϕ (SO(V 0 )) by global theta lifting (with respect to ψ). Another observation is that while the packet Πϕ (V ) always contains automorphic elements (for example the representation corresponding to η = 1), it is possible that none of the elements in the packet Πϕ (W ) are automorphic. We give two examples which illustrate Conjecture 25.1 and the phenomena noted above, in the case dim W = 2. In this case, the conjecture is known by the work of Waldspurger [71, 73].

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

97

Example 1. — Suppose that (ϕ, M ) is a discrete global L-parameter for SO(3) ∼ = f PGL(2) and Sp(2), so that Aϕ = Z/2Z. Suppose that for 3 places v1 , v2 and v3 , the local L-parameter ϕvi corresponds to the Steinberg representation of PGL(2), and ϕv is unramified for all other v. Then (M, 1/2) = −1, so that χϕ is the nontrivial character of Aϕ . f In this case, the local Vogan packets (for both SO(3) and Sp(2)) have size 2 at the 3 places vi , and we label the representations by f Πϕv (SO(3)) = {πv+ , πv− } and Πϕv (Sp(2)) = {σv+ , σv− }, with the minus sign indicating the nontrivial character of Aϕv . At all other places, the local packets are singletons. Thus, the global L-packet of SO(3) has 8 elements, f which we can label as π +++ , π ++− and so on. Similarly, the global L-packet for Sp(2) +++ ++− also has 8 elements, denoted by σ ,σ and so on. Now observe that a representation in the global Vogan packet for SO(3) is automorphic if and only if it has an even number of minus signs in its label, whereas a f representation in the global Vogan packet for Sp(2) is automorphic if and only if it has an odd number of minus signs in its label. Example 2. — Suppose again that (ϕ, M ) is a discrete global parameter for SO(3) f and Sp(2), but now assume that ϕv is reducible for all v. Moreover, suppose that (M, 1/2) = −1, so that χϕ is the nontrivial character of Aϕ = Z/2Z. These conditions can be arranged. f In this case, the global Vogan packets for SO(3) and Sp(2) are both singletons, containing the representation π and σ respectively, which are indexed by the trivial character η of Y Aϕv = 1. v

Hence, ∆∗ (η) is the trivial character of Aϕ . In particular, π is automorphic for SO(3), f whereas σ is not automorphic for Sp(2). 26. Revisiting the global conjecture In this section, we shall revisit the global conjecture formulated in §24. In particular, we shall approach the restriction problem using the framework of the LanglandsArthur conjecture reviewed in §25. We start with a pair of spaces W0 ⊂ V0 which gives rise to a quasi-split group G0 = G(V0 ) × G(W0 ) over F and fix an automorphic generic character θ of U as in the previous section; in particular, θ may depend on the choice of an appropriate additive character ψ in various cases. Given a discrete global L-parameter (ϕ, M, N )

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

98

W. T. GAN, B. H. GROSS & D. PRASAD

for G0 , there is a corresponding submodule Aϕ2 in the automorphic discrete spectrum and we are interested in the restriction of the linear functional F (ν) to this submodule. Recall that a natural symplectic representation R of L G plays a prominent role in the global conjecture 24.1. Using the global component group Aϕ of the parameter ϕ, we may refine the associated L-function L(π, R, s) for π ∈ Πϕ , as follows. If a ∈ Aϕ , we may consider it as an element of Aϕv for any v and then choose any semisimple “ by a again, element in Cϕv projecting to it. Denoting any such element in Cϕv ⊂ G we see that the subspace Ra is a representation of W D(Fv ) under ϕv . Thus, one has the associated L-function Y L(π, Ra , s) := L(Ra , s) v

and epsilon factor (π, Ra , s) :=

Y

(Ra , ψv , s).

v

We are now ready to revisit the global restriction problem. Let us first draw some implications of the various local conjectures we have made so far. (i) According to our local conjectures 17.1 and 17.3, there is a unique representation πv in the local Vogan packet Πϕv such that HomH(Fv ) (πv ⊗ νv , C) 6= 0, and this distinguished representation is indexed by a distinguished (relevant) character χv

of AMv × ANv .

For each v, the representation πχv is a representation of a pure inner form Gv of G0 over Fv , associated to a pair of spaces Wv ⊂ Vv . (ii) According to our unramified local conjecture 21.3, for almost all v, the distinguished character χv is trivial and the representations πχv , θv and νv are all unramified. At these places, the pair of spaces Wv ⊂ Vv is simply W0,v ⊂ V0,v and the group Gv is simply G0 (Fv ). Thus, we can form the restricted direct product groups ( Q GA = J v Gv ; Q HA = Jv ∩Hv Hv , and representations ( πχ = “ ⊗v πχv of GA ; “v νv of HA , ν =⊗ which are restricted tensor products, defined using the unique line of Jv -invariant or Jv ∩Hv -invariant vectors for almost all v. The representation πχ is simply the element in the global Vogan packet Πϕ indexed by the distinguished character χ = ⊗v χv of the compact group that

Q

v

AMv × ANv . It is the only (relevant) element in Πϕ such HomH(A) (πχ ⊗ ν, C) 6= 0.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

99

With these preliminaries out of the way, there are now three questions to address. (1) Are HA ,→ GA the adelic points of algebraic groups H = N · G(W ) ,→ G = G(V ) × G(W ) defined over F , associated to a relevant pair of spaces W ⊂ V over E? In the symplectic case, this question clearly has a positive answer, since there is a unique symplectic vector space in any even dimension over any field. Hence, we focus on the other cases, where the issue is whether the collection (Wv ⊂ Vv ) of local spaces is coherent in the terminology of Kudla. Now these local spaces have the same rank as W0 ⊂ V0 and the same discriminant in the orthogonal case. Hence, they form a coherent collection if and only if we have changed the Hasse-Witt invariant or the hermitian/skew-hermitian discriminants at an even number of places v. This is equivalent to the identity Q   v (Mv ⊗ Nv , ψv ), in the orthogonal and hermitian cases; Y 1= χv (−1, 1) =  Q v −1 v (Mv ⊗ Nv (µv ), ψv ), in the skew-hermitian case. Note that since χv (−1, 1) = χv (1, −1) for all v, the coherence condition for the collection of local quadratic spaces {Vv } is the same as that for {Wv }. Thus, we will have a global pair of spaces W ⊂ V with these localizations if and only if (πχ , R, 1/2) = 1. Assuming this is the case, the second question is: (2) Does the representation πχ in the global Vogan packet Πϕ occur in the space A0 (G) of cusp forms? To answer this question, we exploit the Langlands-Arthur conjecture discussed in §25. Thus, in the orthogonal, hermitian and skew-hermitian cases, we need to see if the distinguished character χ is trivial when restricted to the global component group Aϕ via the diagonal map ∆. This amounts to the assertion that, for all a ∈ Aϕ , Q a   v ((Mv ⊗ Nv ) , ψv ), in the orthogonal or hermitian cases; Y 1= χv (a) =  Q v −1 a v (Mv ⊗ Nv (µv )) , ψv ), in the skew-hermitian case, or equivalently, that (πχ , Ra , 1/2) = 1 for all a ∈ Aϕ . On the other hand, in the symplectic case, using the multiplicity formula given in Conjecture 25.1, we see that we need the distinguished character to be equal on Aϕ = AM × A+ N to the character χϕ × 1 (assuming that M is symplectic and N orthogonal), where χϕ is the character of AM defined in Conjecture 25.1. This translates to the same condition Y 1= ((Mv ⊗ Nv )a , ψv ) = (πχ , Ra , 1/2). v

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

100

W. T. GAN, B. H. GROSS & D. PRASAD

Finally, assuming that (πχ , Ra , 1/2) = 1 for all a ∈ Aϕ , so that πχ occurs in the space of cusp forms, we can ask: (3) Does the linear form F (ν) have nonzero restriction to πχ ? The point is that, when the above conditions on epsilon factors hold, there are no trivial reasons for the central critical L-value L(πχ , R, 1/2) to vanish. Here then is the second form of our global conjecture: Conjecture 26.1. — Let πχ be the representation in the global Vogan packet Πϕ corresponding to the distinguished character χ. Then the following are equivalent: (i) πχ occurs with multiplicity one in A0 (G) and the linear form F (ν) is nonzero on πχ (ii) L(πχ , R, 1/2) 6= 0. 27. The first derivative We maintain the notation and setup of the previous section, so that W0 ⊂ V0 is a pair of spaces over E with quasi-split group G0 = G(V0 ) × G(W0 ) over F . For a given discrete global L-parameter (ϕ, M, N ) of G0 , we have a distinguished representation π = πχ = “ ⊗v πv in the global Vogan packet Πϕ , which is a representation of a restricted direct product Y Gv (Fv ) GA = Jv

and is the unique element in the packet such that HomHA (π ⊗ ν, C) 6= 0. In this final section, we specialize to the orthogonal and hermitian cases (i.e. where  = 1) and assume that (π0 , R, 1/2) = −1 so that L(π0 , R, 1/2) = 0, where π0 is the generic automorphic representation of G0 (A) with parameter ϕ. In this case the group GA does not arise from a pair of orthogonal or hermitian spaces W ⊂ V over E. In Kudla’s terminology, the local data (Wv ⊂ Vv ) is incoherent. Nevertheless, we can formulate a global conjecture in this case, provided that the following condition holds: (∗) There is a non-empty set S of places of F , containing all archimedean primes, such that the groups Gv (Fv ) and Hv (Fv ) are compact for all places v ∈ S. This condition has the following implications: (i) dim W0⊥ = 1. Indeed, this follows from the fact that Hv = Nv .G(Wv ) and a nontrivial unipotent subgroup Nv cannot be compact. Hence, for all places v, we have dim Vv = dim Wv + 1.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

101

If we consider the orthogonal decomposition V0 = W 0 ⊕ L over E, then since Wv ⊂ Vv is relevant for all v, we have Lv = Wv⊥ . Thus, though the collection (Wv ⊂ Vv ) is not coherent, the collection (Wv⊥ ) is. (ii) Any archimedean place v of F is real and the space Vv must be definite. In the hermitian case, we must have Ev = C. Hence, in the number field case, F is totally real and, in the hermitian case, E is a CM field. Moreover, at all archimedean places v of F , the generic representation π0,v of G0 (Fv ) is in the discrete series, and πv is a finite dimensional representation of the compact group Gv (Fv ) = SO(n) × SO(n − 1) or U(n) × U(n − 1) with a unique line fixed by H(Fv ) = SO(n − 1) or U(n − 1). (iii) At finite primes v ∈ S, we must have dim(Vv ) ≤ 4. Indeed, a quadratic form of rank ≥ 5 over Fv represents 0. Hence, for function fields F , we have the following nontrivial cases: ( (3, 2) or (4, 3) in the orthogonal case; (dim Vv , dim Wv ) = (2, 1) in the unitary case. For simplicity, we will assume that F is a totally real number field and S consists only of the archimedean places. In the hermitian case, the quadratic extension E of F is a CM field. Suppose first that the spaces Vv are orthogonal of dimension n ≥ 3. Fix a real place α. If Vα has signature (n, 0), let Wα∗ ⊂ Vα∗ be the unique orthogonal spaces over Fα = R with signatures (n − 3, 2) ⊂ (n − 2, 2). If Vα has signature (0, n), let Wα∗ ⊂ Vα∗ have signatures (2, n − 3) ⊂ (2, n − 2). Since we have modified the Hasse-Witt invariant at a single place of F , and kept the discriminant of Wα∗⊥ ' Wv⊥ equal to the discriminant of Lα , there are unique global spaces W α ⊂ V α over E with localizations ( Wv ⊂ Vv for all v 6= α, Wα∗ ⊂ Vα∗ at α. We note that we can make such a pair of global spaces for any place α of F , having localizations Wv ⊂ Vv for all v 6= α, provided that dim Wα ≥ 3. When dim Wα = 2,

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

102

W. T. GAN, B. H. GROSS & D. PRASAD

we can make such a global space provided that Wα is not split over Fα , i.e. for all primes α which are ramified or inert in the splitting field E of the 2-dimensional space W 0 . The proof is similar to [62, Prop. 7]. We can use the global spaces W α ⊂ V α so constructed to define the groups H α ,→ Gα = SO(V α ) × SO(W α ) over F . These have associated Shimura varieties Σ(H α ) ,→ Σ(Gα ) over C, of dimensions n − 3 and 2n − 5 respectively, which are defined over the reflex field E = F , embedded in C via the place α. The varieties over F are independent of the choice of the real place α, so we denote them simply by Σ(H) ,→ Σ(G), suppressing the mention of α. Next, suppose that the spaces Vv are hermitian over Ev of dimension n ≥ 2. Fix a real place α and a complex embedding z : Eα → C. If Vα has signature (n, 0), let Wα∗ ⊂ Vα∗ be the unique hermitian spaces over Eα with signature (n − 2, 1) ⊂ (n − 1, 1). If Vα has signature (0, n), let Wα∗ ⊂ Vα∗ be the unique hermitian spaces over Eα with signatures (1, n − 2) ⊂ (1, n − 1). Again, since we have modified the hermitian discriminants at a single place α of F , and kept (Wα∗ )⊥ ' Wv⊥ constant, there is a unique pair of global spaces Wα ⊂ V α over E with localizations ( Wv ⊂ Vv , for all v 6= α; Wα∗ ⊂ Vα ∗ at α. Again, we can make such a modification at any place α of F which is not split in the quadratic extension E. As before, we use the global spaces W α ⊂ V α to define groups H α ,→ Gα = U(V α ) × U(W α ) over F . These have associated Shimura varieties Σ(H α ) ,→ Σ(Gα ) over C, of dimensions (n − 2) and (2n − 3) respectively, which are defined over the reflex field = E, embedded in C via the extension z of the place α. These varieties

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

103

over E are independent of the choice of real place α of F , so we denote them simply by: Σ(H) ,→ Σ(G). We sketch the definition of the Shimura variety Σ of dimension n − 1 associated to incoherent hermitian data {Wv } of dimension n which is definite at all real places v of F ; the orthogonal case is similar. Take the modified space W α at a real place α, and let G = ResF/Q U(W α ). We define a homomorphism Y h : SR = ResC/R Gm → GR = U(Wvα ) v|∞

as follows. Let he1 , · · · , en i be an orthogonal basis of Wα∗ , such that the definite space e⊥ 1 has the same sign as the definite space Wα . We set à í z/z 1 h(z) =

..

in U(Wα∗ )

. 1

and h(z) = 1 in all the other (compact) components

Y

U(Wv ).

v6=α

Let X be the GR -conjugacy class of h, which is isomorphic to the unit ball Un−1,1 (R)/ [Un−1 (R) × U1 (R)] in Cn−1 . The pair (G, X) satisfies the axioms for a Shimura variety [15, § 2.1]. The composite homomorphism w : (Gm )R → SR → GR is trivial, and the reflex field of Σ(G) = M (G, X) is equal to E, embedded in C via the homomorphism z extending α. Indeed, the miniscule co-character µ : (Gm )C → GC is defined over E: à í α 1 µ(α) =

..

× 1. . 1

The complex points of Σ(G) are: Σ(G, C) = G(Q)\[X × G(“ Q)]. Y Over E, the variety Σ(G) and the action of G(“ Q) = U(Wv ) on it depends only v finite

on the incoherent family {Wv }. If π ∞ = ⊗v

real

πv

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

104

W. T. GAN, B. H. GROSS & D. PRASAD

is any finite dimensional representation of the compact group

Y

U(Wv ), there is a

v real

local system F on Σ(G) over E associated to π∞ . We now return to the study of the L-function L(π0 , R, s) at s = 1/2, using the arithmetic geometry of the cycle Σ(H) ,→ Σ(G) associated to the incoherent family (Wv ⊂ Vv ). The representation Y π∞ = ⊗v real πv of Gv (Fv ) v real

gives a local system F on Σ(G) which contains the trivial local system C when restricted to the cycle Σ(H). To get the appropriate representation πf = “ ⊗v finite πv of G(“ Q) on the Chow group of Σ(G) with coefficients in F , we need to find this representation in the middle dimensional cohomology of Σ(G) with coefficients in F (which is the “tangent space” of the Chow group). Hence we need HomG(b (π , H d (Σ(G), F )) 6= 0 Q) f with d = dim Σ(G). We put ( dim V α , in the hermitian case; n= dim V α − 1, in the orthogonal case, so that n ≥ 2 in all cases. We have   dim Σ(G) = d = 2n − 3 dim Σ(H) = n − 2   codim Σ(H) = n − 1. Now Matsushima’s formula for cohomology shows that dim HomG(b (π , H d (Σ(G), F )) Q) f is equal to the sum of multiplicities in the cuspidal spectrum X m(π α ⊗ (⊗β 6= α real πβ ) ⊗ πf ) · (G(Rα ) : G(Rα )0 ) πα

over the discrete series representations π α of G(Rα ) with the same infinitesimal and central character as πα . If all of the multiplicities are 1, the middle cohomology of Σ(G) with coefficients in F will contain the motive M ⊗ N over F or E, associated to the parameter of the L-packet of π0 . On the other hand, the conjecture of Birch and Swinnerton-Dyer, as extended by Bloch and Beilinson, predicts that dim HomG(b (π , CH n−1 (Σ(G), F )) Q) f

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

105

is equal to the order of vanishing of the L-function L(π0 , R, s) at the central critical point s = 1/2. If the first derivative is nonzero, we should have an embedding, unique up to scaling πf ,→ CH n−1 (Σ(G), F ) and the Chow group of codim(n − 1) cycles plays the role of the space of automorphic forms in §26. The height pairing against the codimension (n − 1) cycle Σ(H), on which F has a unique trivial system, should give a nonzero linear form F : CH n−1 (Σ(G), F ) → C analogous to the integration of automorphic forms over H(F )\H(A). This form is H(“ Q)-invariant, and our global conjecture in this setting is: Conjecture 27.1. — The following are equivalent: (i) The representation πf occurs in CH n−1 (Σ(G), F ) with multiplicity one and the linear form F is nonzero on πf ; (ii) L0 (π0 , R, 1/2) 6= 0. Remark. — Just as the cohomology of a pro-Shimura variety associated to a reductive group G over Q carries an admissible, automorphic action of G(“ Q), it is reasonable to expect that the Chow groups of cycles defined over the reflex field E will also be admissible and automorphic. We note that this is true for the Shimura curves associated to inner forms G of GL2 (Q): the action of G(“ Q) on the Chow group of zero cycles of degree 0 is the Hecke action on the Mordell-Weil group of the Jacobian over Q, which factors through the action of endomorphisms on the differential forms. Here the multiplicity of a representation πf of G(“ Q) on the Chow group in the tower is conjecturally equal to the order of zero of the standard L-function associated to πf at the central critical point. As in the global conjecture in central value case, one expects a refinement of the above conjecture, in the form of an exact formula relating the pairing hΣ(H)(πf ), Σ(H)(πf )i to the first derivative L0 (π0 , R, 1/2). This would generalize the formula of GrossZagier [27], as completed by Yuan-Zhang-Zhang [84], which is the case n = 2 where the codimension of the cycle is 1. Such a refined formula in higher dimensions has been proposed in a recent preprint of W. Zhang [85].

References [1] J. Adams & D. Barbasch – “Genuine representations of the metaplectic group”, Compositio Math. 113 (1998), p. 23–66.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

106

W. T. GAN, B. H. GROSS & D. PRASAD

[2] A. Aizenbud, N. Avni & D. Gourevitch – “Spherical pairs over close local fields”, to appear in Comment. Math. Helvetici. [3] A. Aizenbud & D. Gourevitch – “Multiplicity one theorem for GLn+1 (R), GLn (R)”, Selecta Math. 2 (2009), p. 271–294. [4] A. Aizenbud, D. Gourevitch, S. Rallis & G. Schiffmann – “Multiplicity one theorems”, Annals of Math. 172 (2010), p. 1407–1434. [5] J. Arthur – “On some problems suggested by the trace formula”, in Lie group representations, II (College Park, Md., 1982/1983), Lecture Notes in Math., vol. 1041, Springer, 1984, p. 1–49. [6] , “Unipotent automorphic representations: conjectures”, Astérisque 171-172 (1989), p. 13–71. [7] , “Automorphic representations of classical groups”, in preparation. [8] , “The endoscopic classification of representations: orthogonal and symplectic groups”, preprint. [9] D. Ban & C. Jantzen – “Degenerate principal series for even-orthogonal groups”, Represent. Theory 7 (2003), p. 440–480. [10] N. Bourbaki – Elements of Mathematics, Lie groups and Lie Algebras, Chapters 7–9, Springer, 2005. [11] W. Casselman – “Canonical extensions of Harish-Chandra modules to representations of G”, Canad. J. Math. 41 (1989), p. 385–438. [12] W. Casselman & J. Shalika – “The unramified principal series of p-adic groups. II. The Whittaker function”, Compositio Math. 41 (1980), p. 207–231. [13] P. Deligne – “Les constantes des équations fonctionnelles des fonctions L”, in Modular functions of one variable, II (Proc. Internat. Summer School, Univ. Antwerp, Antwerp, 1972), Springer, 1973, p. 501–597. Lecture Notes in Math., Vol. 349. [14] , “Les constantes locales de l’équation fonctionnelle de la fonction L d’Artin d’une représentation orthogonale”, Invent. Math. 35 (1976), p. 299–316. [15] , “Variétés de Shimura: interprétation modulaire, et techniques de construction de modèles canoniques”, in Automorphic forms, representations and L-functions (Proc. Sympos. Pure Math., Oregon State Univ., Corvallis, Ore., 1977), Part 2, Proc. Sympos. Pure Math., XXXIII, Amer. Math. Soc., 1979, p. 247–289. [16] A. Fröhlich & J. Queyrut – “On the functional equation of the Artin L-function for characters of real representations”, Invent. Math. 20 (1973), p. 125–138. [17] W. T. Gan & G. Savin – “Representations of metaplectic groups I: epsilon dichotomy and local Langlands correspondence”, to appear in Compositio Math. [18] W. T. Gan & S. Takeda – “The local Langlands conjecture for Sp(4)”, International Mathematics Research Notices (2010), p. 2987–3038. [19] , “The local Langlands conjecture for GSp(4)”, Ann. of Math. 173 (2011), p. 1841–1882. [20] S. Gelbart, I. Piatetski-Shapiro & S. Rallis – Explicit constructions of automorphic L-functions, Lecture Notes in Math., vol. 1254, Springer, 1987. [21] D. Ginzburg, D. Jiang & S. Rallis – “On the nonvanishing of the central value of the Rankin-Selberg L-functions”, J. Amer. Math. Soc. 17 (2004), p. 679–722. [22] , “On the nonvanishing of the central value of the Rankin-Selberg L-functions, II”, Ohio State Univ. Math. Res. Inst. Publ. 11 (2005), p. 157–191. [23] , “Models for certain residual representations of unitary groups”, AMS Contemp. Math. 488 (2009), p. 125–146. [24] B. H. Gross & D. Prasad – “On the decomposition of a representation of SOn when restricted to SOn−1 ”, Canad. J. Math. 44 (1992), p. 974–1002.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

[25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36]

[37] [38] [39] [40] [41] [42] [43] [44]

[45] [46] [47]

107

, “On irreducible representations of SO2n+1 ×SO2m ”, Canad. J. Math. 46 (1994), p. 930–950. B. H. Gross & M. Reeder – “Arithmetic invariants of discrete Langlands parameters”, Duke Math. J. 154 (2010), p. 431–508. B. H. Gross & D. B. Zagier – “Heegner points and derivatives of L-series”, Invent. Math. 84 (1986), p. 225–320. M. Harris, S. S. Kudla & W. J. Sweet – “Theta dichotomy for unitary groups”, J. Amer. Math. Soc. 9 (1996), p. 941–1004. M. Harris & R. Taylor – The geometry and cohomology of some simple Shimura varieties, Annals of Math. Studies, vol. 151, Princeton Univ. Press, 2001. N. Harris – “A refined Gross-Prasad conjecture for unitary groups”, Ph.D. Thesis, University of California, San Diego, 2011. G. Henniart – “Une preuve simple des conjectures de Langlands pour GL(n) sur un corps p-adique”, Invent. Math. 139 (2000), p. 439–455. R. Howe – “Transcending classical invariant theory”, J. Amer. Math. Soc. 2 (1989), p. 535–552. A. Ichino & T. Ikeda – “On the periods of automorphic forms on special orthogonal groups and the Gross-Prasad conjecture”, Geom. Funct. Anal. 19 (2010), p. 1378–1425. H. Jacquet, I. I. Piatetskii-Shapiro & J. Shalika – “Rankin-Selberg convolutions”, Amer. J. Math. 105 (1983), p. 367–464. H. Jacquet & S. Rallis – “On the Gross-Prasad conjecture for unitary groups”, Clay Math. Proc. 13 (2011), p. 205–264. H. Jacquet & J. Shalika – “Rankin-Selberg convolutions: Archimedean theory”, in Festschrift in honor of I. I. Piatetski-Shapiro on the occasion of his sixtieth birthday, Part I (Ramat Aviv, 1989), Israel Math. Conf. Proc., vol. 2, Weizmann, 1990, p. 125– 207. D. Jiang, B. Sun & C.-B. Zhu – “Uniqueness of Bessel models: the archimedean case”, Geom. and Funct. Anal. 20 (2010), p. 690–709. S.-i. Kato, A. Murase & T. Sugano – “Whittaker-Shintani functions for orthogonal groups”, Tohoku Math. J. 55 (2003), p. 1–64. M. J. J. Khoury – “Multiplicity-one results and explicit formulas for quasi-split p-adic unitary groups”, Ph.D. Thesis, The Ohio State University, 2008. M.-A. Knus, A. Merkujev, M. Rost & J.-P. Tignol – The book of involutions, AMS Colloquium Publ., vol. 44, 1998. R. E. Kottwitz – “Stable trace formula: elliptic singular terms”, Math. Ann. 275 (1986), p. 365–399. S. S. Kudla – “On the local theta-correspondence”, Invent. Math. 83 (1986), p. 229– 255. , “Splitting metaplectic covers of dual reductive pairs”, Israel J. Math. 87 (1994), p. 361–401. S. S. Kudla & S. Rallis – “On first occurrence in local theta correspondence”, in Automorphic Representations, L-functions and Applications: Progress and Prospects, Ohio State Univ. Math. Res. Inst. Publ., vol. 11, 2005, p. 273–308. P. Kutzko – “The Langlands conjecture for Gl2 of a local field”, Ann. of Math. 112 (1980), p. 381–412. J.-P. Labesse & R. P. Langlands – “L-indistinguishability for SL(2)”, Canad. J. Math. 31 (1979), p. 726–785. Y. F. Liu – “Relative trace formulae toward Bessel and Fourier-Jacobi periods of unitary groups”, preprint arXiv:1012.4538.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

108

W. T. GAN, B. H. GROSS & D. PRASAD

[48] J. W. Milnor & J. D. Stasheff – Characteristic classes, Princeton Univ. Press, 1974, Annals of Mathematics Studies, No. 76. [49] C. Mœglin – “Classification des séries discrètes pour certains groupes classiques padiques”, in Harmonic analysis, group representations, automorphic forms and invariant theory, Lect. Notes Ser. Inst. Math. Sci. Natl. Univ. Singap., vol. 12, World Sci. Publ., Hackensack, NJ, 2007, p. 209–245. , “Classification et changement de base pour les séries discrètes des groupes [50] unitaires p-adiques”, Pacific J. Math. 233 (2007), p. 159–204. [51] C. Mœglin, M.-F. Vigneras & J.-L. Waldspurger – Correspondances de Howe sur un corps p-adique, Springer Lecture Notes, vol. 1291, 1987. [52] C. Mœglin & J.-L. Waldspurger – Spectral decomposition and Eisenstein series, Cambridge Tracts in Mathematics, vol. 113, Cambridge Univ. Press, 1995. [53] , “La conjecture locale de Gross-Prasad pour les groupes spéciaux orthogonaux: le cas général”, preprint. [54] A. Murase & T. Sugano – “Whittaker-Shintani functions on the symplectic group of Fourier-Jacobi type”, Compositio Math. 79 (1991), p. 321–349. [55] , “Shintani functions and automorphic L-functions for GL(n)”, Tohoku Math. J. 48 (1996), p. 165–202. [56] D. Prasad – “Invariant forms for representations of GL2 over a local field”, Amer. J. Math. 114 (1992), p. 1317–1363. [57] , “On the local Howe duality correspondence”, Internat. Math. Res. Notices (1993), p. 279–287. [58] , “Theta correspondence for unitary groups”, Pacific J. Math. 194 (2000), p. 427– 438. [59] S. Rallis – “On the Howe duality conjecture”, Compositio Math. 51 (1984), p. 333–399. [60] J. Rogawski – “Automorphic representations of unitary groups in three variables”, Annals of Mathematics Studies 123 (1990). [61] F. Sauvageot – “Principe de densité pour les groupes réductifs”, Compositio Math. 108 (1997), p. 151–184. [62] J-P. Serre – A course in arithmetic, Grad. Texts in Math., vol. 7, Springer, 1973. [63] T. A. Springer – Linear algebraic groups, second ed., Progress in Math., vol. 9, Birkhäuser, 1998. [64] B. Sun – “Multiplicity one theorems for symplectic groups”, preprint arXiv:0903.1417. [65] , “Multiplicity one theorems: the Fourier-Jacobi models”, to appear in Amer. J. of Math. [66] B. Sun & C.-B. Zhu – “Multiplicity one theorems: the Archimedean case”, Ann. of Math. 175 (2012), p. 23–44. [67] , “A note on special orthogonal groups following Waldspurger”, preprint. [68] W. J. Sweet – “Functional equations of p-adic zeta integrals and representations of the metaplectic group”, preprint, 1995. [69] J. Tate – “Number theoretic background”, in Automorphic forms, representations and L-functions (Proc. Sympos. Pure Math., Oregon State Univ., Corvallis, Ore., 1977), Part 2, Proc. Sympos. Pure Math., XXXIII, Amer. Math. Soc., 1979, p. 3–26. [70] D. Vogan – “The local Langlands conjecture”, AMS Contemporary Mathematics 145 (1993), p. 305–379. [71] J.-L. Waldspurger – “Correspondance de Shimura”, J. Math. Pures Appl. 59 (1980), p. 1–132. [72] , Démonstration d’une conjecture de dualité de Howe dans le cas p-adique, p 6= 2, Israel Mathematical Conference Proceedings, vol. 2, Weizmann Science Press of Israel, 1990.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

109

, “Correspondances de Shimura et quaternions”, Forum Math. 3 (1991), p. 219–

[73] 307. [74] [75] [76] [77] [78] [79]

[80] [81] [82] [83] [84]

[85] [86]

, “Une formule intégrale reliée à la conjecture locale de Gross-Prasad”, Comp. Math. 146 (2010), p. 1180–1290. , “Une formule intégrale reliée à la conjecture locale de Gross-Prasad, 2e partie: extension aux représentations tempérées”, this volume, 2012. , “Une variante d’un résultat de Aizenbud, Gourevitch, Rallis et Schiffmann”, this volume, 2012. , “Calcul d’une valeur d’un facteur epsilon par une formule intégrale”, to appear in Astérisque. , “La conjecture locale de Gross-Prasad pour les représentations tempérées des groupes spéciaux orthogonaux”, to appear in Astérisque. N. R. Wallach – “On the constant term of a square integrable automorphic form”, in Operator algebras and group representations, Vol. II (Neptun, 1980), Monogr. Stud. Math., vol. 18, Pitman, 1984, p. 227–237. , Real reductive groups. II, Pure and Applied Mathematics, vol. 132, Academic Press Inc., 1992. , “C ∞ vectors”, in Representations of Lie groups and quantum groups (Trento, 1993), Pitman Res. Notes Math. Ser., vol. 311, Longman Sci. Tech., 1994, p. 205–270. A. Weil – Basic number theory, third ed., Die Grund. Math. Wiss., vol. 144, Springer, 1974. S. Yamana – “On the Siegel-Weil formula for quaternionic unitary groups”, to appear in American J. of Math. X. Yuan, S. W. Zhang & W. Zhang – “Heights of CM points I: Gross-Zagier formula”, preprint available at http://www.math.columbia.edu/~szhang/papers/HCMI. pdf. W. Zhang – “Relative trace formula and arithmetic Gross-Prasad conjecture”, preprint, 2009. , “Fourier transform and the global Gan-Gross-Prasad conjecture for unitary groups”, preprint http://www.math.columbia.edu/~wzhang/math/online/transfer. pdf, 2011.

f [87] C. Zorn – “Theta dichotomy and doubling epsilon factors for S pn (F )”, Amer. J. Math. 133 (2011), p. 1313–1364.

W. T. Gan, Department of Mathematics, University of California at San Diego, 9500 Gilman Drive, La Jolla, 92093 • E-mail : [email protected] B. H. Gross, Department of Mathematics, Harvard University, Cambridge, MA 02138 E-mail : [email protected] D. Prasad, School of Mathematics, Tata Institute of Fundamental Research, Colaba, Mumbai400005, India • E-mail : [email protected]

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

Astérisque 346, 2012, p. 111–170

RESTRICTIONS OF REPRESENTATIONS OF CLASSICAL GROUPS: EXAMPLES by Wee Teck Gan, Benedict H. Gross & Dipendra Prasad

Abstract. — In an earlier paper, we considered several restriction problems in the representation theory of classical groups over local and global fields. Assuming the Langlands-Vogan parameterization of irreducible representations, we formulated precise conjectures for the solutions of these restriction problems. In the local case, our conjectural answer is given in terms of Langlands parameters and certain natural symplectic root numbers associated to them. In the global case, the conjectural answer is expressed in terms of the central critical value or derivative of a global L-function. In this paper, using methods of base change and the theta correspondence, we test our conjectures for depth zero supercuspidal representations of unitary groups, and for more general representations of groups of low rank. Résumé (Restrictions de représentation de groupes classiques : exemples). — Dans un article précédent, on a considéré certains problèmes de restriction en théorie des représentations de groupes classiques sur un corps local ou global. Admettant que la paramétrisation de Langlands-Vogan pour les représentations irréductibles est établie, on a formulé des conjectures précises concernant les solutions de ces problèmes. Dans le cas local, la solution conjecturale se présente en termes de paramétres de Langlands et de certains facteurs epsilon symplectiques associés à eux. Dans le cas global, la solution conjecturale est exprimée en termes de valeur, au centre de la bande critique, ou de la dérivée de la fonction L globale. Dans l’article présent, on vérifie ces conjectures pour les représentations supercuspidales en niveau zéro de groupes unitaires et pour des groupes plus généraux de rang bas en utilisant des méthodes de changement de base et de correspondence thêta.

1. Introduction This paper is a sequel to [7], where we considered several restriction problems in the representation theory of classical groups over local and global fields. Assuming the Langlands-Vogan parameterization of irreducible representations, we formulated precise conjectures for the solution of these restriction problems. In the local case, our conjectural answer is given in terms of Langlands parameters and certain natural 2010 Mathematics Subject Classification. — 22E50, 22E55, 11F70, 11R39. Key words and phrases. — Gross-Prasad conjectures, unitary groups, depth-zero supercuspidals, theta correspondence.

© Astérisque 346, SMF 2012

112

W. T. GAN, B. H. GROSS & D. PRASAD

symplectic root numbers associated to them. In the global case, the conjectural answer is expressed in terms of the central critical value or derivative of a global L-function. For the precise statements of the restriction problems and our conjectures, we refer the reader to [7]. The conjectures for the case of special orthogonal groups were contained in the earlier papers [13] and [14] and were suggested by the results of Waldspurger [43, 44, 45], Tunnell-Saito [42], [37], and Prasad [27, 28, 29] in certain low rank cases. Since then, there have been further results in the orthogonal case, both locally and globally; see, for example [30], [15], [10], and [34]. Most notably, in a series of recent papers [46, 47, 48, 49] and [24], Waldspurger and Mœglin-Waldspurger have established the local conjectures of [13, 14], assuming certain expected properties of the characters of representations in tempered L-packets. In this paper, we provide some evidence for the conjectures of [7] in the unitary case. More precisely, we shall consider the restriction problems in the following cases: (i) the depth zero supercuspidal L-packets of DeBacker-Reeder [3], which are associated to tame regular discrete L-parameters; (ii) certain low rank cases, such as U(1) × U(1), U(1) × U(2), U(2) × U(2) and U(2) × U(3). We conclude this introduction by summarizing some notations and conventions which are used throughout the paper. Let k be a local field, equipped with a nontrivial involution σ with fixed field k0 . We will always assume that the characteristic of k is not equal to 2. In Section 1, k = C and in Section 2, k is the unramified quadratic extension of k0 , but from Section 6 onwards, k is non-archimedean and there is no restriction on the ramification of k over k0 . We fix a non-zero element δ of k with trace 0 to k0 , so k = k0 + k0 · δ and σ(δ) = −δ. In addition, ψ will denote a non-trivial additive character of k/k0 whereas ψ0 will denote a non-trivial character of k0 . We can pass from a character ψ0 to a character ψ by defining ψ(x) = ψ0 (δ · x) for all elements x ∈ k of trace zero. In particular, this will be how ψ0 and ψ are related in most parts of the paper. We will consider finite-dimensional hermitian or skew-hermitian spaces over k, typically denoted by V in the hermitian case and W in the skew-hermitian case. Given a hermitian space V , we may convert it to a skewhermitian space by multiplying the hermitian form on V by the trace zero element δ; we denote the resulting skew- hermitian space by Wδ . Then one has an identification of the associated isometry groups: U(V ) = U(Wδ ). Acknowledgments. — W. T. Gan is partially supported by NSF grant DMS-0801071. B. H. Gross is partially supported by NSF grant DMS 0901102. D. Prasad was partially supported by a Clay Math Institute fellowship during the course of this work. We thank P. Deligne, S. Kudla, G. Lusztig, M. Reeder, D. Rohrlich, and J.-L. Waldspurger for their help. We also thank the referee for his/her careful reading of the paper and his/her numerous useful comments, corrections and suggestions.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

113

2. Discrete series parameters We begin with the computation of the distinguished character in [7, Conjecture 17.3] χ = χN × χM : AM × AN →< ±1 >, which is defined using local root numbers, for some discrete series parameter for the group G = U(V ) × U(V0 ), where V0 and V hermitian spaces over k, and V0 ⊂ V of odd codimension. In general, these discrete series parameters have the form M M = Mi i

N

M

=

Nj

j

where the Mi are distinct conjugate-symplectic representations and the Nj are distinct conjugate-orthogonal representations of the Weil-Deligne group of k. The dimension of M is even and the dimension of N is odd. In this case, the centralizer CM × CN of the Langlands parameter is finite. We will only consider the case where each Mi = C(αi ) and each Nj = C(βj ) is one dimensional. Then αi is a character of k × /Nk × with αi |k× = ωk/k0 , and βj is a 0

character of k × /k0× . In this case, we have the component groups M AM = Z/2Z · ei M AN = Z/2Z · fj . These vector spaces have dimension equal to dim M and dim N over Z/2Z, which is as large as possible. We have M ei =−1 N

fj =−1

= C(αi ) = C(βj ).

Fix a nontrivial additive character ψ of k which is trivial on k0 . By the definition of the character χ, we have the formulae χ(ei )

=

(C(αi ) ⊗ N, ψ)

χ(fj )

=

(M ⊗ C(βj ), ψ).

Using the additivity of the local epsilon factors, this becomes Y χ(ei ) = (αi βk , ψ) k

χ(fj )

=

Y

(αk βj , ψ).

k

Since the products αi βj are all conjugate-symplectic characters of k × , we need a formula to compute their root numbers. We will do this in two different cases - when

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

114

W. T. GAN, B. H. GROSS & D. PRASAD

k/k0 = C/R, which we take up now, and then when k/k0 is unramified which we do in the next section. Proposition 2.1. — Assume that k0 = R and choose an isomorphism z : k → C. Let α = z −2a · (z z¯)a = (¯ z /z)a be a conjugate-symplectic character of k × , where a is a half integer, and let ψ = e2πiTr(iz) = e2π(¯z−z) Then ( (α, ψ) =

+1 if a > 0; −1 if a < 0.

Proof. — Tate [40, (3.2.5)] gives the formula (α, ψ0 ) = i2a when a > 0 and ψ0 (z) = e2πiTr(z) . Since ψ(z) = ψ0 (iz), we find (α, ψ) = i2a · α(i) = +1 in this case. When a < 0 we must conjugate the isomorphism z : k → C to use Tate’s formula. This changes the character ψ, and hence the sign of . Corollary 2.2. — Assume that k0 = R, choose an isomorphism z : k → C, and let ψ = e2π(¯z−z) . If M is the sum of the distinct symplectic characters αi = (¯ z /z)ai , where each ai is a half integer, and N is the sum of the distinct orthogonal characters βj = (¯ z /z)bj , where each bj is an integer, then χ(ei )

=

(−1)mi

χ(fj )

=

(−1)nj

where mi

=

#{r : ai + br < 0}

nj

=

#{r : ar + bj < 0}.

Finally, we note that in the case when k0 = R, we may order the distinct characters αi and βj in the parameter ϕ so that ( a1 > a2 > a3 · · · ∈ 21 Z − Z b1 > b2 > b3 · · · ∈ Z. Corollary 2.3. — For i < j, we have

ASTÉRISQUE 346

χ(ei )χ(ej )

=

(−1)mij

χ(fi )χ(fj )

=

(−1)nij .

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

115

where mij

=

#{r : ai + br > 0 > aj + br }

nij

=

#{r : bi + ar > 0 > bj + ar }.

Since we know how to describe the representations in the L-packets of discrete series parameters when k0 = R [15], the calculation of χ(ei )χ(ej ) and χ(fi )χ(fj ) allows us to say something about the representation π = π(ϕ, χ) = π1 ⊗ π2 of G(R) with d(π) = 1. The irreducible representations π1 and π2 are discrete series representations of even and odd dimensional unitary groups, with infinitesimal characters a1 > a2 > a3 > · · · b1 > b2 > b3 > · · · in X ? + ρ respectively. Moreover, in the chambers defined by their Harish-Chandra parameters, the simple root walls corresponding to ei − ei+1 is compact

⇐⇒

χ(ei ) · χ(ei+1 ) = −1

fi − fi+1 is compact

⇐⇒

χ(fi ) · χ(fi+1 ) = −1.

More generally, for i < j, the positive root ei − ej is compact

⇐⇒

χ(ei ) · χ(ej ) = (−1)i+j

fi − fj is compact

⇐⇒

χ(fi ) · χ(fj ) = (−1)i+j .

This determines the signature of the unitary group G(R), and in almost all cases the discrete series representation π. We end this section with a remark about branching from U(n, 1) to U(n). According to a theorem of Harish-Chandra, an irreducible admissible (g, K)-module is determined by the action of U (g)K on a given K-type which appears in the representation space. Here, U (g) denotes the universal enveloping algebra of the complexified Lie algebra g of G and U (g)K is the centralizer of K in U (g). Further, the action of K × U (g)K on the corresponding isotypical component is irreducible. By a theorem of Kostant, for G = U(n, 1) and K = U(n) × U(1), U (g)K is generated by the centers of the universal enveloping algebras of G and K, and thus is abelian. This proves that any irreducible representation of U(n) appears with multiplicity at most one in the sum of representations in a given L-packet of U(n, 1) (since all the members of an L-packet have the same infinitesimal character). 3. Depth zero supercuspidals In this section, we test the restriction conjecture for some tamely ramified discrete parameters ϕ of unitary groups. We begin by calculating the local root numbers, assuming that k0 is non-archimedean with residue field Fq and k is the unramified quadratic extension of k0 .

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

116

W. T. GAN, B. H. GROSS & D. PRASAD

Proposition 3.1. — Assume that k0 is non-archimedean, and let k be the unramified quadratic field extension of k0 . Let ψ be an additive character of k which is trivial on both k0 and the maximal ideal of the ring of integers Ak , but is nontrivial on Ak . Let α be a conjugate-symplectic character of k × of conductor f (α). Then (α, ψ) = (−1)f (α)+1 . Proof. — When k/k0 is unramified, every conjugate-symplectic character α has the form α = β · µ, where β : k × /k0× → C× is a conjugate-orthogonal character and µ is the unramified quadratic character of k × (which is conjugate-symplectic). By [7, Section 5] and [6, Theorem 3], we have (β, ψ) = +1 for any character ψ of k which is trivial on k0 . Since µ is unramified, we have [40, (3.4.6)] (α, ψ) = (β, ψ) · µ(π f (β)+n(ψ) ). Since f (β) = f (α) and n(ψ) = −1, this gives the formula in the proposition. Corollary 3.2. — Assume that k0 is non-archimedean. Let k be the unramified quadratic field extension of k0 and µ the quadratic unramified character of k × . Let ψ be an additive character of k which is trivial on both k0 and the maximal ideal of the ring of integers Ak , but is nontrivial on Ak . Let M = ⊕i C(αi )

and

N = ⊕j C(βj )

where the αi ’s are mutually distinct, tamely ramified, conjugate-symplectic characters, and the βj ’s are mutually distinct, tamely ramified, conjugate-orthogonal characters. Order these characters so that α1 β1 = α2 β2 = · · · = αp βp = µ, for p ≥ 0 and αi βj 6= µ for any pair {i, j} with i > p and j > p. Then ( −1 when i ≤ p; χ(ei ) = +1 when i > p. Similarly, ( −1 when j ≤ p; χ(fj ) = +1 when j > p. Finally, χ(−1, 1) = χ(1, −1) = (−1)p . Proof. — Since our characters are all tamely ramified, we find f (αi βj ) = 1,

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

117

unless i = j ≤ p, in which case the product αi βi , i ≤ p, is equal to the unramified character µ and f (αi βi ) = 0. Taking the product of epsilon factors giving χ gives the desired result. We take the parameter M N

=

M

C(αi )

=

M

C(βj )

given by the sum of distinct conjugate-symplectic and distinct conjugate-orthogonal characters of k × . We assume that all of these characters are tamely ramified: f (αi ) = f (βj ) = 1. The L-packet Πϕ of depth zero supercuspidal representations of the pure inner forms of G = U(V ) × U(V0 ) have been constructed by DeBacker and Reeder [3]; we briefly summarize their results in this case. Let V be a hermitian space of dimension n over k. A parameter ϕ of the above type for the unitary group U(V ) = Un gives, by restriction to the units of k × , a regular complex character ρ of the anisotropic torus S = U(1)n (see [3, Section 4.3]). The torus S comes equipped with νi : S → U(1) which are the projections onto the ith factor of S = U(1)n . An embedding ι : S → U(V ) will be called admissible if V is the direct sum of orthogonal lines Li = kvi on which S operates as svi = νi (s)vi , for all s ∈ S. The U(V )-conjugacy class of admissible embeddings ι depends only on the signs i = (−1)ordhvi ,vi i , which must satisfy the one relation Y i = (−1)ord(discV ) . i

Since the two hermitian spaces V and V 0 of dimension n have distinct hermitian discriminants, all the values for i are possible, and hence there are exactly 2n conjugacy classes of admissible embeddings ι of S into U(V ) and U(V 0 ). These conjugacy classes correspond bijectively to the characters χ = χι of the group Aϕ , where χ(ei ) = i . For each embedding ι : S → U(V ), there is a unique maximal compact subgroup Kι ⊂ U(V ) which contains the image. This is the subgroup stabilizing the lattice, M Lι = Ak v i , where we normalize the basis vectors of our S-stable lines to satisfy 0 ≤ ordhvi , vi i ≤ 1. The compact-open subgroup Kι is hyperspecial if and only if all of the inner products hvi , vi i have valuations of the same parity.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

118

W. T. GAN, B. H. GROSS & D. PRASAD

If we define L∨ ι = {x ∈ V |hx, `i ∈ Ak , for all ` ∈ Lι }, then ∨ $L∨ ι ⊂ Lι ⊂ Lι .

The hermitian form on V restricted to Lι gives rise to a non-degenenerate hermitian form on Lι /($L∨ ι ) with values in Ak /$, and the multiple of the hermitian form on V by $ gives rise to a non-degenerate hermitian form on L∨ ι /Lι with values in Ak /$. Thus there is a natural map from Kι to ¯ ι (Fq ) = Ur (Fq ) × Un−r (Fq ), K where r is the number of vi with (−1)ordhvi ,vi i = −1. ¯ ι (Fq ), and the regular tame character ρ of S(Fq ) The torus S(Fq ) embeds in K allows us to construct an irreducible, supercuspidal representation Rι (S, ρ) of the ¯ ι (Fq ), using the method of Deligne and Lusztig. We view this as a finite group K representation of the compact group Kι , and define the representation πχ = πι ,

of U(V )

as the compact induction of Rι (S, ρ). These are the 2n depth zero supercuspidal representation in the L-packet Πϕ . The Vogan bijection between the set Πϕ and the group of homomorphisms from Aϕ to h±1i is normalized as follows. Assume that the hermitian space V is split and even dimensional. Let L be an A-lattice in V with an orthogonal basis whose inner products are units in A. Let NL be the unipotent radical of an Iwahori subgroup of the hyperspecial maximal subgroup K = Aut(L) in U(V ). The construction of [7, §12] over the ring A gives a surjective homomorphism n

f + f0 : NL → A 2 −1 + A− where A− is the eigenspace where σ = −1 on A, which consists of the elements of trace 0 to A0 . By [4], the character χ = 1 of Aϕ corresponds to the unique representation π1 in the L-packet of ϕ which is induced from a generic, cuspidal representation of the reductive quotient Un (Fq ) of K = U(L). All of the generic characters of the unipotent radical N (Fq ) of a Borel subgroup of Un (Fq ) are conjugate, and we construct one of them in the following manner. Let ψ be an additive character of k which is trivial on k0 and the maximal ideal P of A, but is nontrivial on A. Since A is unramified over A0 , we have A0 + A− = 2 · A + A− . Hence, for elements z in A− , the character z 7→ ψ(z/2) is nontrivial on A− /P− . Then the composition n 7→ ψ(Σf (n)) · ψ(f0 (n)/2)

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

119

defines a character of NL which is the inflation of a generic character of N (Fq ) under the natural homomorphism NL → N (Fq ). Hence the representation π1 corresponding to the trivial character of Aϕ is generic for the character obtained by scaling the additive character ψ used in the computation of the root number in Proposition 3.1 and Corollary 3.2 by the factor 1/2, or equivalently by the factor 2. This is the normalization predicted in [7, Conjecture 17.3]. Now consider the parameter of G = U(V ) × U(V0 ) = Un × Um which is given by M M = C(αi ) M N = C(βj ). From the calculation of the character χ = χN × χM of Aϕ in the previous section, we conclude that the irreducible representation πχ of G = U(V ) × U(V0 ) is compactly induced from a maximal compact subgroup with reduction isomorphic to (Ud (Fq ) × Un−d (Fq )) × (Ud (Fq ) × Um−d (Fq )). Here d ≥ 0 is the number of pairs (αi , βi ) with αi βi = µ. The finite dimensional representation that we are inducing has the form (R ⊗ R(α)) ⊗ (R∨ ⊗ R(β)) where R is the Deligne-Lusztig representation of Ud (Fq ) associated to the character (α1 , α2 , . . . , αd ) of the maximal torus U1 (Fq )d and R∨ is its dual representation, associated to the character (β1 , β2 , . . . , βd ) = (α1−1 , α2−1 , . . . , αd−1 ). (We have abused notation here to denote αi , βj now to be characters of U(1)(Fq ) obtained from the corresponding characters of the local field in a natural way.) The remaining representations R(α) of Un−d (Fq ) and R(β) of Um−d (Fq ) are associated to characters whose components αi and βj satisfy αi βj 6= µ for all i, j. As support for [7, Conjecture 17.3], we will prove: Theorem 3.3. — Let πχ be the depth zero supercuspidal representation of G = U(V ) × U(V0 ) defined above, which corresponds to the distinguished character in [7, conjecture 17.3]. Then πχ possesses a Bessel model, in the sense that dim HomH (πχ , ν) = 1 where (H, ν) is as defined in [7, § 12]. To prove the existence of a (unique) Bessel model for πχ , it is sufficient to establish the existence of a Bessel model for the representation R(α) ⊗ R(β) of Un−d × Um−d , as there is clearly a unique Ud × Ud invariant linear form on (R ⊗ R∨ ). We will do this in the following two sections, after first studying the situation for general linear groups.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

120

W. T. GAN, B. H. GROSS & D. PRASAD

4. Branching laws for GLn (Fq ) In this section, we calculate the restriction of a representation of GLn (Fq ) to GLn−1 (Fq ) where GLn−1 (Fq ) sits inside GLn (Fq ) in the natural way as ! A 0 A 7→ . 0 1 These branching laws are surely known in the literature, such as in the work of Thoma [41]; however, we have preferred to give a different independent treatment. We begin by recalling the notion of twisted Jacquet functor. Let P = M · N be any group such that N is a normal subgroup of P and let ϕ be a character of N whose stabilizer in M is denoted by Mϕ . The data (N, ϕ) defines the twisted Jacquet functor from the category of smooth representations of P to the category of smooth representations of Mϕ . It associates to a representation V of P the largest quotient VN,ϕ of V on which N operates via the character ϕ; clearly VN,ϕ is a representation space for Mϕ . The twisted Jacquet functor is exact. Now let En−1 be the mirabolic subgroup of GLn (Fq ) consisting of matrices whose last row is equal to (0, 0, · · · , 0, 1) and let Nn be the group of upper triangular unipotent matrices in GLn (Fq ). We fix a nontrivial character ψ0 of Fq and let ψn be the character of Nn , given by ψn (u) = ψ0 (u1,2 + u2,3 + · · · + un−1,n ). For a representation π of GLn (Fq ), let π i = the i-th derivative of π, which is a representation of GLn−i (Fq ). To recall the definition of π i , if Rn−i = GLn−i (Fq ) · Vi is the subgroup of GLn (Fq ) consisting of matrices ! g v 0

z

with g ∈ GLn−i (Fq ), v ∈ M(n−i)×i , z ∈ Ni , and if the character ψi of Ni is extended to Vi by extending it trivially across M(n−i)×i , then we have π i = πVi ,ψi . If π is an irreducible cuspidal representation of GLn (Fq ), then π i = π for i = 0, and π n = 1, the trivial representation of the trivial group GL0 (Fq ) = {e}. All the other derivatives of π are 0. The following proposition is from Bernstein-Zelevinsky [2, Lemma 4.5], where it was established for non-archimedean local fields, but their proof works for finite fields as well. It is known as the Leibnitz rule for derivatives.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

121

Proposition 4.1. — For π1 a representation of GLn1 (Fq ) and π2 of GLn2 (Fq ), we let π1 × π2 denote the representation of GLn1 +n2 (Fq ) induced from the corresponding representation of the parabolic subgroup with Levi subgroup GLn1 (Fq ) × GLn2 (Fq ). Then there is a composition series of the k-th derivative (π1 × π2 )k whose successive quotients are π1i × π2k−i for i = 0, · · · , k. Here is a generality from Bernstein and Zelevinsky [2, §3.5]. Proposition 4.2. — Any representation Σ of En−1 has a natural filtration of E = En−1 modules 0 = Σ0 ⊂ Σ1 ⊂ Σ2 ⊂ · · · ⊂ Σn = Σ such that n−i Σi+1 /Σi = indE ⊗ ψn−i ) Ri (Σ

for i = 0, · · · , n − 1,

where Ri = GLi (Fq ) · Vn−i is the subgroup of GLn (Fq ) consisting of ! g v 0

z

with g ∈ GLi (Fq ), v ∈ Mi×(n−i) , z ∈ Nn−i , and the character ψn−i on Nn−i is extended to Vn−i by extending it trivially across Mi×(n−i) . As a consequence of the above two propositions, we have the following corollary. Corollary 4.3. — Let n = n1 + · · · + nr be a sum of positive integers, and let πi be an irreducible cuspidal representation of GLni (Fq ) for i = 1, · · · , r. Let Π = π1 × · · · × πr be the corresponding parabolically induced representation of GLn (Fq ). Then the restriction of the representation π1 × · · · × πr of GLn (Fq ) to GLn−1 (Fq ) is a sum of the following representations: πi1 × πi2 × · · · × πis × Σ[n − 1 − (ni1 + · · · + nis )] where 1 ≤ i1 < i2 < · · · < is ≤ r (the empty sequence is allowed) with ni1 +· · ·+nis < n, and GL (F ) Σ[m] = indNmm q ψm denotes the Gelfand-Graev representation of GLm (Fq ), with Σ[1] equal to the regular representation of F× q and Σ[0] denoting the trivial representation of the trivial group. Proof. — By Proposition 4.2, the restriction of Π to En−1 is the sum of E

Πi+1 /Πi = indRn−1 (Πn−i ⊗ ψn−i ). i Since GLn−1 (Fq ) · Ri = En−1 for any i, it follows that (Πi+1 /Πi )|GLn−1 (Fq ) = Πn−i × Σ[n − 1 − i],

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

122

W. T. GAN, B. H. GROSS & D. PRASAD

where Σ[n − 1 − i] is the Gelfand-Graev module of GLn−1−i (Fq ). It only remains to calculate the derivatives Πn−i of Π, but this follows from Proposition 4.1. As a simple consequence of this corollary, we have the following. Theorem 4.4. — Let n = n1 + · · · + nr be a sum of positive integers, and let πi be an irreducible cuspidal representation of GLni (Fq ), for i = 1, · · · , r. Let n − 1 = m1 + · · · + ms be a sum of positive integers, and let µi be an irreducible cuspidal representation of GLmi (Fq ). Assume that the representations µ1 , · · · , µs are pairwise distinct, so that the corresponding parabolically induced representation µ1 × · · · × µs of GLn−1 (Fq ) is irreducible. Then dim HomGLn−1 (Fq ) (π1 × · · · × πr , µ1 × · · · × µs ) is equal to s Y

(1 + di ) ≥ 1,

i=1

where di is the multiplicity with which µi appears in the set {π1 , . . . , πr }. In particular, if the πi ’s are mutually distinct as well, then dim HomGLn−1 (Fq ) (π1 × · · · × πr , µ1 × · · · × µs ) = 2d where d is the cardinality of the set {π1 , . . . , πr } ∩ {µ1 , . . . , µs }. Corollary 4.5. — The restriction of the representation π1 × · · · × πr of GLn (Fq ) to GLn−1 (Fq ) contains the representation µ1 × · · · × µs of GLn−1 (Fq ) (with µi ’s cuspidal and mutually distinct) with multiplicity one if and only if the sets {π1 , · · · , πr } and {µ1 , · · · , µs } have no common elements; in other cases, the multiplicity is a +ve even integer. 5. Branching laws for Un (Fq ) In this section, we use the method of base change, also called Shintani descent, to deduce some conclusions about branching laws for the restriction of a representation of Un (Fq ) to Un−1 (Fq ) from the corresponding results for general linear groups obtained in the previous section. The result is then applied to give a proof of Theorem 3.3. We make crucial use of the multiplicity 1 theorem for restriction of representations of unitary groups over p-adic fields, which was recently proved by Aizenbud, Gourevitch, Rallis and Schiffmann in [1]. A simple consequence of their result is: Proposition 5.1. — Let π1 be an irreducible cuspidal representation of Un−1 (Fq ) and let π2 = IP (σ) be a (possibly reducible) principal series representation of Un (Fq ), where P is a parabolic subgroup of Un (Fq ) and σ is an irreducible cuspidal representation of a Levi

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

123

factor of P . We allow the possibility that P = Un , in which case π2 = σ is cuspidal. Then dim HomUn−1 (Fq ) (π2 , π1 ) ≤ 1. Proof. — Let k0 be a local field with Fq as its residue field and let k be its unramified quadratic extension. Then one can find quasi-split unitary groups U(V0 ) and U(V ) with V0 ⊂ V , such that U(V0 )×U(V ) over k0 contains a hyperspecial maximal compact subgroup K0 × K with reductive quotient Un−1 (Fq ) × Un (Fq ). Moreover, one may find a maximal parabolic subgroup P˜ of U(V ), such that P˜ ∩ K maps to the parabolic P in the reductive quotient Un (Fq ). We commit the usual abuse of notation in denoting by π1 the representation of K0 obtained from the representation π1 of Un−1 (Fq ) through the natural map from K0 to Un−1 (Fq ). Let π ˜1 be the depth zero supercuspidal representation of U(V0 ) which is obtained from π1 by compact induction, so that U(V0 )

π ˜1 = indK0

π1 .

Similarly, let σ ˜ be a depth zero supercuspidal representation of the Levi factor of P˜ which contains σ as a type. Since the center of a Levi subgroup is non-compact, there are many choices for σ ˜ , and we may consider the principal series representation IP˜ (˜ σ) of U(V ) which is irreducible for a generic choice of σ ˜ ; this is possible by a result of Waldspurger, cf. [39]. Moreover, if K1 denotes the kernel of the natural projection map K  Un (Fq ), then one has IP˜ (˜ σ )K1 = IP (σ). Now by Frobenius reciprocity, we have σ ), π1 ) σ ), π ˜1 ) = dim HomK0 (IP˜ (˜ dim HomU(V0 ) (IP˜ (˜ = dim HomUn−1 (Fq ) (IP˜ (˜ σ )K0,1 , π1 ) where K0,1 is the kernel of the projection map K0  Un−1 (Fq ). Since K0,1 ⊂ K1 , we have IP˜ (˜ σ )K0,1 ⊃ IP˜ (˜ σ )K1 = IP (σ) = π2 . Thus we conclude that dim HomU(V0 ) (IP˜ (˜ σ ), π ˜1 ) ≥ HomUn−1 (Fq ) (π2 , π1 ). By [1], the LHS is bounded above by 1 for a generic choice of σ ˜ (so that IP˜ (˜ σ ) is irreducible), and hence so is the RHS. This proves the proposition. Remark 5.2. — We note that the above multiplicity one result for unitary groups over finite fields, proved via known multiplicity one result for p-adic fields, is weaker in some aspect, and stronger in some other aspect, than the corresponding result for p-adic fields. It is weaker since it assumes that the representation π1 of Un−1 (Fq )

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

124

W. T. GAN, B. H. GROSS & D. PRASAD

is cuspidal; it is stronger than the p-adic result in that it does not assume that the representation π2 of Un (Fq ) is irreducible, but only assumes that it is obtained from parabolic induction of an irreducible representation. Presumably such a stronger result should also be true in the p-adic context, and is in fact true if the cuspidal representation π1 of Un−1 is compactly induced, which conjecturally is always the case (for cuspidal representations). A corollary of the above proposition is the uniqueness of Bessel models for cuspidal representations of unitary groups over finite fields. Proposition 5.3. — Let π1 be an irreducible cuspidal representation of Un (Fq ), and let π2 be an irreducible cuspidal representation of Um (Fq ) with n > m but m 6≡ n mod 2. (i) Let P be a maximal parabolic subgroup of Un+1 (Fq ) with Levi factor GLr (Fq2 ) × Um (Fq ) (so that m + 2r = n + 1) and let τ be a cuspidal representation of GLr (Fq2 ). Consider the principal series representation IP (τ  π2 ) of Un+1 (Fq ). Then, with the data (H, ν) defined as in [7, § 12], we have HomH(F ) (π1  π2 , ν) ∼ = HomU (F ) (IP (τ  π2 ), π ∨ ) n

q

q

1

(ii) We have: dim HomH(Fq ) (π1  π2 , ν) ≤ 1. Proof. — (i) This is the finite field analog of [7, Theorem 15.1], with the same proof. (ii) If n = m + 1, (ii) is a special case of Proposition 5.1. In the general case when n > m + 1, we choose τ in the context of (i) so that the induced representation IP (τ  π2 ) is irreducible. Then (ii) follows immediately from (i) and Proposition 5.1. The above propositions allow us to study the restriction problem from Un (Fq ) to Un−1 (Fq ) using Shintani descent. We begin by giving a brief review of this notion. Let G be a connected reductive algebraic group over Fq and let m ≥ 1 be a fixed integer. The group G(Fqm ) comes equipped with its Frobenius automorphism F , whose set of fixed points is G(Fq ). There is a natural map, called the norm mapping, {F -conjugacy classes in G(Fqm )} −→ {conjugacy classes in G(Fq )} which is a bijection. The norm mapping thus induces an isomorphism of vector spaces {class functions on G(Fq )} −→ {F -class functions on G(Fqm )}, which is called the base change map, and whose inverse is called Shintani descent. Furthermore, the base change map is an isometry: hχ01 , χ02 iG(Fqm ) hχ1 , χ2 iG(Fq ) = , #G(Fq ) #G(Fqm ) where χ1 and χ2 are class functions on G(Fq ) which are Shintani descents of the F -class functions χ01 and χ02 on G(Fqm ). Here we have used the standard notation X hf1 , f2 iG = f1 (g)f2 (g −1 ). G

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

125

According to Deligne-Lusztig, given a maximal torus T of G defined over Fq , and a character θ : T (Fq ) → C× , there is a (virtual) representation of G(Fq ) denoted by R(T, θ), which is called a Deligne-Lusztig representation. Now given a character θ as above, one has the character θ0 : T 0 = T (Fqm ) → C× obtained by composing θ with the norm mapping: T (Fqm ) → T (Fq ). Thus one may consider the Deligne-Lusztig representation R(T 0 , θ0 ). The following lemma is [5, 5.16]: Lemma 5.4. — Suppose that G has connected center. Then if R(T, θ) is irreducible, so is R(T 0 , θ0 ). Henceforth, we assume that G has connected center and that R(T, θ) is irreducible. The irreducible representation R(T 0 , θ0 ) is invariant by F and thus can be extended (in m ways) to the semi-direct product G(Fqm ) o hF i. For any such extension, the restriction of its character to the coset G(Fqm ) · F is a F -class function, and one may consider its Shintani descent. The following is a basic fact in the theory of Shintani descent: Proposition 5.5. — There is an extension of the irreducible representation R(T 0 , θ0 ) of G(Fqm ) to G(Fqm ) o hF i whose associated Shintani descent is the representation R(T, θ) of G(Fq ). Now we can begin our study of the restriction problem for unitary groups over finite fields. Let π1

= R(T1 , θ1 )

π2

= R(T2 , θ2 )

be irreducible Deligne-Lusztig representations of Un (Fq ) and Un−1 (Fq ) respectively, and let χi be the character of πi . We shall consider the quadratic base change of πi . By Proposition 5.5, there are extensions of the irreducible representations π10 π20

=

R(T10 , θ10 )

of

GLn (Fq2 ),

=

R(T20 , θ20 )

of

GLn−1 (Fq2 ),

to GLn (Fq2 ) o Z/2 and GLn−1 (Fq2 ) o Z/2 respectively, whose associated Shintani descents are χ1 and χ2 respectively. Fixing such an extension in each case, we denote the corresponding character of this distinguished extension to GLn (Fq2 ) o Z/2 and GLn−1 (Fq2 ) o Z/2 respectively, by χ01 and χ02 . From hχ01 , χ02 iGLn−1 (Fq2 )ohF i = hχ01 , χ02 iGLn−1 (Fq2 ) + hχ01 , χ02 iGLn−1 (Fq2 )·F .

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

126

W. T. GAN, B. H. GROSS & D. PRASAD

we find, hχ01 , χ02 iGLn−1 (Fq2 )ohF i

=

#GLn−1 (Fq2 )

=

hχ01 , χ02 iGLn−1 (Fq2 ) #GLn−1 (Fq2 ) hχ01 , χ02 iGLn−1 (Fq2 )

Equivalently, ô ñ 0 0 hχ1 , χ2 iGLn−1 (Fq2 )ohF i = 2 2#GLn−1 (Fq2 )

#GLn−1 (Fq2 )

+ +

hχ01 , χ02 iGLn−1 (Fq2 )·F #GLn−1 (Fq2 )

,

hχ1 , χ2 iUn−1 (Fq ) . #Un−1 (Fq )

hχ01 , χ02 iGLn−1 (Fq2 ) #GLn−1 (Fq2 )

+

hχ1 , χ2 iUn−1 (Fq ) . #Un−1 (Fq )

Now we observe that: (i) the left hand side of this last equality is an even integer; (ii) the quantity (iii)

hχ01 ,χ02 iGLn−1 (F

q2

)

was computed in Theorem 4.4, under the assump#GLn−1 (Fq2 ) tion that R(T20 , θ20 ) is an irreducible representation; hχ1 ,χ2 iU (Fq ) the quantity #Un−1n−1 is equal to 0 or 1 in certain cases, by Proposition (Fq ) 5.1. hχ1 ,χ2 iU

(F )

q Together, these observations allow one to compute #Un−1n−1 in certain situations. (Fq ) Namely, let us assume that π1 and π2 are irreducible Deligne-Lusztig representations, and suppose further that π2 is cuspidal. Then the quadratic base change π10 and π20 of π1 and π2 are irreducible full principal series representations of GLn (Fq2 ) and GLn−1 (Fq2 ). Thus, Theorem 4.4 implies that ( hχ01 , χ02 iGLn−1 (Fq2 ) 1, if the cuspidal supports of π10 and π20 are disjoint, = #GLn−1 (Fq2 ) an even integer, otherwise.

hχ1 ,χ2 iU

(F )

q On the other hand, by Proposition 5.1, #Un−1n−1 is either 0 or 1. Therefore we (Fq ) get the following theorem as our only option.

Theorem 5.6. — Let π1 and π2 be irreducible Deligne-Lusztig representations of Un (Fq ) and Un−1 (Fq ) respectively, and suppose that π2 is cuspidal. Then dim HomUn−1 (Fq ) (π1 , π2 ) 6= 0 if and only if the cuspidal supports of the base change representations π10 and π20 of GLn (Fq2 ) and GLn−1 (Fq2 ) respectively are disjoint, in which case the Hom space has dimension 1. In particular, this theorem completes the proof of Theorem 3.3. Indeed, in the setting of Theorem 3.3, we need to show that the distinguished representation πχ = π1 × π2 of U(V ) × U(V0 ) satisfies HomH (πχ , ν) 6= 0.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

127

By the argument in the proof of Proposition 5.1, it is sufficient to show that the representation R(α) ⊗ R(β)

of Un−d (Fq ) × Um−d (Fq )

satisfies HomH(Fq ) (R(α) ⊗ R(β), ν) 6= 0. The desired nonvanishing then follows from Proposition 5.3(i) and the above theorem, using the fact that the quadratic base change of R(α) and R(β) have disjoint cuspidal support.

6. Langlands-Vogan packets for small unitary groups The rest of this paper is devoted to verifying [7, Conjecture 17.3] or its variant [7, Conjecture 20.1] in various low rank examples in the hermitian and skew-hermitian cases. In this section, we explicate the Langlands-Vogan parameterization of irreducible representations of U(V ) where V is a hermitian (or skew-hermitian) space over k of dimension ≤ 3. When dimk V = 1, the group U(V ) is naturally isomorphic to the subgroup k 1 of norm one elements in k × , via its scalar action on V . The map x 7→ x/xσ gives an isomorphism of k × /k0× with U(V ). The only other pure inner form of U(V ) is the group U(V 0 ) where V 0 is obtained from V by scaling the hermitian form on V by an element in k0× r Nk × . In this case, an L-parameter for U(V ) is a 1-dimensional conjugate-orthogonal representation M of W D(k), which corresponds via local class field theory to a character of k × /k0× , and hence to characters χM of U(V ) and χ0M of U(V 0 ). The Vogan packet associated to M is then ΠM = {χM , χ0M }. The component group AM is Z/2Z and the trivial character of AM corresponds to the character χM of U(V ). Now consider the case when dim V = 2. We take V to be the split hermitian space, and denote the other rank 2 hermitian space (which is anisotropic) by V 0 . In this case, the groups U(V ) and U(V 0 ) are closely related to the group GL2 and its inner form D× , where D is the unique quaternion division algebra over k0 . More precisely, given a quaternion algebra B over k0 (possibly split), we fix an embedding k ,→ B of algebras over k0 and regard B as a 2-dimensional vector space over k via left multiplication. All such embeddings of k into B are conjugate under Autk0 (B) by the Skolem-Noether theorem. There is an element b ∈ B (of trace zero) which normalizes

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

128

W. T. GAN, B. H. GROSS & D. PRASAD

k and whose conjugation action on k is the involution σ; moreover, all other such elements are of the form λ · b for λ ∈ k. We thus have a decomposition B = k · 1 + k · b. Define a nondegenerate hermitian form on B by hx, yi = projection of x · y onto k · 1, where y 7→ y is the canonical involution on B; let VB be the associated hermitian space. If B is split, then VB is the split hermitian space V , whereas if B is the quaternion division algebra D over k0 , then VB is the anisotropic hermitian space V 0 . The associated unitary similitude group is given by GU(VB ) ∼ = (B × × k × )/∆k × 0

×

with an element (b, t) ∈ B × k

×

acting on B by (b, t)(x) = txb−1 .

The similitude character is given by (b, t) 7→ Nt · Nb−1 , so that U(VB ) = {(b, t) ∈ GU(VB ) : Nb = Nt}. Observe that U(VB ) is a subgroup of GU+ (VB ) = ((B × )+ × k × )/∆k0× , where (B × )+ = {b ∈ B × : Nb ∈ Nk × }. Moreover, it is easy to see that GU+ (VB ) = U(VB ) · ZGU(VB ) , where ZGU(VB ) = (k0× × k × )/∆k0× ∼ = k× is the center of GU(VB ). For later purposes, we describe here a nondegenerate rank 1 hermitian subspace of VB . Consider the nondegenerate subspace LB = k · b ,→ B and observe that its orthogonal complement L⊥ B = k · 1 is isomorphic to h1i. The pointwise stabilizer of L⊥ in U(B) is the diagonal subgroup B ∆ U(LB ) ∼ = k × /k0× −−−−→ (B × × k × )/∆k0× .

We now come to the representation theory of U(VB ). Observing that the L-packets of GU(VB ) are all singletons, we take an L-packet of U(VB ) to be the set of irreducible constituents of the restriction of an irreducible representation of GU(VB ) to U(VB ). Since GU+ (VB ) = U(VB ) · ZGU(VB ) ,

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

129

when considering the restriction of an irreducible representation of GU(VB ) to U(VB ), we may as well consider the restriction problem to GU+ (VB ) in place of U(VB ). Note that if τ  χ is an irreducible representation of GU(VB ) = (B × × k × )/∆k0× , then its restriction to GU+ (VB ) is equal to τ |(B × )+  χ, and it is known that τ |(B × )+ is either irreducible or is the sum of two inequivalent irreducible summands. Moreover, the latter case holds if and only if τ ⊗ ωk/k0 ∼ = τ, in which case we say that τ is dihedral with respect to k/k0 . Then the L-packet of U(VB ) associated to τ is the set ΠB,τ,χ = {(τ α  χ)|U(VB ) : τ α is an irreducible summand of τ |(B × )+ }, which has cardinality 1 or 2. Observe that if µ is any character of k0× , then ΠB,τ ⊗(µ−1 ◦det),χ·(µ◦N) = ΠB,τ,χ . If N is the L-parameter of τ , we also write ΠB,N,χ for ΠB,τ,χ . To attach L-parameters to these packets, recall that an L-parameter in this case is a two dimensional conjugate-symplectic representation M of W D(k). Now we note: Proposition 6.1. — (i) Let τ  χ be an irreducible representation of GU(V ) = (GL2 (k0 ) × k × )/∆k0× , so that ωτ · χ|k× = 1. If N is the L-parameter of τ , then the 0 representation M = N |W D(k) ⊗ χ of W D(k) is conjugate-symplectic. (ii) Conversely, any 2-dimensional conjugate-symplectic representation M of W D(k) arises in this way from an irreducible representation τ  χ of GU(V ), which is welldefined up to twisting by (µ−1 ◦ det)  µ ◦ N for some character µ of k0× . Proof. — By [7, Thm. 8.1], we know that giving a parameter for the unitary group U(V ) is equivalent to giving a 2-dimensional conjugate-symplectic representation M of W D(k). Thus, it suffices to compare the standard description of the L-group of U(V ), or rather GU(V ), with that which arises from the identification GU(V ) ∼ = (GL2 (k0 ) × k × )/∆k × . 0

The L-group of GU(V ) is L

GU(V ) = [GL2 (C) × C× ] o Z/2Z

in which the action of Z/2Z on GL2 (C) × C× is via the automorphism (g, α) → (w0 t g −1 w0−1 , α det g) = ((det g)−1 · g, α · det g), with w0 =

0

1

−1

0

! .

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

130

W. T. GAN, B. H. GROSS & D. PRASAD

On the other hand, the L-group of H = (GL2 (k0 ) × k × )/∆k0× is L

H = [GL2 (C) × {(C× × C× ) o Z/2Z}]1 ,

where the Z/2Z action on C× × C× in L H is via permuting the two factors, and [ ]1 in L H refers to the set of elements (g; α, β) with α · β · det g = 1. The isomorphism H∼ = GU(V ) induces a natural isomorphism L

GU(V ) = [GL2 (C) × C× ] o Z/2Z

−→

L

H = [GL2 (C) × {(C× × C× ) o Z/2Z}]1

given by (g, α) 7→ (gα; α−1 · det g −1 , α−1 ) To complete the proof of the proposition, given a representation τ  χ of H = (GL2 (k0 ) × k × )/∆k0× , we get a Langlands parameter with values in L H which, by the isomorphism above, gives a parameter with values in L GU(V ). Composing this with the natural projection map L

GU(V ) = [GL2 (C) × C× ] o Z/2Z → GL2 (C) o Z/2Z = L U(V )

whose kernel is C× , we get a parameter in L U(V ), and therefore a conjugatesymplectic representation of W D(k). This representation of W D(k) is none other than N |W D(k) ⊗ χ where N is the L-parameter of τ . This proves (i). Conversely, given a parameter for U(V ), we lift it to L GU(V ) using a well-known theorem of Tate (on the vanishing of the 2nd cohomology group of W (k) with values in C× ), and thus obtain a parameter for H. This proves (ii). In view of the above proposition, we set the L-parameter associated to the packet ΠB,τ,χ to be the conjugate-symplectic representation M = N |W D(k) ⊗ χ, with N the L-parameter of τ . Given a conjugate-symplectic M , with associated pair (τ, χ) as in Proposition 6.1(ii), the associated Vogan packet is [ ΠM = · B ΠB,N,χ , where the union is taken over the two quaternion algebras over k0 . Remark 6.2. — It has been shown by Konno-Konno [19] that the above construction of L-parameters agrees with the one supplied by the theory of twisted endoscopy (i.e. base change to GL(2) over k), which has been achieved by Rogawski [35] using the stable trace formula. The following table lists the various possibilities of M , ΠM and the component group AM , depending on the type of τ ’s.

ASTÉRISQUE 346

131

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

τ non-dihedral principal series (with respect to k/k0 ) non-dihedral discrete series (with respect to k/k0 ) dihedral principal series (with respect to k/k0 ) dihedral discrete series (with respect to k/k0 )

M P + σ P ∨, P  σP ∨ irreducible conjugate-symplectic 2 · M 0, 0 M conjugate-symplectic M1 + M2 , M1  M2 conjugate-symplectic

ΠM 1 representation on U(V ) 1 representation on U(V ) and 1 on U(V 0 ) 2 representations on U(V ) 2 representations on U(V ) and 2 on U(V 0 )

AM trivial Z/2Z Z/2Z Z/2Z × Z/2Z

If the conjugate-symplectic representation M is of the last two types in the above table, we shall call M dihedral with respect to k/k0 . If it is of the first two type, we shall call it non-dihedral with respect to k/k0 . From the above table, we see that #ΠM = #AM = #Irr(AM ). To index the representations in ΠM by Irr(AM ), we need to fix a generic character of U(V ) (where V is the split hermitian space). According to [7, Prop. 12.1(2)], a generic character of U(V ) is specified by giving a nontrivial additive character ψ : k/k0 → S1 . We briefly recall how this is done. Let {e, f } be a basis of V such that he, ei = 0 = hf, f i,

he, f i = 1.

This is a unique such basis up to conjugation by U(V ). Let N be the unipotent radical of the Borel subgroup of U(V ) fixing the line spanned by e. Then there is a natural map N → k defined by n 7→ hnf − f, f i, which takes values in the subspace of trace zero elements in k. Composing this map with the non-trivial character ψ : k/k0 → S1 , we get a unitary character θ : N → C× in general position, and the pair (N, θ) is unique up to conjugacy by U(V ), for a fixed choice of ψ. If a representation of U(V ) has nonzero Whittaker model with respect to (N, θ), we shall say that it is ψ-generic. Having fixed ψ : k/k0 → S1 , we then decree that (i) the trivial character of AM corresponds to the ψ-generic element in ΠM ; (ii) a character of AM corresponds to a representation of U(V ) if and only if it is trivial on the image of the central element −1 ∈ L U(V ). From the above table, we see that these requirements completely determine the bijection J(ψ) : ΠM ↔ Irr(AM ), except in the last case, where τ is a dihedral (with respect to k/k0 ) discrete series representation of U(V ) which is a compact unitary group, using the two characters of AM which are nontrivial on the central −1. However, in §8, we shall resolve this issue when we describe an alternative construction of these Vogan packets using theta correspondence.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

132

W. T. GAN, B. H. GROSS & D. PRASAD

Finally, we consider the case when dim V = 3. In this case, the only other pure inner form of U(V ) is the group U(V 0 ) where V 0 is the hermitian space obtained from V via scaling by an element of k0× r Nk × . In this case, the Vogan packets have been defined by Rogawski [35] via base change to GL(3) over k using the stable trace formula. The L-parameters are conjugate-orthogonal representations M of W D(k) of dimension 3. When M is irreducible, the associated Vogan packet is said to be stable; it consists of a representation of U(V ) and the same representation regarded as a representation of U(V 0 ). The component group AM is Z/2Z and we decree that the trivial character correspond to a representation of U(V ). On the other hand, when M is reducible, the associated Vogan packet is said to be endoscopic. In §8, we shall describe a construction of the endoscopic packets, and the labelling of their representations by Irr(AM ), via the approach of theta correspondence.

7. Theta correspondence The goal of this section is to review the necessary background and framework for the theta correspondence for unitary groups. This is necessary for the construction of endoscopic Vogan packets of U(2) and U(3) which will be given in the following section. Let V be a hermitian space and W a skew-hermitian space over k. To consider the theta correspondence for the dual pair U(V ) × U(W ), one requires certain additional data: (i) an additive character ψ0 : k0 → S1 ; (ii) a character µ : k × → C× such that µ|k× = ωk/k0 ; 0 (iii) a trace zero element δ ∈ k × . To elaborate, the tensor product Resk/k0 (V ⊗k W ) has a natural symplectic form defined by hv1 ⊗ w1 , v2 ⊗ w2 i = Trk/k0 (hv1 , v2 iV · hw1 , w2 iW ). Note that many authors (for example [17]) include a factor 1/2 on the right hand side, but we shall not follow this convention here. In any case, there is a natural map i : U(V ) × U(W ) −→ Sp(V ⊗ W/k0 ). One has the metaplectic S1 -cover Mp(V ⊗ W ) of Sp(V ⊗ W ), and the character ψ0 (together with the form h−, −i on V ⊗ W ) determines a Weil representation ωψ0 of Mp(V ⊗W ). To obtain a representation of U(V )×U(W ) from ωψ0 , however, one needs to specify a splitting of the map i to the metaplectic cover. This is quite subtle, but was completely understood by Gelbart-Rogawski [8], Kudla [21] and Harris-Kudla-Sweet [17]; it requires the additional data above. More precisely, the data (V, ψ0 , µ) determines a splitting iV,µ,ψ0 : U(W ) ,→ Mp(V ⊗ W ),

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

133

whereas the data (W, ψ0 , µ, δ) determines a splitting iW,µ,δ,ψ0 : U(V ) ,→ Mp(V ⊗ W ) whose image commutes with that of iV,µ,ψ0 . In [17], such splittings are constructed for any pair of characters (χ, χ0 ) of k × satisfying dim W and χ0 |k× = ωk/k . 0

dim V χ|k× = ωk/k 0 0

0

In their terminology, our splittings are relative to the pair of characters and χ0 = µdim W .

χ = µdim V

In particular, by [17, Corollary A.8], a property of this splitting is that the images of the centers of U(V ) and U(W ) are identified, so that the resulting theta correspondence preserves the central characters. Using the above splittings, one obtains a Weil representation ωψ0 ,µ = ωψ0 ◦ (iW,µ,ψ0 ,δ × iV,µ,ψ0 ) of U(V ) × U(W ), where we have suppressed the data (V, W, δ) from the notation. The Weil representation ωψ0 ,µ depends only on the orbit of ψ0 under Nk × . Thus, given an irreducible representation π of U(W ), we have its big and small theta lift Θψ0 ,µ (π) and θψ0 ,µ (π) on U(V ). By a result of Waldspurger, θψ0 ,µ (π) is either zero or is irreducible when p 6= 2. For the groups of low rank discussed in this paper, one can check that this is true for all p. It would appear that, by restricting (χ1 , χ2 ) (as in [17]) to have the special form taken here, we are losing one degree of freedom. However, this lost degree of freedom can be regained by allowing twisting of the theta lifts by 1-dimensional characters of U(V ), i.e. if we consider θψ0 ,µ (π) ⊗ (χ ◦ det) as well. It is also useful to consider the theta correspondence for similitude groups. Let R ⊂ GU(V ) × GU(W ) be the subgroup consisting of elements (g, h) such that the product of the similitude factors, sim(g)·sim(h) = 1. Then the Weil representation ωψ0 ,µ has a natural extension to R. Now observe that R ⊂ GU+ (V ) × GU+ (W ) where GU+ (V ) consists of those elements g ∈ GU(V ) such that sim(g) lies in the image of the similitude map of GU(W ), and analogously for GU+ (W ). Then one may consider the induced representation GU+ (V )×GU+ (W )

Ωψ0 ,µ = indR +

ωψ0 ,µ

+

of GU (V ) × GU (W ), which depends only on the orbit of ψ0 under Nk × (and is independent of ψ0 in some cases). We can now consider the theta correspondence for GU+ (V ) × GU+ (W ) associated to Ωψ0 ,µ . In particular, for a representation π of GU+ (W ), we have its big and small theta lifts Θψ0 ,µ (π) and θψ0 ,µ (π) on GU+ (V ). In this paper, we will be considering the theta correspondence for U(V ) × U(W ) with | dim V − dim W | ≤ 1. In this case, there are some rather precise conjectures

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

134

W. T. GAN, B. H. GROSS & D. PRASAD

about the behavior of the theta correspondence in the literature (see for example [17, § 7] and [31]). We formulate these as the following working hypothesis. Working hypothesis. — Let V be a hermitian space and let W be a skew-hermitian space, and consider the theta correspondence for U(V ) × U(W ) relative to the data (ψ0 , µ). For an irreducible representation π of U(V ), let θψ0 ,µ (π) denote the (small) theta lift of π to U(W ). (a) If dim V = dim W , then the Langlands parameters of π and θψ0 ,µ (π) are the same (if the latter is nonzero). For a given L-parameter M , the theta correspondence induces a permutation of the Vogan packet ΠM to itself. This bijection is given by translation by a character of the component group AM , as given in [31] in terms of the root numbers of conjugate-symplectic representations of the Weil-Deligne group. (b) If dim V = dim W − 1, then the Langlands parameters M of π and N of θψ0 ,µ (π) are related to each other by: N = µ−1 M + µdim V . The theta correspondence relative to (ψ0 , µ) gives an injection θψ0 ,µ,V,W : ΠV,M ,→ ΠW,N . This injection can be naturally described in terms of the characters of the component groups of M and N as follows. Assume for simplicity that µdim V does not occur in µ−1 M , so that AN = Z/2Z × AM . For an appropriately normalized Langlands-Vogan parameterization, the above injection is described by the natural map Irr(AM ) −→ Irr(AN ) = {±1} × Irr(AM ) given by ρ 7→ (, ρ) where the sign  is completely determined by ρ and the space W . Moreover, as V and W vary over all hermitian and skew-hermitian spaces of the specified dimensions, one has [ ΠN = · V,W θψ0 ,µ,V,W (ΠV,M ), where the union is disjoint and we ignore the theta lifts which are zero. The disjointness of the union means that if V 6= V 0 and W 6= W 0 , then θψ0 ,µ,V,W (ΠV,M ) ∩ θψ0 ,µ,V 0 ,W (ΠV 0 ,M ) = ∅, and, θψ0 ,µ,V,W (ΠV,M ) ∩ θψ0 ,µ,V,W 0 (ΠV,M ) = ∅. While the second statement is part of definitions (since U(W ) and U(W 0 ) are to be considered as different groups, even though they may be isomorphic), the first statement is in fact a consequence of the main result of [17] on theta dichotomy (as extended by [12]).

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

135

In the following, we shall consider the low rank cases, with dim V ≤ 2 and dim W ≤ 3. In these cases, we shall use the above working hypothesis as a guide to label the representations in endoscopic L-packets of U(2) and U(3) which can be constructed using the theta correspondence. We note that these low rank cases are the only ones in which the Langlands-Vogan parameterization is fully understood for U(V ) and U(W ). For example, statement (a) for dim V = 1 is a result of Moen [25], Rogawski [36] and Harris-Kudla-Sweet [17] (see Theorem 9.1 below), whereas the case when dim V = 2 is verified in Theorem 11.2 below. On the other hand, statement (b) for dim V = 1 is easy to check, and the case of dim V = 2 is due to Gelbart-RogawskiSoudry [9].

8. Endoscopic packets and theta correspondence The goal of this section is to describe an alternative construction of the endoscopic packets of the unitary group U(V ), via theta correspondence, when dim V = 2 or 3. We shall rely heavily on the framework and notation of the previous two sections. Our first case of interest is the theta correspondence for a skew-hermitian space W and a hermitian space V with dim W = 1 and

dim V = 2.

We shall use the associated theta correspondence to construct certain Vogan packets on U(V ). Recall that in §6, we have given a construction of the rank 2 hermitian spaces VB in terms of quaternion algebras B over k0 . Suppose that M = M1 + M2 is a 2-dimensional conjugate-symplectic representation of W D(k), with Mi conjugatesymplectic (but not necessarily distinct). As we explained in §6, such an M gives rise to a Vogan packet ΠM of U(VB ). If we fix an additive character ψ : k/k0 −→ S1 then there should be an associated bijection J(ψ) : ΠM ←→ Irr(AM ). It is the Vogan packet ΠM , together with the bijection J(ψ), that we would like to construct using theta correspondence. In fact, since the Vogan packets on U(VB ) are defined by restriction from GU(VB ), it will be better to consider the theta correspondence for the similitude groups GU(W ) × GU+ (VB ), with GU(W ) ∼ = k×

and GU+ (VB ) = ((B × )+ × k × )/k0× .

To set up the theta correspondence, we need to fix the data ψ0 , µ, and the trace zero element δ; these are as in the introduction.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

136

W. T. GAN, B. H. GROSS & D. PRASAD

Let W be a rank 1 skew-hermitian space with discriminant δ, and W 0 the other rank 1 skew-hermitian space. For any a ∈ k0× , let Wa denote the rank 1 skew-hermitian space obtained from W by scaling by a. Finally, with M = M1 + M2 as above, we set µ = M1 , and let χ be any character of k × such that χ/χσ = M1 · M2 . This is possible since M1 · M2 is a character of k × /k0× . The choice of χ is not unique but any two choices differ by a character of k × which is σ-invariant, or equivalently by one that factors through the norm map to k0× . In any case, we have M = M1 + M2 = µ + χ/χσ · µ−1 , and the packet ΠM is obtained by the restriction of τ χ, where τ is the representation of B × with L-parameter W D(k ) N = IndW D(k)0 µχ−1 . Now we may consider the theta correspondence associated to the Weil representation Ωψ0 ,µ of GU(Wa ) × GU+ (VB ). Regarding χ as a character of GU(Wa ), we have the theta lift ΘWa ,VB ,ψ0 ,µ (χ) = θWa ,VB ,ψ0 ,µ (χ) on GU+ (VB ). With B × = GL2 (k0 ), the character ψ determines a generic character of GU+ (VB ). We let τ + be the constituent of τ |GL2 (k0 )+ such that the representation τ +  χ of GU+ (VB ) is ψ-generic, and let τ − denote the other constituent. We also let τ 0 be the Jacquet-Langlands lift of τ to D× , if it exists. With these notations, we have the following proposition which follows by a computation of the Whittaker module of the Weil representation with respect to the maximal unipotent subgroup of U(1, 1); this computation is standard and will therefore be not carried out here. Proposition 8.1. — If B is split, so that VB = V , then ( θψ0 ,µ,V,W (χ) = τ +  χ, θψ0 ,µ,V,W 0 (χ) = τ −  χ. If B is non-split, so that VB = V 0 , then θψ0 ,µ,V 0 ,W (χ) + θψ0 ,µ,V 0 ,W 0 (χ) = τ 0  χ, where the RHS is interpreted as 0 if τ 0 does not exist. In particular, upon restriction to U(V ) or U(V 0 ), the set {θψ0 ,µ,V,W (χ), θψ0 ,µ,V,W 0 (χ), θψ0 ,µ,V 0 ,W (χ), θψ0 ,µ,V 0 ,W 0 (χ)} is the Vogan packet ΠM associated to the L-parameter M = M1 + M2 = µ + µ−1 χ/χσ .

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

137

Using the above construction of endoscopic packets of U(V ), we can define the bijection J(ψ) : ΠM ←→ Irr(AM ), as follows. Consider the case when M1 6= M2 , so that AM = Z/2Z × Z/2Z; this is the only case where the bijection ΠM ↔ Irr(AM ) has some ambiguity. We set   π ++ = θψ0 ,µ,V,W (χ)    π −− = θ 0 (χ) ψ0 ,µ,V,W

 π +− = θψ0 ,µ,V 0 ,W 0 (χ)    π −+ = θ ψ0 ,µ,V 0 ,W (χ). In other words, the recipe for labelling is that π 1 ,2 = θψ,µ,VB ,Wa (χ) where ( 1 · 2 = (B) =

1 if B is split; −1, if B is not split,

and 2 = ωk/k0 (a). Equivalently, if η is a a character of AM , then πη = θψ,µ,VB ,Wa (χ) if and only if ( η(a1 ) = (B) · ωk/k0 (a), η(a2 ) = ωk/k0 (a). We leave it to the reader to verify that under this system of bijections J(ψ), the various desiderata of the Vogan parameterization listed in [7, § 9 and § 10] are satisfied. In particular, the trivial character of AM corresponds to the unique ψgeneric representation of the packet, and if ψ 0 belongs to the other Nk × -orbit, then the unique ψ 0 -generic representation corresponds to the character η0 (ai ) = (−1)dim Mi . Indeed, when M is irreducible, η0 is trivial, whereas when M = M1 + M2 is reducible, then η0 is the character (−−) of AM = Z/2Z × Z/2Z. It will be useful to convert the above classification into the setting of rank 2 skewhermitian spaces. Using the trace zero element δ, let WB,δ be the skew-hermitian space obtained from VB by scaling by δ; we shall frequently write WB for WB,δ . Then we have GU(WB ) = GU(VB ) as subsets of Endk (B). Moreover, the notions of L-parameters and L-packets are the same for U(VB ) and U(WB ). The only difference lies in the data needed to specify a

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

138

W. T. GAN, B. H. GROSS & D. PRASAD

bijection of a Vogan packet with the set of characters of the component group. In the case of VB , we used an additive character ψ : k/k0 −→ S1 , whereas for the case of WB , one needs an additive character of k0 . However, it is easy to check that if a representation π of U(VB ) is generic with respect to ψ, then regarded as a representation of U(WB ), π is generic with respect to the character ψ0 of k0 which we have fixed, and the bijection J(ψ) : ΠM ←→ Irr(AM ) for U(VB ) is the bijection J(ψ0 ) for U(WB ). For a character η of AM , we then have πη = θψ0 ,µ,Va ,WB (χ) where µ and χ are obtained from M as before, Va is the rank 1 hermitian space with discriminant a, and ( η(a1 ) = (B) · ωk/k0 (a) η(a2 ) = ωk/k0 (a). Let N be a conjugate-symplectic representation of W D(k) of dimension 2 considered as an L-parameter for U(WB ). Let ΠN be the Vogan packet associated to N , together with the bijection J(ψ0 ) : ΠN ←→ Irr(AN ) associated to the additive character ψ0 . Then for η ∈ Irr(AN ), we may consider the theta lift θψ0 ,µ,WB ,Va (πη ), where πη ∈ ΠN is the representation of U(WB ) (this uniquely specifies B) indexed by η under J(ψ0 ). As the element a varies over the two representatives of k0× /Nk × , and the character η varies over Irr(AN ), we obtain a collection of 2 · #ΠN representations (some of which might be zero). It was shown by Gelbart-Rogawski-Soudry [9] that the set of representations so obtained is the Vogan packet associated to the endoscopic parameter M given by: M = µ2 + N · µ−1 . The following lemma, which was shown in [9], addresses more precisely the issue of nonvanishing of these theta lifts. Lemma 8.2. — Let M = M1 + M2 = µ2 + N · µ−1 as above. If M  3M1 , assume without loss of generality that M1 is distinct from any irreducible constituent of M2 . (i) If M  3M1 , then the representations θψ0 ,µ,Va ,WB (πη ) are always nonzero. (ii) If M = 3M1 , then N = 2 · µ3 and AN ∼ = Z/2Z, so that we may regard η = ±1, depending on whether η is trivial or not. The representation θψ0 ,µ,Va ,WB (πη ) is nonzero if and only if ωk/k0 (discVa ) = η.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

139

In each case above, the non-zero representations are mutually distinct. Moreover, the representation θψ0 ,µ,Va ,WB (πη ) is generic if and only if πη is generic with respect to ψ0,disc(Va ) . We may now define a labeling of the elements in ΠM by the irreducible characters of AM . (i) If M  3M1 , and M1 does not occur in M2 , then AM = AM1 × AM2 = AM1 × AN . For a character χ = (, η) ∈ Irr(AM1 ) × Irr(AN ), we set π χ = π ,η = θψ0 ,µ,WB ,Va (πηV ) with  · η(−1) = ωk/k0 (a), and ηV =

( η, if ωk/k0 (discV ) = 1; η · ηN,0 , if ωk/k0 (discV ) = −1,

where ηN,0 is the character of AN which indexes the ψ 0 -generic element of ΠN (where ψ 0 is a character of k/k0 which is not in the Nk × -orbit of ψ). More simply, when disc(V ) = 1, we have χ(a1 ) = ωk/k0 (a) · η(−1) = ωk/k0 (a) · (B) & χ|AM2 = η. In particular, for a character χ of AM = AM1 × AN , πχ is a representation of U(V ) if and only if χ(−1, −1) = 1. (ii) If M = 3M1 = 3µ2 , then AM ∼ = AN = Z/2Z. For a character η = ± of AM , we set π η = θψ0 ,µ,Va ,WB (πη·ωk/k0 (discV ) ) with ωk/k0 (a) = η. By part (ii) of the above lemma, this condition ensures that the theta lift above is nonzero. In particular, the trivial character of AM corresponds to a representation of U(V ) whereas the nontrivial character corresponds to the same representation regarded on U(V 0 ). Note that since dim V = 3, there is only one orbit of generic characters for U(V ), and hence the Vogan parameterization in this case is canonical. So it is instructive to observe that the above parameterization is independent of the choice of ψ0 (or equivalently (ψ, δ)). We leave this to the reader, as well as the verification that the

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

140

W. T. GAN, B. H. GROSS & D. PRASAD

above definition satisfies the desiderata of the Vogan parameterization listed in [7, § 9 and § 10].

9. Skew-hermitian case: U(1) × U(1) Having explicated the Langlands-Vogan parameterization of the unitary groups U(V ) with dim V ≤ 3, we are now in a position to verify instances of [7, Conjecture 17.3]. We begin with the case when W0 = W are skew-hermitian spaces with dim W0 = dim W = 1. Let W 0 be the other skew-hermitian space of dimension 1. In this case the following result from [17, Corollary 8.5] is equivalent to our conjecture: Theorem 9.1. — For each a ∈ k0× , let Wa be the rank 1 skew-hermitian space with discriminant a · δ, and for each b ∈ k0× , let Vb be the rank 1 hermitian space with discriminant b. Given a character η of k × /k0× , which can be regarded as a character of U(Wa ), we have HomU(Wa ) (η, ωWa ,Vb ,ψ0 ,µ ) 6= 0 ⇐⇒ (η · µ−1 , ψ0 (Tr(δ−))) = ωk/k0 (a · b). Remark 9.2. — We note that our convention here differs from [17] in two aspects. Namely, we have adopted the convention that on Wa ⊗ Vb , the symplectic form is Tr(h−, −iWa ⊗ h−, −iVb ). In [17], the symplectic form is 1 · Tr(h−, −iσWa ⊗ h−, −iVb ). 2 Besides the factor of 1/2, the skew-hermitian form on Wa is conjugated by σ, which is necessitated by the convention adopted by [17] that skew-hermitian forms are linear in the second variable and hermitian forms are linear in the first variable. Conjugating the form on Wa by σ has the effect of replacing δ by −δ in [17, Corollary 8.5]. To apply the above theorem to [7, Conjecture 17.3], set η = α ·β for α, β characters of U(1), in the theorem, and note that the distinguished character χ0 of AM × AN = Z/2Z × Z/2Z given in [7, Conjecture 17.3] satisfies χ0 (−1, 1) = χ0 (1, −1) = (M ⊗ N (µ−1 ), ψ0 (Tr(δ−))). Thus, Theorem 9.1 implies that χ0 is trivial ⇐⇒ HomU(W ) (α · β, ωW,ψ0 ,µ ) 6= 0 and χ0 is nontrivial ⇐⇒ HomU(W 0 ) (α0 · β 0 , ωW 0 ,ψ0 ,µ ) 6= 0. This verifies [7, Conjecture 17.3] for this case.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

141

10. Restriction from U(2) to U(1) In this section, we consider the restriction problem from U(2) to U(1). This problem has been studied by H. Saito [38] and T. Konno [20], but we shall give an independent treatment here and relate the result to [7, Conjecture 17.3]. Recall that in §6, we have given a construction of rank 2 hermitian spaces VB using quaternion algebras B over k0 , together with a non-degenerate rank 1 subspace: LB ,→ VB , such that L⊥ B = h1i. When B is split, this gives a pair of split hermitian spaces L ⊂ V , with disc(L) = −1. On the other hand, if B is the quaternion division algebra D, one obtains a relevant pair L0 ⊂ V 0 with V 0 anisotropic. The groups G = G(V ) × G(L) and G0 = G(V 0 ) × G(L0 ) are relevant pure inner forms of each other. Suppose that M is a conjugate-symplectic 2-dimensional representation of W D(k), with component group AM , so that M determines a Vogan packet ΠM of U(V ). In this section we will be interested in determining HomU(LB ) (πB ⊗ η, C) for πB ∈ ΠM,B and η the character of U(LB ) corresponding to N . Since the embedding U(LB ) ,→ U(VB ) ⊂ GU + (VB ) is given by the diagonal map k × /k0× ,→ (B × × k × )/∆k0× , we see that M M

HomU(LB ) (πB ⊗ η, C) = Homk× (τ, χ−1 η −1 ) + Homk× (τ 0 , χ−1 η −1 ).

B πB ∈ΠM,B

Now we note the following theorem of Waldspurger [45], Tunnell [42] and Saito [37]: Theorem 10.1. — Let τ be an irreducible admissible representation of GL2 (k0 ) with L-parameter N (τ ) and Jacquet-Langlands lift τ 0 on D× . For any character ν of k × , with ν|k× = ωτ , we have 0

dim Homk× (τ, ν) + dim Homk× (τ 0 , ν) = 1. Moreover, Homk× (τ, ν) 6= 0 ⇐⇒ (N (τ )|W D(k) ⊗ ν −1 , ψ) = 1, where ψ is any non-trivial character of k/k0 .

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

142

W. T. GAN, B. H. GROSS & D. PRASAD

Applying this theorem to the case at hand, with ν = χ−1 · η −1 , we immediately deduce [7, Conjecture 17.1] (multiplicity one in L-packets). In fact, when τ is not dihedral with respect to k/k0 , this theorem also implies [7, Conjecture 17.3]. Indeed, in this case, τ  χ remains irreducible when restricted to U(V ), so that 0 ΠM = {πM , πM }.

Moreover, AM ∼ = AN ∼ = Z/2Z and the distinguished character χ0 of AM ×AN satisfies χ0 (−1, 1) = χ0 (1, −1) = (N (τ )|W D(k) ⊗ χ · η, ψ). Hence we deduce that χ0 is trivial ⇐⇒ HomU(L) (πM ⊗ η, C) 6= 0 and 0 χ0 is nontrivial ⇐⇒ HomU(L0 ) (πM ⊗ η, C) 6= 0. Suppose then that τ is dihedral with respect to k/k0 , so that

N (τ )|W D(k) = α + ασ for a character α of k × . In this case, τ is the sum of two distinct irreducible summands when restricted to GL2 (k0 )+ and the same holds for its Jacquet-Langlands lift τ 0 (if it exists). A refinement of Theorem 10.1 was obtained in the paper [29] of the third author, as well as in [38]. However, the results in the papers [29] and [38] fall slightly short of establishing [7, Conjecture 17.3]. The rest of this section completes the analysis of [29] and [38], thus proving [7, Conjecture 17.3]. When τ is dihedral with respect to k/k0 , we have M = M1 + M2 with Mi conjugate-symplectic (not necessarily distinct). Using theta correspondence, we have described in §8 a construction of the packet ΠM as well as a bijection J(ψ) : ΠM ↔ Irr(AM ), depending on an additive character ψ of k/k0 . Thus, if M1 6= M2 , then each element π 1 ,2 of ΠM is specified by a pair of signs (1 , 2 ). Similarly, If M1 = M2 , then ΠM contains two representations π ++ and π −− . In either case, the representation π ++ is the unique ψ-generic representation in ΠM . Here is the main theorem of this section, which completes the verification of [7, Conjecture 17.3]. Theorem 10.2. — Suppose that VB = LB ⊕ L1 is a 2-dimensional hermitian space, where L1 is a hermitian line with discriminant 1 and ωk/k0 (−disc(LB )) = (B). Suppose that M = M1 +M2 is an L-parameter of U(VB ) with Mi conjugate-symplectic, and let ΠM be its associated Vogan packet and AM its component group. Let ψ be a non-trivial character of k/k0 , which induces a bijection J(ψ) : ΠM ↔ Irr(AM ). Then for any character η of U(LB ), HomU(LB ) (π 1 ,2 ⊗ η, C) 6= 0 if and only if (M1 ⊗ η, ψ2 ) = 1

ASTÉRISQUE 346

and

(M2 ⊗ η, ψ2 ) = 2 ,

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

143

where ψ2 (x) = ψ(2x). Remark. — Note that when M1 = M2 , then there are no representations on the anisotropic U(V 0 ) to consider, and the two root numbers in question must have the same sign. Proof. — We assume that M1 6= M2 , since the case M1 ∼ = M2 is similar. Then we have AM = Z/2Za1 × Z/2Za2 . Let us first recall the construction of the associated packet ΠM and the bijection J(ψ) : ΠM ↔ Irr(AM ) Setting µ = M1

and χ/χσ = M1 · M2 ,

the packet ΠM consists of the representations (with B, c varying): π 1 ,2 = θψ0 ,µ,VB ,Wc (χ) where B’s are the two quaternion algebras over k0 considered as hermitian spaces over k; Wc is the rank 1 skew-hermitian space of discriminant cδ; and ψ is related to ψ0 as everywhere else in the paper by the identity ψ(x) = ψ0 (δx) for all trace zero elements x of k. Moreover, the bijection J(ψ) is specified by: (∗)

1 = (B) · ωk/k0 (c) and 2 = ωk/k0 (c).

Now consider the seesaw diagram U(LB + L1 )

U(Wc ) × U(Wc )

U(LB ) × U(L1 )

∆U(Wc ).

We start with the character χ on ∆U(Wc ) and the character η −1 on U(LB ), and consider the theta correspondence with respect to the additive character ψ0 . Then the seesaw identity gives HomU(LB ) (π 1 ,2 , η −1 ) = HomU(Wc ) (θψ0 ,µ,Wc ,LB (η −1 ) ⊗ ωψ0 ,µ,Wc , χ). Hence, HomU(LB ) (π 1 ,2 , η −1 ) 6= 0 if and only if the following two conditions hold: (a)

θψ0 ,µ,Wc ,LB (η −1 ) 6= 0,

in which case, θψ0 ,µ,Wc ,LB (η −1 ) = η −1 ; and (b)

HomU(Wc ) (η −1 ⊗ ωψ0 ,µ,Wc , χ) 6= 0.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

144

W. T. GAN, B. H. GROSS & D. PRASAD

But both (a) and (b) are special cases of Theorem 9.1 [17, Corollary 8.5]. We deduce that (a) holds if and only if (µ−1 η −1 , ψ0 (Tr(δ−)) = ωk/k0 (disc(LB )) · ωk/k0 (c) or equivalently (c)

(M1 ⊗ η, ψ2 ) = ωk/k0 (−discLB ) · ωk/k0 (c) = (B) · ωk/k0 (c).

Similarly, (b) holds if and only if (µ−1 · η · χ/χσ , ψ0 (Tr(δ−))) = ωk/k0 (c), or equivalently (d)

(M2 ⊗ η, ψ2 ) = ωk/k0 (c).

In view of (∗), the theorem is proved.

11. Theta correspondence for U(2) × U(2) Before moving on to the next case of [7, Conjecture 17.3], we need to establish some results about the theta correspondence for U(2) × U(2). More precisely, let VB be the rank 2 hermitian space introduced in §6, and let WB 0 be the rank 2 skew-hermitian space obtained from VB 0 by scaling by the trace zero element δ ∈ k × fixed in the introduction. In this section, we will be interested in establishing the precise theta correspondence for the dual pair U(VB ) × U(WB 0 ) relative to the data (ψ0 , µ, δ). The first result is the following proposition due to Harris [16, Lemma 4.3.3] and Konno-Konno [19, Prop. 5.3 and Thm. 5.4]. Proposition 11.1. — Let M be a 2-dimensional conjugate-symplectic representation of W D(k) which gives rise to a L-packet ΠM,B for U(VB ) and ΠM,B 0 for U(WB 0 ). (i) For any π ∈ ΠM,B , θψ0 ,VB ,WB0 ,µ (π) 6= 0 ⇐⇒ (M ⊗ µ−2 , ψ) = (B) · (B 0 ). Note that the root number above is independent of the choice of the additive character ψ of k/k0 . (ii) If the condition of (i) holds, then θψ0 ,VB ,WB0 ,µ (π) belongs to ΠM,B 0 . In other words, the theta correspondence is the identity map on L-parameters. Thus, under the theta correspondence for (ψ0 , µ, δ), there is a unique B 0 such that the theta lift gives a bijection ΠM,B ←→ ΠM,B 0 .

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

145

If the parameter M is non-dihedral (with respect to k/k0 ), then #ΠM,B = 0 or 1. Hence the above proposition completely determines the theta lift of the representations in ΠM . When M is dihedral with respect to k/k0 , then #ΠM,B = 0 or 2, and in the latter case, there are two possible bijections ΠM,B ←→ ΠM,B 0 , which the above proposition does not resolve. In [31], the third author has formulated a precise conjecture addressing this issue. The following theorem confirms the conjecture in [31] for this case: Theorem 11.2. — Suppose that M = M1 + M2 is dihedral with respect to k/k0 . Fix the additive character ψ of k/k0 which gives bijections ΠM ←→ Irr(AM ), and let ψ0 be the additive character of k0 such that ψ is related to ψ0 as everywhere else in the paper by the identity ψ(x) = ψ0 (δ · x) for all trace zero elements of k. Then the permutation of ΠM induced by the theta correspondence associated to (ψ0 , µ, δ) is given by multiplication by the character ρ0 of AM defined by ρ0 (ai ) = (Mi ⊗ µ−2 , ψ2 ) with ψ2 (x) = ψ(2x) = ψ0 (Tr(δx)). Proof. — Consider first the case where B 0 is split whereas B is arbitrary. In this case, the two elements in ΠM,B 0 can be distinguished by the Whittaker models they support. Computing Whittaker models of the Weil representation ωψ0 ,VB ,WB0 ,µ , one sees that for πρ ∈ ΠM,B , the representation θψ0 ,VB ,WB0 ,µ (πρ ) of U(WB 0 ) is ψ0 -generic if and only if HomU(LB ) (πρ∨ , µ−2 ) 6= 0. By the result of the previous section, this holds if and only if ρ(a1 ) = (M1 ⊗ µ−2 , ψ2 ) and ρ(a2 ) = (M2 ⊗ µ−2 , ψ2 ), as desired. This establishes the result when one of B or B 0 is split. The only remaining case is where B and B 0 are both non-split, so that (M1 ⊗ µ−2 , ψ2 ) · (M2 ⊗ µ−2 , ψ2 ) = 1. In this case, the desired result can be proved by a global method. We give a brief sketch of this. Let π be a representation in ΠM,B , so that θψ0 ,µ (π) also belongs to ΠM,B . We have: Proposition 11.3. — Using the above notations, one can find: 1. a totally real number field F of odd degree over Q and such that Fv0 = k0 for some finite place v0 of F ; 2. an additive character Ψ of AF /F such that Ψv0 = ψ0 ; 3. a totally imaginary quadratic extension E of F such that Ev0 ∼ = k;

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

146

W. T. GAN, B. H. GROSS & D. PRASAD

4. a trace zero element ∆ ∈ E such that ∆v = δ up to Nk × ; 5. an idele class character Σ of AE × such that Σv0 = µ and Σ|A× = ωE/F ; F 6. a quaternion algebra B over F ramified precisely at v0 and all the infinite places, so that Bv0 = B; this gives a hermitian space VB over F which is isomorphic to VB over Fv0 ; 7. a cuspidal representation Π of U(VB ) such that (a) Πv0 = π; (b) Π belongs to a global endoscopic packet (i.e. the base change of Π to E is non-cuspidal); (c) L(BCE/F (Π) ⊗ Σ−2 , 1/2) 6= 0 Proof. — One can certainly find the number fields F and E satisfying (1) and (3) (see Lemma 15.3 below), after which one can find Ψ as in (2), ∆ as in (4), Σ as in (5) and B as in (6). With these objects fixed, we need to find a cuspidal representation Σ as in (7). Clearly, there is no difficulty in find Π satisfying (7a) and (7b). The main difficulty is to find Π which satisfies (7c) as well. Recall that the representation π is a summand in the restriction of a representation τ  χ of (B × × k × )/∆k0× , so that ωτ · χ|k× = 1 and the L-parameter of π is the L0 parameter of the representation BC(τ ) ⊗ χ of B ⊗k0 k ∼ = GL2 (k). The fact that π is dihedral means that τ is dihedral, so that BC(τ ) = α ⊕ ασ for some character α of k × , so that M = M1 + M2 = αχ + ασ χ. Before commencing the construction of Π, we recall that we are assuming that (BC(τ ) ⊗ χµ−2 ) = (M1 ⊗ µ−2 , ψ2 ) · (M2 ⊗ µ−2 , ψ2 ) = 1. By Tunnell-Saito [37, 42], this condition implies that Homk× (τ, χ−1 · µ2 ) = 0, and if JL(τ ) is the Jacquet-Langlands lift of τ to GL2 (k0 ), then Homk× (JL(τ ), χ−1 · µ2 ) 6= 0. By globalizing the character α of k × , one can find a dihedral cuspidal representation T of GL2 (AF ) such that Tv0 = JL(τ ). Then using [P6, Lemma 1], one can find a character C of A× E such that Cv0 = χ and such that T is globally distinguished by C · Σ2 ; necessarily we have ωT · C|A× = 1. Then by Waldspurger [44], one concludes F that L(BC(T) ⊗ CΣ−2 , 1/2) 6= 0. Now let Π = JLB (T)  C−1

on U(VB ),

so that L(BC(Π) ⊗ Σ−2 , 1/2) = L(BC(T) ⊗ CΣ−2 , 1/2) 6= 0. This completes the construction of Π.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

147

Using the Π constructed in the proposition, we have: (BC(Π) ⊗ Σ−2 , 1/2) = 1. In particular, the set S = {v : (BCEv /Fv (Πv ) ⊗ Σ−2 v , Ψv,2 ) = −1} has even cardinality and does not contain the place v0 . Let B0 be the quaternion algebra over F such that (B0v ) 6= (Bv ) ⇐⇒ v ∈ S. In other words, B0 is obtained from B by switching the local invariants of B at the set S. Since v0 ∈ / S, we have B0v0 ∼ = B. Moreover, by Proposition 11.1, for each place v of F , ΘΨv ,Σv ,VBv ,WB0 (Πv ) 6= 0. v

By [16], the nonvanishing of the central L-value above implies that the global theta lift is nonvanishing as well: ΘΨ,Σ,∆,VB ,WB0 (Π) 6= 0. Now the assertion of the theorem has been checked for all finite places of F outside v0 , since at least one of Bv or B0v is split at any v 6= v0 . At the archimedeanplaces, the groups U(VB ⊗ Fv ) are compact and the theta correspondence over R involving compact groups is completely known (c.f. [26] or [18] for example). Using this, one can verify the analog of the assertion of the theorem over R (cf. [31]); we omit the details here. Thus the assertion of the theorem is true for all places of F over v0 . If the result of the theorem is not true at the place v0 , we would have a cuspidal representation ΘΨ,Σ,VB ,WB0 (Π) of U(WB0 ) which violates the Labesse-Langlands multiplicity formula for global endoscopic packets of U(2). This gives the desired contradiction. For example, suppose that S is empty so that B = B0 . Then if the result of the theorem holds at all v 6= v0 but fails at v0 , the cuspidal representation ΘΨ,Σ,VB ,WB0 (Π) of U(WB0 ) would differ from the cuspidal representation Π at an odd number of places v. This is a contradiction. 12. Trilinear forms for U(2) In this section, we return to the skew-hermitian case of [7, Conjecture 17.3]. In particular, we consider the case when W0 = W

with dim W0 = dim W = 2.

Thus, let WB = WB,δ be the rank 2 skew-hermitian case obtained from VB by scaling by δ. Fix an additive character ψ0 of k0 , and a character µ of k × so that µ|k× = ωk/k0 . 0

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

148

W. T. GAN, B. H. GROSS & D. PRASAD

This determines the Weil representation ωψ0 ,µ for U(WB ). Given two conjugatesymplectic representations M and N of W D(k) of dimension 2, with corresponding Vogan packet ΠM and ΠN of U(WB ), we are interested in computing HomU(WB ) (πM ⊗ πN ⊗ ωψ0 ,µ , C) as πM and πN vary over all representations in ΠM and ΠN . Note that the representation ωψ0 ,µ is not an irreducible representation of U(WB ). However, we may decompose ωψ0 ,µ according to central characters M ωψ0 ,µ = ωψ0 ,µ [χ] χ

as χ runs over characters of ZU(WB ) ∼ = k × /k0× . In fact, this decomposition is simply the decomposition of the Weil representation for the dual pair U(V1 )×U(WB ) where V1 is the one dimensional hermitian space of discriminant 1. Thus, each summand ωψ0 ,µ [χ] is an irreducible representation of U(WB ). Moreover, it belongs to an endoscopic packet of U(WB ) constructed in Proposition 8.1. Now, because of central character reasons, it is clear that HomU(WB ) (πM ⊗ πN ⊗ ωψ0 ,µ [χ], C) = 0 unless det M · det N = χ. For this χ, we have HomU(WB ) (πM ⊗ πN ⊗ ωψ0 ,µ , C) = HomU(WB ) (πM ⊗ πN ⊗ ωψ0 ,µ [χ], C). In particular, [7, Conjecture 17.3] amounts to a question about invariant trilinear forms on U(WB ). Given that the group U(WB ) can be described in terms of GL2 (k0 ) and its inner form, we shall see that this question can be related to a question about invariant trilinear forms for GL2 which has been addressed in a series of papers by the third author [27, 28, 32]; we recall his result here: Theorem 12.1. — Let N1 , N2 and N3 be 2-dimensional representations of W D(k0 ), with associated representations πi,B of B × . Assume that det N1 · det N2 · det N3 = 1. Then X dim HomB × (π1,B ⊗ π2,B ⊗ π3,B , C) = 1. B

Moreover, HomB × (π1,B ⊗ π2,B ⊗ π3,B , C) 6= 0 ⇐⇒ (N1 ⊗ N2 ⊗ N3 ) = (B). To apply this theorem to the case of U(WB ), we need to consider the group (B × )+ and calculate dim Hom(B × )+ (π1 ⊗ π2 ⊗ π3 , C).

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

149

More generally, let G be a subgroup of GL2 (k0 ) containing SL2 (k0 ). The group G is uniquely determined by the subgroup × kG ⊂ k0×

consisting of determinant of elements of G. Thus, for any quaternion algebra B, it makes sense to define a corresponding subgroup GB inside B × containing SL1 (B). Restricting irreducible representations of B × to GB , one gets a notion of L-packet of representations of GB . It is known that irreducible representations of GL2 (k0 ) restrict to G with multiplicity 1, but this need not be the case for representations of B × if B is non-split. For a representation πB of GB , let m(πB ) denote the multiplicity with which it appears in the restriction of an irreducible representation of B × . Now we have: Theorem 12.2. — For i = 1, 2 and 3, let Ni be a 2-dimensional representation of Q W D(k0 ) with associated representation π ˜B,i of B × . Assume that i det Ni = 1. Then X × dim HomGB (˜ πB,1 ⊗ π ˜B,2 ⊗ π ˜B,3 , C) = #(k0× /k0×2 kG ). B

In particular, X X

m(πB,1 ) · m(πB,2 ) · m(πB,3 ) · dim HomGB (πB,1 ⊗ πB,2 ⊗ πB,3 , C)

B πB,1 ,πB,2 ,πB,3

is equal to × #(k0× /k0×2 kG ),

where the inner sum is taken over irreducible representations πB,i of GB which are contained in the representations π ˜B,i of B × . Proof. — Clearly, HomGB (˜ πB,1 ⊗ π ˜B,2 ⊗ π ˜B,3 , C) ∼ =

X

HomB × (˜ πB,1 ⊗ π ˜B,2 ⊗ π ˜B,3 , Cχ ),

× χ:k0× /kG →Z/2

where the χ’s range over characters of B × trivial on GB identified to characters of × k0× /kG with values in Z/2, and Cχ denotes the 1-dimensional representation χ ◦ NB × of B . By Theorem 12.1, we have X dim HomB × (˜ πB,1 ⊗ π ˜B,2 ⊗ π ˜B,3 , Cχ ) = 1, B

for all characters χ of order ≤ 2 (by absorbing χ in one of the π ˜B,i ’s without affecting the central character). Adding up the contribution of the various χ’s, we get the conclusion of the theorem. Specializing this theorem to the case GB = (B × )+ and noting that, in this case, m(πB,i ) = 1 for each B, we obtain:

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

150

W. T. GAN, B. H. GROSS & D. PRASAD

Corollary 12.3. — In the context of Theorem 12.2, let G = GL2 (k0 )+ . Then one has, X X dim HomGB (πB,1 ⊗ πB,2 ⊗ πB,3 , C) = 2, B πB,1 ,πB,2 ,πB,3

where the inner sum is taken over irreducible representations πB,i of GB which are contained in the representations π ˜B,i of B × . We can now apply the corollary to the group GU+ (WB ) or equivalently U(WB ). Corollary 12.4. — Let Mi be conjugate-symplectic representations of W D(k) with associated L-packet ΠMi ,B of U(WB ). Assume that det M1 · det M2 · det M3 = 1. Then (i) X X dim HomU(WB ) (π1 ⊗ π2 ⊗ π3 , C) = 2. B πi ∈ΠMi ,B

(ii) If one of the Mi ’s, say M1 , is dihedral with respect to k/k0 , so that #ΠM1 ,B0 = 2 for B0 split, then dim HomU(WB ) (π1 ⊗ π2 ⊗ π3 , C) ≤ 1 for each B. If the above Hom space is nonzero, then dim HomU(WB0 ) (π10 ⊗ π20 ⊗ π30 , C) = 0 for B 0 6= B. Proof. — The first assertion follows immediately from the previous corollary and the definition of L-packets for U(W ) given in §6. To deduce the last assertion, note that if HomU(WB ) (π1 ⊗ π2 ⊗ π3 , C) 6= 0, then we also have HomU(WB ) (π1c ⊗ π2c ⊗ π3c , C) 6= 0, where πic denotes the conjugate of πi by an element c ∈ GU(WB ) r GU+ (WB ). Since dim HomU(WB ) (π1 ⊗ π2 ⊗ π3 , C) + dim HomU(WB ) (π1c ⊗ π2c ⊗ π3c , C) ≤ 2, each of these dimensions must be equal to 1, and all other Hom spaces must be 0. Remark 12.5. — Since k0× /k0×2 is a 2-group whose cardinality can be made arbitrarily large by choosing k0 appropriately, and since the L-packet of representations of SL2 (k) is bounded by 4 [22], it follows from Theorem 12.2 that dim HomSL2 (k) (π1 ⊗ π2 ⊗ π3 , C) can be made arbitrarily large. Now we can return to [7, Conjecture 17.3], so that M and N are two 2-dimensional conjugate-symplectic representations of W D(k) which determine Vogan packets ΠM and ΠN of U(WB ). For a fixed additive character ψ0 of k0 , we have obtained a bijection J(ψ0 ) : ΠM ←→ Irr(AM )

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

151

and similarly for ΠN . We are interested in computing HomU(WB ) (πM ⊗ πN ⊗ ωψ0 ,µ ) for πM ∈ ΠM and πN ∈ ΠN . If M and N are non-dihedral (with respect to k/k0 ), so that ΠM and ΠN both contain at most one representation of each U(WB ) (as B varies), then [7, Conjecture 17.3] is a consequence of Theorem 12.1. Indeed, we have AM × AN = Z/2Z × Z/2Z and the distinguished character χ0 satisfies χ0 (−1, 1) = χ0 (1, −1) = (M ⊗ N (µ−1 ), ψ) for any character ψ of k/k0 . On the other hand, if ΠM is obtained by the restriction of the representation τM  χM of GU(WB ) and ΠN is obtained from τN  χN , then the epsilon factor occurring in Theorem 12.1 is (ρτM ⊗ ρτN ⊗ Ind(µ−1 χM χN ),ψ0 ) = (ρτM |W D(k) ⊗ ρτN |W D(k) ⊗ µ−1 · χM · χN , ψ0 (Tr)) = (M ⊗ N (µ−1 ), ψ0 (Tr)) = (M ⊗ N (µ−1 ), ψ). This verifies [7, Conjecture 17.3] in this case. When at least one of M or N is dihedral with respect to k/k0 , we may appeal to the theta correspondence. Since the case when exactly one of them is dihedral with respect to k/k0 is similar and easier, we shall give the details only when both M and N are dihedral with respect to k/k0 Thus, let M = M1 + M2

and N = N1 + N2 ,

with Mi and Ni conjugate-symplectic (not necessarily distinct), and write their component groups as AM = Z/2Ze1 × Z/2Ze2

and AN = Z/2Zf1 × Z/2Zf2 .

In this case, the packet ΠM can be obtained by theta correspondence from U(1). Set ν = M1 and M1 · M2 = η/η σ , for some character η of k × . If La denote the rank 1 hermitian space with discriminant a, then ΠM = {θψ0 ,ν,WB ,La (η|U(La ) ) : a ∈ k0× /Nk × , (B) = ±1}. Relative to the additive character ψ of k/k0 , we have the labelling πρM = θψ0 ,ν,WB ,La (η|U(La ) )

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

152

W. T. GAN, B. H. GROSS & D. PRASAD

if and only if ρM (e1 ) = (B) · ωk/k0 (a) and ρM (e2 ) = ωk/k0 (a). Similarly, a representation in ΠN has the form πρN , so that ρN (f1 ) · ρN (f2 ) = (B). Now consider the seesaw diagram U(La + L−1 )

U(WB ) × U(WB )

U(La ) × U(L−1 )

∆U(WB )

and note that the rank 2 hermitian space La + L−1 is isomorphic to VB 0 with (B 0 ) = ωk/k0 (a). We start with the representation η|U(La ) of U(La ), so that the representation we obtain on U(WB ) is precisely πρM = θψ0 ,ν,WB ,La (η|U(La ) ). On the other side of the seesaw, we start with the representation µ · ν · πρ∨N of U(WB ). Note that taking contragredient has the following effect on the Vogan parameterization: for any character ρN of AN , the representation πρ∨N has Vogan parameter (N ∨ , ρN · β0 ) where β0 is the character of AN ∨ = AN given by β0 (bi ) = ωk/k0 (−1). Now the seesaw identity gives:

HomU(WB ) (πρM ⊗ ωψ−1 ,ν,WB , µ · ν · πρ∨N ) = HomU(La ) (Θψ0 ,ν 2 ,WB ,La +L−1 (µνπρ∨N ), η|U(La ) ). 0

Since ωψ−1 ,ν,WB = ωψ−1 ,µ−1 ,WB ⊗ µν = ωψ0 ,µ ⊗ µν, 0

0

we see that the LHS of this identity is equal to the desired space HomU(WB ) (πρM ⊗ πρN ⊗ ωψ0 ,µ , C). On the other hand, the RHS is nonzero if and only if conditions (a) and (b) below are satisfied: (a) Θψ0 ,ν 2 ,WB ,La +L−1 (µ · ν · (πρN )∨ ) 6= 0. According to Theorem 11.2, this holds if and only if (N ∨ ⊗ µνν −2 , ψ) = (B) · ωk/k0 (a), or equivalently (N ⊗ M1 (µ−1 ), ψ) = (B) · ωk/k0 (a) = ρM (e1 ).

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

153

If this is satisfied, then by Theorem 11.2, the theta lift is equal to the representation πρN ∨ · µν of U(La + L−1 ), with ρN ∨ (fi ) = ρN (fi ) · ωk/k0 (−1) · (Ni ⊗ M1 (µ−1 ), ψ−2 ). (b) HomU(La ) (πρN ∨ µν, η/η σ ) 6= 0. This is a branching problem for U(2) × U(1) which we have resolved in §10. Using the results there, we see that the desired nonvanishing holds if and only if ρN ∨ (fi ) = ρN (fi ) · ωk/k0 (−1) · (Ni ⊗ M1 (µ−1 ), ψ−2 ) = (Ni ⊗ M2 (µ−1 ), ψ2 ) or equivalently ρN (fi ) = (Ni ⊗ M (µ−1 ), ψ2 ) = (Ni ⊗ M (µ−1 ), ψ). Finally, since ρM (−1) = ρN (−1) = (B), we conclude that ρM (e2 ) = (N ⊗ M2 (µ−1 ), ψ). Thus we conclude that HomU(WB ) (πρM ⊗ πρN ⊗ ωψ0 ,µ , C) 6= 0 if and only if ρM × ρN is the distinguished character χ0 of [7, Conjecture 17.3]. 13. Restriction from U(3) to U(2): endoscopic case In this section, we consider the restriction problem for U(3) × U(2). Using theta correspondence, we establish [7, Conjecture 17.3] for endoscopic packets of U(3). In the following section, we shall consider the stable packets of U(3). We fix a pair V0 ⊂ V of split hermitian spaces of dimensions 2 and 3 respectively with V /V0 of discriminant 1. Let V00 ⊂ V 0 be the other pair of hermitian spaces of dimensions 2 and 3, such that V /V0 ∼ = V 0 /V00 . More concretely, for each quaternion algebra B over k0 , we have a rank 2 hermitian space VB . Then the rank 3 hermitian space VB,b = VB + Lb has discriminant satisfying ωk/k0 (disc(VB,b )) = (B) · ωk/k0 (b). If we take b = 1, then as B varies, the pair VB ⊂ VB,1 gives the pairs V0 ⊂ V and

V00

0

⊂V .

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

154

W. T. GAN, B. H. GROSS & D. PRASAD

Suppose first that N is a 2-dimensional conjugate-symplectic representation of W D(k) with associated Vogan packet ΠN of U(VB ). If N = ⊕i Ni , then we write Y Y AN = ANi = Z/2Zfi . i

i

For the fixed additive character ψ of k/k0 , we translate ψ by −2 · disc(V ) = −2 and use the resulting character ψ−2 to fix the Vogan parameterization J(ψ−2 ) : ΠN ←→ Irr(AN ). Now consider a 3-dimensional conjugate-orthogonal representation M = M 1 + M2 ∼ 3M1 , with dim Mi = i and such that each Mi is conjugate-orthogonal. Unless, M = we may further assume that M1 does not occur in M2 . We shall assume that this is the case (i.e., M 6∼ = 3M1 ), since the other case is similarly handled. Then AM = AM1 × AM2 and we write: AM1 = Z/2Ze and AM2 =

Y

Z/2Zei

i

if M2 = ⊕i M2,i . Moreover, we shall assume that the conjugate-orthogonal character M1 has a conjugate-symplectic square root. This can be achieved by twisting M , and since this twist can be absorbed into N for the purpose of the restriction problem, there is no loss of generality in making this assumption on M1 . Under this assumption on M1 , we have described in §8 a construction of the Vogan packet ΠM as well as a bijection ΠM ←→ Irr(AM ) which is canonical in this case (i.e. independent of the additive character). To recall the construction briefly, we set M1 = µ2 for some conjugate-symplectic character µ and set N 0 = M2 · µ, so that N 0 is conjugate-symplectic and AN 0 = AM2 . Then, for quaternion algebras B and B 0 over k0 , one considers the theta correspondence for U(WB 0 ) × U(VB,1 ) relative to the data (ψ0,−2 , µ, δ), where ψ0 is our fixed additive character of k0 . The packet ΠM is then the theta lift of the packet ΠN 0 of U(WB 0 ). For the labelling of the representations in ΠM by Irr(AM ), we refer the reader to the end of §8. Now we would like to determine HomU(VB ) (πM ⊗ πN , C),

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

155

for πM ∈ ΠM and πN ∈ ΠN . We examine this restriction problem using the seesaw diagram U(VB + L1 ) U(WB 0 ) × U(WB 0 )

U(VB ) × U(L1 )

U(WB 0 ).

η On U(WB 0 ), we start with a representation πN 0 ∈ ΠN 0 indexed by a character η of AN 0 , so that η(−1) = (B 0 ).

On U(VB ), we start with a representation (πρN )∨ associated to a character ρN of AN , so that ρN (−1) = (B). Then we have the seesaw identity: η HomU(VB ) (Θψ0,−2 ,µ (πN 0 ) ⊗ πρN , C) η = HomU(WB0 ) (Θψ0,−2 ,µ2 ,VB ,WB0 (πρ∨N ) ⊗ ωψ0,−2 ,µ,L1 ,WB0 , πN 0 ).

Further, for the representations we have at hand, one can easily check that the two big theta lifts in the see-saw identity are equal to their respective small theta lifts. Now note that η πρM = θψ0,−2 ,µ (πN 0) with ρM |AN 0 = η

and ρM (e) = (B 0 ) · η(−1) = (B) · (B 0 ).

Moreover, (πρN )∨ has Vogan parameter (relative to J(ψ0,−2 )) (N ∨ , ρN ∨ ) = (N ∨ , ρN · β0 ) with β0 (fi ) = ωk/k0 (−1). Then the seesaw identity reads: η HomU(VB ) (πρM ⊗ πρN , C) = HomU(WB0 ) (θψ0,−2 ,µ2 ,VB ,WB0 (πρN ∨ ) ⊗ ωψ0,−2 ,µ,WB0 , πN 0 ).

The RHS is nonzero if and only if (i) and (ii) below hold. (i) θψ0,−2 ,µ2 ,VB ,WB0 (πρN ∨ ) 6= 0. By proposition 11.1, this holds if and only if (N ∨ µ−2 , ψ−2 ) = (B) · (B 0 ) = ρM (e), or equivalently (N ⊗ M1 , ψ) = ρM (e). Moreover, by Theorem 11.2, when this holds, we have θψ0 ,µ2 ,VB ,WB0 ((πρN ∨ ) = πρN ∨ ·ρ0

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

156

W. T. GAN, B. H. GROSS & D. PRASAD

where ρ0 is the character of AN ∨ = AN given by ρ0 (fi ) = (Ni∨ µ−2 , ψ−1 ) = (Ni ⊗ M1 , ψ). η (ii) HomU(WB0 (πρN ∨ ·ρ0 ⊗ ωψ0,−2 ,µ,WB0 , πN 0 ) 6= 0. This question was addressed in the previous section, and we deduce that the desired nonvanishing holds if and only if the character (ρN · ρ0 , η) ∈ Irr(AN ) × Irr(AN 0 )

is the distinguished character χ0 in [7, Conjecture 17.3] for the skew-hermitian case for (WB 0 , µ). More precisely, the desired nonvanishing holds if and only if ρN (fi ) · (Ni ⊗ M1 , ψ) = (Ni∨ ⊗ (N 0 )∨ (µ), ψ−1 ) = (Ni ⊗ M2 , ψ), so that ρN (fi ) = (Ni ⊗ M, ψ), and η(ei ) = ((Ni0 )∨ ⊗ N ∨ (µ), ψ−1 ) = (M2,i ⊗ N, ψ). This shows that HomU(VB ) (πρM ⊗ πρN , C) 6= 0 if and only if the character ρM ×ρN is the distinguished character χ0 of [7, Conjecture 17.3], computed using the additive character ψ of k/k0 .

14. Restriction from U(3) to U(2): stable case We now consider the restriction problem for stable Vogan packets of U(3). We preserve the notation of the previous sections. In particular, we have the pairs of spaces V0 ⊂ V and V00 ⊂ V 0 , with dim V = dim V 0 = 3, dim V0 = dim V00 = 2, with V0 the split hermitian space, and disc(V /V0 ) = disc(V 0 /V00 ) = 1. We will use the additive character ψ−2 to normalize the Vogan parameterization for U(V0 ). Let M be an irreducible 3-dimensional conjugate-orthogonal representation of W D(k), so that its associated Vogan packet has the form 0 ΠM = {πM , πM }, 0 where πM is a representation of U(V ) and πM is the same representation considered 0 on U(V ). If M is an irreducible representation of the Weil group W (k), then the representation πM is supercuspidal. Otherwise,

M = µ  St3 where µ is a conjugate-orthogonal character of W (k) and St3 denotes the irreducible 3dimensional representation of SL2 (C). In this case, the representation πM is a twisted Steinberg representation πM = St ⊗ (µ ◦ det).

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

157

On the other hand, let N be an arbitrary 2-dimensional conjugate-symplectic representation of W D(k) with associated Vogan packet ΠN of U(V0 ). We would like to determine HomU(V0 ) (πM ⊗ πN , C) for πM ∈ ΠM and πN ∈ ΠN . We shall reduce this question to the case when ΠM and ΠN are both supercuspidal packets, by first treating the other cases directly. The supercuspidal case will then be handled by a global method in the next two sections. We first consider the case when M = µ  St3 . Since we can absorb the twist by µ into the parameter N , we may assume without loss of generality that µ = 1. In this case, πM = StU(V ) is a quotient of a (un-normalized) principal series representation: U(V )

0 −−−−→ C −−−−→ IndBV (1) −−−−→ StU(V ) −−−−→ 0, where BV denotes a Borel subgroup in U(V ). We now have the following proposition. Proposition 14.1. — (i) If N is not the parameter of the Steinberg representation of U(V0 ), we have î U(V ) ó HomU(V0 ) (StU(V ) ⊗ πρN , C) = HomU(V0 ) ( IndBV (1) ⊗ πρN , C) = HomU(L) (πρN , C). In particular, HomU(V0 ) (StU(V ) ⊗ πρN , C) 6= 0 if and only if ρN (fi ) = (Ni , ψ) = (Ni ⊗ M, ψ). (ii) If N is the parameter of the Steinberg representation of U(V0 ), so that ΠN = {StU(V0 ) , 1U(V00 ) }, we have HomU(V0 ) (StU(V ) ⊗ StU(V0 ) , C) 6= 0. On the other hand, HomU(V00 ) (StU(V 0 ) , C) = 0. Proof. — (i) Part (i) is proved by a standard application of Mackey theory, which reduces the restriction problem for U(V ) × U(V0 ) to one for U(V0 ) × U(L). Indeed, it is a special case of [7, Theorem 15.1], and so we omit its proof here. (ii) The case of U(V00 ) is obvious by Mackey theory, as in (i). The statement for U(V0 ) is a special case of the following general lemma. Lemma 14.2. — Let G = U(V ), and H = U(V0 ) for V0 a codimension one subspace of V such that a maximal isotropic subspace of V0 continues to be maximal isotropic in V . Then the Steinberg representation StG of G contains the Steinberg representation StH of H as a quotient. Proof. — Let L1 ⊂ L2 ⊂ · · · ⊂ Ld be a maximal isotropic flag in V0 with dim Lr = r for all 1 ≤ r ≤ d. By the hypothesis of the lemma, this is also a maximal isotropic flag in V . Let BH and BG be the stabilizer of this flag in H and G respectively. These are minimal parabolic subgroups in H and G respectively, and it is known that any parabolic in H (resp. G) containing BH (resp. BG ) is obtained as the stabilizer of a

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

158

W. T. GAN, B. H. GROSS & D. PRASAD

partial flag Li1 ⊂ Li2 ⊂ · · · ⊂ Lij . It follows that intersection with H gives a bijection between parabolics in G containing BG and parabolics in H containing BH . Now note that X IndG StG = IndG P (1), BG (1)/ P ⊃BG

where P run over all parabolics containing but not equal to BG , and induction refers to un-normalized induction. It follows that the restriction map from functions on BG \G to BH \H gives a surjection from the Steinberg representation of G to the Steinberg representation of H. Remark 14.3. — The previous lemma and the proof works exactly the same way for orthogonal groups too, except for the pair (V, V0 ) for which the even dimensional quadratic space is split. The reason being that for even dimensional split quadratic space V0 , with a maximal isotropic flag L1 ⊂ L2 ⊂ · · · ⊂ Ld , the parabolics which contain the stabilizer of this flag (which is a Borel subgroup in SO(V0 )) are not parametrized by the stabilizer of a partial flag Li1 ⊂ Li2 ⊂ · · · ⊂ Lij . This description is valid for all other quadratic spaces (except direct sum of hyperbolic planes), cf. [23, Chapter 1.III.2]. The proposition verifies [7, Conjecture 17.3] when M = µ ⊗ St3 and N is arbitrary. We may thus restrict attention to the case when M is an irreducible representation of W (k), so that ΠM is a stable supercuspidal packet consisting of the supercuspidal 0 representation πM = πM on U(V ) = U(V 0 ). We first consider the case when N = P + (P σ )∨

or µ ⊗ St2 ,

where P and (P σ )∨ are not necessarily distinct. In such cases, the associated representations of U(V0 ) are contained in principal series representations of U(V0 ) induced from a Borel subgroup B0 . Thus, we need to compute: U(V0 )

HomU(V0 ) (πM , IndB0

(χ))

for a supercuspidal representation πM of U(V ). By Frobenius reciprocity, we see that this is equal to HomT ((πM )U0 , χ) where U0 is the unipotent radical of the Borel subgroup B0 of U(V0 ). We note that U0 = UV1 with UV1 the center of the unipotent radical UV of a Borel subgroup BV of U(V ). Before proceeding further, let us note the following lemma. Lemma 14.4. — Let π be an irreducible generic supercuspidal representation of U(V ) (dim V = 3) with central character ω. Let BV be a Borel subgroup of U(V ), and UV the unipotent radical of BV with center UV1 = [UV , UV ]. Let ψ : UV → C× be a nondegenerate character of UV . Then there is an isomorphism πU 1 ∼ (ω  ψ), = indBV V

ASTÉRISQUE 346

ZV ·UV

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

159

of BV -modules, where ZV denotes the center of U(V ). Proof. — Let ` : π → C be a Whittaker functional for the character ψ : UV → C× . Since ψ restricted to UV1 = [UV , UV ] is trivial, Frobenius reciprocity gives a homomorphism V φ` : πUV1 → IndB ZV ·UV (ω  ψ),

of BV -modules. Since π is supercuspidal, by the standard argument of Kirillov theory, the image of φ` lands inside the compactly induced representation which is easily seen to be irreducible, hence φ` is a surjective homomorphism onto the compactly supported induced representation. Since BV operates transitively on the set of nontrivial characters of UV , uniqueness of Whittaker models implies that the map φ` must be injective. It follows from the lemma above that (πM )UV1 is isomorphic to the regular representation of T ∼ = k × on S (k × ) where T is the quotient of a maximal torus in BV by the center of U(V ). Thus we have U(V0 )

HomU(V0 ) (πM , IndB0

(χ)) = Homk× ( S (k × ), χ) = C.

In particular, this verifies [7, Conjecture 17.3] when N = P + (P σ )∨ ,

with P 6= (P σ )∨ ,

as the principal series representation on U(V0 ) is irreducible. If P ∼ = (P σ )∨ , then the parameter N is dihedral and the corresponding principal series is the sum of two irreducible summands. In this case, we have not determined which of these summands contributes to the 1-dimensional Hom space above. The issue of which representation supports the Hom will be settled by Theorem 16.1 below. Finally, when N = µ ⊗ St2 , we may assume without loss of generality that µ = 1 (by absorbing µ into M ). Then ΠN = {StU(V0 ) , 1U(V00 ) }, and U(V0 )

0 −−−−→ C −−−−→ IndB0

1 −−−−→ StU(V0 ) −−−−→ 0.

The above computation shows that U(V0 )

HomU(V0 ) (πM , IndB0

1) = C.

On the other hand, by [9], we have HomU(V0 ) (πM , C) = 0 and 0 HomU(V00 ) (πM , C) = 0.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

160

W. T. GAN, B. H. GROSS & D. PRASAD

0 Indeed, if these Hom spaces were not zero, πM and πM would be obtainable as a theta lifting from some U(2), contradicting the fact that M is a stable parameter of U(3). Thus, we conclude that HomU(V0 ) (πM , StU(V0 ) ) 6= 0,

which is what [7, Conjecture 17.3] predicts.

15. A global argument The methods of theta correspondence pursued in the previous sections are inadequate to handle those representations of U(3) whose Langlands parameters M are irreducible, since such representations do not figure in the theta correspondence with a smaller unitary group. For such representations, however a global argument can be provided. The global argument rests on our ability to globalize the local situation such that the following hold: (i) the global cuspidal representation Π has nonzero global period; (ii) the analogous branching laws are known for all local components of Π other than that at the place of interest; (iii) the nonvanishing of the global period implies the non-vanishing of a certain central critical L-value, as suggested by our global conjectures in [7]. We shall be able to achieve (i) and (ii) using a result of the third author and SchulzePillot [33] (and also [32]), and the requirement (iii) is a theorem due to Ginzburg, Jiang, and Rallis [11, Theorem 4.6] in certain cases. The main result of this section is the following theorem. Theorem 15.1. — Let V0 be a 2 dimensional hermitian subspace of a hermitian space V of dimension 3 over k. Suppose that πM (resp. πN ) is an irreducible representation of U(V ) (resp. U(V0 )) with Langlands parameter M (resp. N ). Then (M ⊗ N, ψ) is independent of the additive character ψ of k/k0 and so may be denoted as (M ⊗ N ). Suppose that HomU(V0 ) (πM ⊗ πN , C) 6= 0. Then ( (M ⊗ N )

=

1

−1

if U(V ) × U(V0 ) is quasi − split otherwise

Remark 15.2. — Let Stn denote the unique irreducible representation of SL2 (C) of dimension n, considered as an irreducible representation of Wk0 . From the formulae about epsilon factors, cf. [40], it follows that (Stn ) = ±1 for all integers n, and (Stn ) = −1 if and only if n is even. Therefore by the Clebsch-Gordan theorem about tensor product of representations of SL2 (C), (Stn+1 ⊗ Stn ) = (−1)n , hence (Stn+1 ⊗ Stn ) = 1 if and only if n is even. Therefore theorem 15.1 (stated and proved here only for n = 2) is in accordance with Lemma 14.2 about Steinberg representation of U(n) whose parameter is Stn for general n.

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

161

The method that we follow to prove this theorem is pretty general, but it is based on a global theorem of Ginzburg, Jiang, and Rallis [11, theorem 4.6] which assumes that automorphic forms on unitary groups U(n) have base change to GL(n). This is known at the moment only for generic automorphic representations on quasi-split unitary groups. However, by Rogawski [35], base change is known for any unitary group in 3 variables, which is why we have restricted ourselves to U(3) in the above theorem. Nonetheless, we have formulated some of the preliminary results below in greater generality. We begin with the following globalization result about local fields, which will be applied to globalize hermitian spaces over local fields keeping unitary groups at infinity compact. Lemma 15.3. — Let k be a quadratic extension of a non-archimedean local field k0 . Then there exists a totally real number field F with k0 as its completion, and a quadratic totally imaginary extension E of F with corresponding completion k; further, we can assume that the degree of F over Q is any integer d ≥ (the degree of k over the corresponding Qp ). Proof. — This follows from combining the weak approximation theorem (for the additive group) with the Krasner’s lemma. For the globalization of hermitian forms over a local field, we will need the wellknown classification of a hermitian form over a number field, according to which a hermitian form over a number field is determined by 1. the discriminant, and 2. the signatures at the infinite places. Moreover, given any discriminant, and signatures at infinite places (except for obvious compatibility between discriminant and signatures), there is such a global hermitian form with the given local constraints. We also note the following exact sequence from classfield theory, × 0 → F × /NE × → A× F /NAE → Gal(E/F ) → 0,

from which it follows that one can construct an element in F × which is trivial in Fv× /NEv× at all the finite places except k0 , and which at the infinite places has the desired signs, except that the product of the signs is 1 or −1, depending on whether the element in k0× /Nk × is trivial or nontrivial. Before proceeding further, let’s recall that a hermitian space of dimension n is said to be quasi-split if it contains a maximal isotropic subspace of dimension d where d is the integral part of n/2. It is known that an even dimensional hermitian space over a non-archimedean local field is quasi-split if and only if its discriminant is (−1)d where d = n/2, and any odd dimensional hermitian space over a non-archimedean local field is quasi-split. (A hermitian space is quasi-split if and only if the corresponding unitary group is quasi-split in the sense of algebraic groups.)

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

162

W. T. GAN, B. H. GROSS & D. PRASAD

From the classification theorem of hermitian forms over a number field recalled above, the following lemma follows easily; we omit the proof. Lemma 15.4. — Let V be a hermitian space over k of dimension n = 2d. Let F be a totally real number field with completion k0 at a place v0 of F , and let E be a quadratic totally imaginary extension of F with corresponding completion k. Then there is a hermitian space V over E satisfying (a) V ⊗F k0 = V ; (b) U(V ⊗F Fv ) is quasi-split for all finite places v 6= v0 ; (c) at all the infinite places v of F , V ⊗F Fv has signature (n, 0) if and only if we are in one of the following situations: 1. The integer d is odd, the hermitian space V is quasi-split, and the degree of F over Q is even. 2. The integer d is odd, the hermitian space V is not quasi-split, and the degree of F over Q is odd. 3. The integer d is even, the hermitian space V is quasi-split. Corollary 15.5. — Let V0 ⊂ V be hermitian spaces over k, with dimk (V /V0 ) = 1. Let F be a totally real number field with completion k0 at a place v0 of F , and let E be a totally imaginary quadratic extension of F with corresponding completion k. Then there are hermitian spaces over E V0 ⊂ V satisfying (a) V0 ⊗E k = V0 and V ⊗E k = V , so that dimE (V/V0 ) = 1 (b) the corresponding unitary groups U(V0 ) and U(V) are quasi-split at all the finite places of F different from v0 ; (c) for all infinite places v of F , U(V ⊗ Fv ) is the compact group U(n + 1, 0), if and only if the even dimensional hermitian space in the pair (V, V0 ) satisfies one of the three options of the previous lemma. Proof. — The necessity of the condition is obvious. For the other direction, observe that since an odd dimensional hermitian space is automatically quasi-split at any finite place, we first construct the even dimensional hermitian space in the pair (V, V0 ) , and construct the odd dimensional one by adding or subtracting a one dimensional hermitian space from the even dimensional one, keeping track only of the place corresponding to k0 , and the places at infinity. Proof of Theorem 15.1. — By the results of the previous two sections, we already know the desired result if M is reducible or is the parameter of a twisted Steinberg representation. So we assume that M is an irreducible representation of W (k), so that πM is a supercuspidal representation of U(V ). Similarly the theorem is already

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

163

known if πN is a principal series representation, or a twisted Steinberg representation of U(V0 ). So we will assume in the rest of the proof that both πM and πN are supercuspidal representations. We globalize the local spaces V0 ⊂ V to V0 ⊂ V as in the above corollary, so that U(V) is compact at infinity. It is then easy to see that we can globalize the representation πM of U(V ) to a cuspidal automorphic representation Π1 of U(V)(A) in such a way that it is unramified at all the finite places of F except k0 . It is important to note that all local components of Π1 belong to generic L-packets. Indeed, by the results of Rogawski [35], one knows that the base change BC(Π1 ) of Π1 to GL3 (E) is cuspidal, since the base change of πM to GL3 (k) is supercuspidal. Thus, all the local components of BC(Π1 ) are generic, so that the L-parameters of all local components of Π1 are generic. By [32, Lemma 1], we can globalize πN to an automorphic representation Π0 such that the global period integral Z f0 f1 6= 0, U(V0 )\U(V0 )(A)

for some f0 in Π0 , and f1 in Π1 . By the theorems due to Ginzburg, Jiang, and Rallis [11, Theorem 4.6], the non-vanishing of the global period integral implies the nonvanishing of a central critical L-value: 1 L( , ΠE ⊗ ΠE 1 ) 6= 0, 2 0 E where ΠE 0 and Π1 denote base change of Π0 and Π1 to E. This implies that the global root number, 1 ( , ΠE ⊗ ΠE 1 ) = 1. 2 0 Let Π0 = ⊗w Π0,w , and Π1 = ⊗w Π1,w , with Π0,v = πN , and Π1,v = πM . From the nonvanishing of the period integral, it follows that HomU(V0,w ) (Π0,w ⊗ Π1,w , C) 6= 0 for all places w of F . Since, by construction, the representations Π1,w are unramified and generic for all finite places w 6= v, we know the validity of Theorem 15.1 for such representations. Thus we have: 1 w ( , ΠE ⊗ ΠE 1,w ) = 1 2 0,w for all finite places w 6= v. Since the global epsilon factor is a product of local epsilon factors, we have 1 1 ⊗ ΠE ( , M ⊗ N ) · ∞ ( , ΠE 1,∞ ) = 1. 2 2 0,∞ Thus, to complete the proof of Theorem 15.1, we need to address the branching problem at the infinite places. In particular, we shall show:

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

164

W. T. GAN, B. H. GROSS & D. PRASAD

Proposition 15.6. — Let V0 be a codimension 1 hermitian subspace of a positive definite hermitian space V of dimension n + 1 over C. Suppose that π1 (resp. π0 ) is a finite dimensional irreducible representation of U(V ) (resp. U(V0 )). Let the Langlands parameter of π1 (resp. π0 ) be σ1 (resp. σ0 ). Suppose that HomU(V0 ) (π1 ⊗ π0 , C) 6= 0. Then ( n(n+1) 1 if n ≡ 0, 3 mod 4 (σ1 ⊗ σ0 ) = (−1) 2 = −1 if n ≡ 1, 2 mod 4. This proposition completes the proof of Theorem 15.1, since one knows by Lemma 15.4 that there are an even number of places at infinity if U(V0 ) is quasi-split, and an odd number of places at infinity when U(V0 ) is not quasi-split since dim V0 = 2 (or any odd multiple of 2). The rest of the section is devoted to the proof of the proposition. In fact, it is a simple consequence of the well-known branching law, recalled below in Lemma 15.8, from the compact group U(n+1) to U(n), combined with the value of the epsilon factor given by the following Lemma 15.7, which has been demonstrated in Proposition 2.1. Lemma 15.7. — Let ψ be the additive character on C given by ψ(z) = e−2πiy where z = x + iy. For n a half-integer but not an integer, i.e., n ∈ 21 Z \ Z, let χn denote the character χn (z) = (¯ z /z)n = e−2niθ for z = reiθ ∈ C× . Then for n ∈ 21 Z \ Z, ( 1 if n > 0 (χn , ψ) = −1 if n < 0. Lemma 15.8. — Let π0 (resp. π1 ) be a finite dimensional irreducible representation of the compact group U(n) (resp. U(n + 1)) with L-parameter restricted to C× given by an n-tuple of half-integers σ0 = {−λn < −λn−1 < · · · < −λ1 } (resp. σ1 = {µ1 < µ2 < · · · < µn+1 } an (n + 1)-tuple of half-integers), where all the λ0i s are half-integers but not integers if n is even, and are integers if n is odd, and the µ0i s are all integers if n is even, and half-integers but not integers if n is odd, i.e., σ0

=

χ−λn + · · · + χ−λ1 ,

σ1

=

χµ1 + · · · + χµn+1 .

and

Then HomU(n) (π1 ⊗ π0 , C) 6= 0 if and only if µ1 < λ1 < µ2 < λ2 < · · · < λn < µn+1 . Corollary 15.9. — With notation as in Lemma 15.8, and assuming that π0∨ appears in the restriction of π1 , one has (χµk ⊗ σ0 ) = (−1)n−k+1 , f or all k,

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

165

and therefore, (σ1 ⊗ σ0 )

=

n+1 Y

(−1)n−k+1

k=1

=

(−1)

n(n+1) 2

.

Remark 15.10. — It should be mentioned that the global method followed in the proof of Theorem 15.1 proves that if there is an invariant linear form, then the epsilon factor has the expected value predicted in [7]. The natural variant for unitary groups of the theorem of Waldspurger in [46] will prove that such an invariant form always exists on a relevant pair of unitary groups. This will then strengthen Theorem 15.1 to an if and only if statement. 16. A finer global argument In the previous section, we used a global argument to prove Theorem 15.1, which says that a nonzero invariant form for a Vogan packet ΠM × ΠN is supported on the quasi-split group U(V ) × U(V0 ) if and only if (M ⊗ N ) = 1. One can refine this argument to compute other epsilon factors which arise in [7, Conjecture 17.3] when N is reducible. We give a sketch of this refined argument in this section. Suppose that N = N1 + N2 , where Ni is conjugate symplectic of dimension 1, with associated component group AN . In §8, we have defined a bijection J(ψ) : ΠN ↔ Irr(AN ) which depends on the fixed additive character ψ : k/k0 → C× . When N1 6= N2 , AN = Z/2Z × Z/2Z and thus a representation π0 ∈ ΠN is labelled by a pair of signs (η1 (π0 , ψ), η2 (π0 , ψ)). When N1 = N2 , (which corresponds to a reducible unitary principal series), AN = Z/2, and we have the label η1 (π0 , ψ) = η2 (π0 , ψ) ∈ {±1}. Now the main result of this section is: Theorem 16.1. — Let V0 ⊂ V be hermitian spaces over a non-archimedean local field k with dim V = 3 and dim V0 = 2. Let π1 be an irreducible representation of U(V ) belonging to a generic L-packet with Langlands parameter M . Let π0 be a dihedral representation of U(V0 ) with Langlands parameter N = N1 ⊕ N2 with Ni conjugate symplectic. If HomU(V0 ) (π1 ⊗ π0 , C) 6= 0, then we have (M ⊗ N1 , ψ)

= η1 (π0 , ψ)

(M ⊗ N2 , ψ)

= η2 (π0 , ψ).

Proof. — We already know the desired result in all cases except when π1 is a stable supercuspidal representation. To take care of this remaining case, we globalize everything in sight. More precisely, (i) we first globalize the local fields k0 ⊂ k to global fields F ⊂ E with F totally real and E totally complex.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

166

W. T. GAN, B. H. GROSS & D. PRASAD

(ii) next, we globalize V0 ⊂ V to hermitian spaces V0 ⊂ V over F , keeping V0 quasi-split at all the finite places away from k, and V positive definite at all real spaces; this is possible by Corollary 15.5. (iii) we then globalize the representation π0 of U(V0 ) to a dihedral automorphic representation Π0 of U(V0 ) which is unramified outside the finite place of F corresponding to k0 . This is possible as it amounts to globalizing a character of k × to a Grössencharacter on A× E unramified at all the finite places different from k, cf. [32, Lemma 3]. Further, we may ensure that the Grössencharacter on A× E is not Galois invariant, so that the automorphic representation Π0 on U(V0 ) is cuspidal. If N1 6= N2 , there is no issue about this, but if N1 = N2 , one needs to observe that the flexibility allowed by [32, Lemma 3] makes the representation at infinity to be discrete series. (iv) we next globalize π1 of U(V ) to an automorphic representation Π1 of U(V) so that it is a principal series representation at all finite places of E away from k and with nonzero period integral: Z f0 f1 6= 0, U(V0 )\U(V0 )(A)

for some f0 in Π0 , and f1 in Π1 . This is possible by an application of relative trace formula exactly as in the proof of [33, Theorem 4.1], though the result in this reference is proved only for a character on the subgroup; we grant ourselves such a generalization here. Further, we note that since π1 is stable, all local components of Π1 belong to generic L-packets. Now by the theorem of Ginzburg, Jiang, and Rallis [11, Theorem 4.6], the nonvanishing of the period integral in (iii) above implies the nonvanishing of the central critical L-value: E L( 12 , ΠE 0 ⊗ Π1 ) 6= 0, E where ΠE 0 and Π1 denote base change of Π0 and Π1 to E, which are automorphic representations of GL2 (AE ) and GL3 (AE ) respectively. We note that the work of Ginzburg, Jiang, and Rallis is at the moment subject to the hypothesis that ΠE 0 and ΠE are cuspidal, which is not the case here. However, it seems likely that their theorem 1 can be strengthened to give what we need; again we grant ourselves this extension here. In the case at hand, ΠE 0 is an Eisenstein series corresponding to a sum of two Grössencharacters Ξ1 + Ξ2 , and therefore the L-function being considered factorizes as E E L(s, [Ξ1 + Ξ2 ] ⊗ ΠE 1 ) = L(s, Ξ1 ⊗ Π1 ) · L(s, Ξ2 ⊗ Π1 ). 1 E The nonvanishing of L( 12 , ΠE 0 ⊗ Π1 ) then implies the nonvanishing of both L( 2 , Ξ1 ⊗ 1 E E Π1 ) and L( 2 , Ξ2 ⊗ Π1 ). The two L-functions being considered are both selfdual, and hence the corresponding global root numbers are 1:

( 12 , Ξ1 ⊗ ΠE 1 ) = 1,

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

167

and ( 12 , Ξ2 ⊗ ΠE 1 ) = 1. By the multiplicity formula of Labesse-Langlands [22] or Rogawski [35], since the representation Π0 is automorphic, we have: Y η1 (Π0 ) := η1 (Π0,v , Ψv ) = 1, v

and η2 (Π0 ) :=

Y

η2 (Π0,v , Ψv ) = 1.

v

Here, Ψ is a character of AE /EAF → C× , with local components Ψv , and the values ηi (Π0,v , Ψv ) = ±1 are the labels for the members inside a Vogan packet defined in §6 and recalled at the beginning of this section. In view of the above, we get that: Y Y (A) 1 = ( 21 , Ξ1 ⊗ ΠE (Ξ1,v ⊗ ΠE η1 (Π0,v , Ψv ), 1)= 1,v ) = v

v

and similarly, (B)

1 = ( 12 , Ξ2 ⊗ ΠE 1)=

Y v

(Ξ2,v ⊗ ΠE 1,v ) =

Y

η2 (Π0,v , Ψv ).

v

We note that at the places v of F split in E, the unitary groups U(Vv ) and U(V0,v ) become GL3 (Fv ) and GL2 (Fv ) respectively. At such places, the signs η1 and η2 are trivial (by definition); further, it is easy to see that if the place v of F splits into two E places {v 0 , v 00 } of E, then (Ξ1,v0 ⊗ ΠE 1,v 0 ) · (Ξ1,v 00 ⊗ Π1,v 00 ) = 1. This means that in the above product formulae (A), (B), we can ignore places of F split in E. Since we have globalized Π1 keeping it unramified at the finite places away from k, we know that the theorem being proved is known by the results of the previous sections. By the product formulae (A) and (B), our theorem is proved at this remaining place! We end with a summary of the status of [7, Conjecture 17.3] for U(3) × U(2), as treated in the last 4 sections of this paper. If the L-parameter is M ⊗ N , then we have: 1. If M is endoscopic, [7, Conj. 17.3] is verified by Section 13. 2. If M is Steinberg, [7, Conj. 17.3] is done by Prop. 14.1. 3. If M is stable supercuspidal, and N corresponds to an irreducible principal series of U(2), or a twisted steinberg representation, [7, Conj. 17.3] is verified by Lemma 14.4, and the ensuing discussion. 4. If M stable supercuspidal and N corresponds to a dihedral principal series representation, then [7, Conj. 17.3] is verified by Lemma 14.4 and the ensuing discussion, together with Theorem 16.1. 5. If M stable supercuspidal and N (stable or dihedral) supercuspidal, then [7, Conj. 17.3] is partially verified by Theorems 15.1 and 16.1. More precisely, we show that the only representation in ΠM × ΠN which could possibly support an invariant form is the one corresponding to the distinguished character in [7,

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

168

W. T. GAN, B. H. GROSS & D. PRASAD

Conj. 17.3]. However, we have not shown that this distinguished representation is actually distinguished! References [1] A. Aizenbud, D. Gourevitch, S. Rallis & G. Schiffmann – “Multiplicity one theorems”, Ann. of Math. 172 (2010), p. 1407–1434. [2] I. N. Bernstein & A. V. Zelevinsky – “Induced representations of reductive p-adic groups. I”, Ann. Sci. École Norm. Sup. 10 (1977), p. 441–472. [3] S. DeBacker & M. Reeder – “Depth-zero supercuspidal L-packets and their stability”, Ann. of Math. 169 (2009), p. 795–901. , “On some generic very cuspidal representations”, Compos. Math. 146 (2010), [4] p. 1029–1055. [5] P. Deligne & G. Lusztig – “Representations of reductive groups over finite fields”, Ann. of Math. 103 (1976), p. 103–161. [6] A. Fröhlich & J. Queyrut – “On the functional equation of the Artin L-function for characters of real representations”, Invent. Math. 20 (1973), p. 125–138. [7] W. T. Gan, B. H. Gross & D. Prasad – “Symplectic local root numbers, central critical L-values and restriction problems in the representation theory of classical groups”, this volume, 2012. [8] S. S. Gelbart & J. D. Rogawski – “L-functions and Fourier-Jacobi coefficients for the unitary group U(3)”, Invent. Math. 105 (1991), p. 445–472. [9] S. S. Gelbart, J. D. Rogawski & D. Soudry – “Endoscopy, theta-liftings, and period integrals for the unitary group in three variables”, Ann. of Math. 145 (1997), p. 419–476. [10] D. Ginzburg, D. Jiang & S. Rallis – “On the nonvanishing of the central value of the Rankin-Selberg L-functions. II”, in Automorphic representations, L-functions and applications: progress and prospects, Ohio State Univ. Math. Res. Inst. Publ., vol. 11, de Gruyter, 2005, p. 157–191. [11] , “Models for certain residual representations of unitary groups”, in Automorphic forms and L-functions I. Global aspects, Contemp. Math., vol. 488, Amer. Math. Soc., 2009, p. 125–146. [12] Z. Gong & L. Grenié – “An inequality for local theta correspondence”, manuscrit, 2008. [13] B. H. Gross & D. Prasad – “On the decomposition of a representation of SOn when restricted to SOn−1 ”, Canad. J. Math. 44 (1992), p. 974–1002. [14] , “On irreducible representations of SO2n+1 ×SO2m ”, Canad. J. Math. 46 (1994), p. 930–950. [15] B. H. Gross & M. Reeder – “From Laplace to Langlands via representations of orthogonal groups”, Bull. Amer. Math. Soc. (N.S.) 43 (2006), p. 163–205. [16] M. Harris – “L-functions of 2×2 unitary groups and factorization of periods of Hilbert modular forms”, J. Amer. Math. Soc. 6 (1993), p. 637–719. [17] M. Harris, S. S. Kudla & W. J. Sweet – “Theta dichotomy for unitary groups”, J. Amer. Math. Soc. 9 (1996), p. 941–1004. [18] M. Harris, J. S. Li & B. Y. Sun – “Theta correspondences for close unitary groups”, to appear in a volume in honor of S. Kudla. [19] K. Konno & T. Konno – “CAP automorphic representations of UE/F (4) I. Local A-packets”, Kyushu Univ. Preprint Series (2003–2004).

ASTÉRISQUE 346

RESTRICTION PROBLEMS FOR CLASSICAL GROUPS

169

[20] T. Konno – “Local Gross-Prasad conjecture for U(2)”, slides of a talk given in Workshop on Representation theory and Automorphic Forms, National Univ. of Singapore, 2008. [21] S. S. Kudla – “Splitting metaplectic covers of dual reductive pairs”, Israel J. Math. 87 (1994), p. 361–401. [22] J.-P. Labesse & R. P. Langlands – “L-indistinguishability for SL(2)”, Canad. J. Math. 31 (1979), p. 726–785. [23] C. Mœglin, M.-F. Vignéras & J.-L. Waldspurger – Correspondances de Howe sur un corps p-adique, Lecture Notes in Math., vol. 1291, Springer, 1987. [24] C. Mœglin & J.-L. Waldspurger – “La conjecture locale de Gross-Prasad pour les groupes spéciaux orthogonaux: le cas général”, to appear in Astérisque. [25] C. Moen – “The dual pair (U(3), U(1)) over a p-adic field”, Pacific J. Math. 127 (1987), p. 141–154. [26] A. Paul – “Howe correspondence for real unitary groups”, J. Funct. Anal. 159 (1998), p. 384–431. [27] D. Prasad – “Trilinear forms for representations of GL(2) and local -factors”, Compositio Math. 75 (1990), p. 1–46. , “Invariant forms for representations of GL2 over a local field”, Amer. J. Math. [28] 114 (1992), p. 1317–1363. [29] , “On an extension of a theorem of Tunnell”, Compositio Math. 94 (1994), p. 19– 28. [30] , “Some applications of seesaw duality to branching laws”, Math. Ann. 304 (1996), p. 1–20. [31] , “Theta correspondence for unitary groups”, Pacific J. Math. 194 (2000), p. 427– 438. , “Relating invariant linear form and local epsilon factors via global methods”, [32] Duke Math. J. 138 (2007), p. 233–261. [33] D. Prasad & R. Schulze-Pillot – “Generalised form of a conjecture of Jacquet and a local consequence”, J. reine angew. Math. 616 (2008), p. 219–236. [34] D. Prasad & R. Takloo-Bighash – “Bessel models for GSp(4)”, J. reine angew. Math. 655 (2011), p. 189–243. [35] J. D. Rogawski – Automorphic representations of unitary groups in three variables, Annals of Math. Studies, vol. 123, Princeton Univ. Press, 1990. [36] , “The multiplicity formula for A-packets”, in The zeta functions of Picard modular surfaces, Univ. Montréal, 1992, p. 395–419. [37] H. Saito – “On Tunnell’s formula for characters of GL(2)”, Compositio Math. 85 (1993), p. 99–108. [38] , “Two remarks on a theorem of Dipendra Prasad”, Pacific J. Math. 234 (2008), p. 185–199. [39] F. Sauvageot – “Principe de densité pour les groupes réductifs”, Compositio Math. 108 (1997), p. 151–184. [40] J. Tate – “Number theoretic background”, in Automorphic forms, representations and L-functions (Proc. Sympos. Pure Math., Oregon State Univ., Corvallis, Ore., 1977), Part 2, Proc. Sympos. Pure Math., XXXIII, Amer. Math. Soc., 1979, p. 3–26. [41] E. Thoma – “Die Einschränkung der Charaktere von GL(n, q) auf GL(n−1, q)”, Math. Z. 119 (1971), p. 321–338. [42] J. B. Tunnell – “Local -factors and characters of GL(2)”, Amer. J. Math. 105 (1983), p. 1277–1307. [43] J.-L. Waldspurger – “Correspondance de Shimura”, J. Math. Pures Appl. 59 (1980), p. 1–132.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

170

[44] [45] [46] [47] [48] [49]

W. T. GAN, B. H. GROSS & D. PRASAD

, “Sur les valeurs de certaines fonctions L automorphes en leur centre de symétrie”, Compositio Math. 54 (1985), p. 173–242. , “Correspondances de Shimura et quaternions”, Forum Math. 3 (1991), p. 219– 307. , “Une formule intégrale reliée à la conjecture locale de Gross-Prasad”, Compositio Mathematica 146 (2010), p. 1180–1290. , “Une formule intégrale reliée à la conjecture locale de Gross-Prasad, 2e partie: extension aux représentations tempérées”, this volume, 2012. , “Calcul d’une valeur d’un facteur epsilon par une formule intégrale”, to appear in Astérisque. , “La conjecture locale de Gross-Prasad pour les représentations tempérées des groupes spéciaux orthogonaux”, to appear in Astérisque.

W. T. Gan, Department of Mathematics, University of California at San Diego, 9500 Gilman Drive, La Jolla, 92093 • E-mail : [email protected] B. H. Gross, Department of Mathematics, Harvard University, Cambridge, MA 02138 E-mail : [email protected] D. Prasad, School of Mathematics, Tata Institute of Fundamental Research, Colaba, Mumbai400005, India • E-mail : [email protected]

ASTÉRISQUE 346

Astérisque 346, 2012, p. 171–311

UNE FORMULE INTÉGRALE RELIÉE À LA CONJECTURE LOCALE DE GROSS-PRASAD, 2e PARTIE : EXTENSION AUX REPRÉSENTATIONS TEMPÉRÉES par Jean-Loup Waldspurger

Résumé. — Soit V un espace vectoriel sur un corps p-adique F , soit q une forme quadratique non-dégénérée sur V et soit D une droite non isotrope de V . Notons W l’hyperplan orthogonal à D, G et H les groupes spéciaux orthogonaux de V et W . Soient π, resp. ρ, une représentation admissible et irréductible de G(F ), resp. H(F ). La représentation ρ apparaît comme quotient de la restriction de π à H(F ) avec une certaine multiplicité m(π, ρ). On sait que m(π, ρ) ≤ 1. Dans un article précédent, sous l’hypothèse que π était supercuspidale, nous avons prouvé une formule qui calculait m(π, ρ) comme une intégrale de fonctions déduites des caractères de π et ρ. Ici, nous prouvons la même formule sous l’hypothèse que π et ρ sont toutes deux tempérées. Nous imitons la preuve de la formule des traces locale d’Arthur. En supposant vérifiées les propriétés attendues des L-paquets, nous prouvons grâce à notre formule une version faible de la conjecture locale de Gross-Prasad pour les L-paquets tempérés. Abstract (Symplectic local root numbers, central critical L-values, and restriction problems in the representation theory of classical groups) Let V be a vector space over a p-adic field F , let q be a non-degenerate quadratic form over V and let D be a non-isotropic line in V . Denote W the hyperplane orthogonal to D, G and H the special orthogonal groups of V and W . Let π, resp. ρ, be an irreducible admissible representation of G(F ), resp. H(F ). The representation ρ appears as a quotient of the restriction of π to H(F ) with a certain multiplicity m(π, ρ). We know that m(π, ρ) ≤ 1. In a preceding paper, assuming that π was supercuspidal, we have proved a formula that computes m(π, ρ) as an integral of functions deduced from the characters of π and ρ. Here, we prove the same formula for π and ρ tempered. We follow the proof due to Arthur of the local trace formula. Using our formula and assuming some usual properties of L-packets, we prove a weak form of the local Gross-Prasad conjecture for tempered L-packets.

Introduction Cet article fait suite à [17]. Rappelons les définitions des principaux objets. Soit F un corps local non archimédien de caractéristique nulle. Soit (V, qV ) un espace Classification mathématique par sujets (2010). — 11S37, 22E50. Mots clefs. — Représentations tempérées; groupes spéciaux orthogonaux; conjecture locale de GrossPrasad.

© Astérisque 346, SMF 2012

172

J.-L. WALDSPURGER

quadratique, c’est-à-dire que V est un espace vectoriel de dimension finie sur F et qV est une forme quadratique non dégénérée sur V . Soit (W, qW ) un autre espace quadratique. On suppose que l’on a une décomposition orthogonale V = W ⊕ D0 ⊕ Z, où D0 est une droite et Z est muni d’une base {vi ; i = ±1, . . . , ±r} telle que qV (vi , vj ) = δi,−j pour tous i, j. On note G et H les groupes spéciaux orthogonaux de V et W , que l’on considère comme des groupes algébriques définis sur F . Le groupe H se plonge naturellement dans G. Introduisons le sous-groupe parabolique de G formé des éléments qui conservent le drapeau F vr ⊂ F vr ⊕ F vr−1 ⊂ · · · ⊂ F vr ⊕ · · · ⊕ F v1 . Notons U son radical unipotent. Fixons un élément non nul v0 ∈ D0 et un caractère continu non trivial ψ de F . On définit un caractère ξ de U (F ) par l’égalité ! X qV (uvi , v−i−1 ) . ξ(u) = ψ i=0,...,r−1

Soient π, resp. ρ, une représentation admissible irréductible de G(F ), resp. H(F ), dans un espace complexe Eπ , resp. Eρ . On note HomH,ξ (π, ρ) l’espace des applications linéaires ϕ : Eπ → Eρ telles que ϕ(π(hu)e) = ξ(u)ρ(h)ϕ(e) pour tous u ∈ U (F ), h ∈ H(F ), e ∈ Eπ . On note m(ρ, π) la dimension de cet espace. D’après [1, théorème 10 ] et [8, corollaire 15.2], ce nombre vaut 0 ou 1. Il est indépendant des divers choix effectués. Supposons G et H quasi-déployés sur F et affectons les notations d’un indice i : Vi , Gi etc. Supposons pour cette introduction dim(Wi ) ≥ 3. A équivalence près, il y a un unique espace quadratique que nous notons (Va , qVa ) tel que dim(Va ) = dim(Vi ), que les discriminants de qVi et qVa soient égaux mais leurs indices de Witt soient distincts. On introduit de même un espace quadratique (Wa , qWa ). Le couple (Va , Wa ) vérifie les mêmes propriétés que (Vi , Wi ) : à isomorphisme près, on a encore l’égalité Va = Wa ⊕D0 ⊕Z, avec les mêmes D0 et Z que ci-dessus. Les groupes spéciaux orthogonaux Ga , resp. Ha , de Va , resp. Wa , sont des formes intérieures de Gi , resp. Hi . On note Temp(Gi ), Temp(Ga ) etc. les ensembles de représentations tempérées et irréductibles de Gi (F ), Ga (F ) etc. On admet que ces ensembles sont unions disjointes de L-paquets vérifiant certaines propriétés encore conjecturales. Précisément on admet les propriétés (1), (2) et (3) de [17] 13.2. Soient Πi , resp. Σi , un L-paquet dans Temp(Gi ), resp. Temp(Hi ). Il peut correspondre à Πi un L-paquet dans Temp(Ga ), que l’on note Πa . Ou bien il n’y a pas de tel L-paquet et on pose Πa = ∅. On définit de même Σa . La multiplicité m(ρ, π) est bien définie pour tout (ρ, π) ∈ (Σi × Πi ) ∪ (Σa × Πa ). Théorème. — Sous ces hypothèses, il existe un unique couple (ρ, π) ∈ (Σi × Πi ) ∪ (Σa × Πa ) tel que m(ρ, π) = 1. C’est une partie de la conjecture 6.9 de [10]. Ce théorème résulte aisément d’une formule qui calcule m(ρ, π) comme une somme d’intégrales de fonctions qui se déduisent

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

173

des caractères de ρ et π. Plus précisément, revenons aux notations sans indices du début de cette introduction. Soient π et ρ des représentations admissibles irréductibles de G(F ) et H(F ). On introduit une expression mg´eom (ρ, π) pour la définition de laquelle on renvoie à l’introduction de [17]. Théorème. — Supposons π et ρ tempérées et irréductibles. Alors on a l’égalité m(ρ, π) = mg´eom (ρ, π). Cf. 7.1. Dans [17], on avait démontré cette égalité sous les hypothèses que π était cuspidale et ρ admissible. Ici, on élargit l’hypothèse sur π qui n’est plus que tempérée. Par contre, on impose une hypothèse plus forte à ρ qui est elle-aussi tempérée. Comme dans [17], le second théorème implique le premier. Evidemment, dans [17], l’hypothèse de cuspidalité présente dans le second théorème se retrouvait dans le premier. C’est cette hypothèse que nous faisons disparaître dans le présent article. Décrivons l’idée principale de la preuve du second théorème. Rappelons que, pour une fonction f ∈ Cc∞ (G(F )), on dit que f est très cuspidale si et seulement si, pour tout sous-groupe parabolique propre P 0 = M 0 U 0 de G (avec une notation familière) et pour tout m0 ∈ M 0 (F ), on a l’égalité Z f (m0 u0 )du0 = 0. U 0 (F )

Soient ρ ∈ Temp(H) et f une fonction très cuspidale sur G(F ). On note θρ le caractère de ρ. Pour tout N ∈ N, on introduit une fonction κN sur G(F ) qui est la fonction caractéristique de l’image réciproque d’un sous-ensemble compact de H(F )U (F )\G(F ) qui est de plus en plus grand quand N tend vers l’infini. Posons Z Z Z JN (θρ , f ) = θρ (h)f (g −1 hug)ξ(u)κN (g)du dh dg. H(F )U (F )\G(F )

H(F )

U (F )

On montre que, quand N tend vers l’infini, cette expression a une limite. En fait, et c’est cela qui est fructueux, il y a deux façons de calculer la limite. L’une, que l’on peut qualifier de géométrique, a été développée en [17], et conduit à une égalité limN →∞ JN (θρ , f ) = Jg´eom (θρ , f ), où le membre de droite est une somme d’intégrales sur certains sous-tores de H(F ). Dans le présent article, on calcule la limite d’une autre façon, que l’on peut qualifier de spectrale. On obtient une égalité (cf. théorème 6.1) : limN →∞ JN (θρ , f ) = Jspec (θρ , f ), où Jspec (θρ , f ) =

X

|W L ||W G |−1 (−1)aL

L∈ L (Mmin ) −1 [i A ∨O : i A ∨ t(π)−1 L,F ]

X O∈{Πell (L)};m( O,ρ)=1

Z i A∗ L,F

JLG (πλ , f )dλ.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

174

J.-L. WALDSPURGER

Tous les termes de cette formule seront définis dans l’article. Disons simplement ici que, dans le cas où L = G, les objets O sont simplement les représentations irréductibles tempérées et elliptiques de G(F ) et, si l’on pose plus simplement π = O, G la condition m( O, ρ) = 1 n’est autre que m(ρ, π) = 1 tandis que JG (π, f ) = θπ (f ). G Pour L quelconque, π est un élément fixé dans O et JL (πλ , f ) est la valeur sur f du caractère pondéré associé à πλ . On a donc l’égalité Jg´eom (θρ , f ) = Jspec (θρ , f ) qui, bien sûr, rappelle fortement la formule des traces locale d’Arthur. De fait, la preuve reprend très largement celle de [6]. Dans les deux membres de la formule apparaissent des distributions qui ne sont pas invariantes : intégrales orbitales pondérées et caractères pondérés. Le procédé mis au point par Arthur, appliqué en particulier dans [7] à la formule des traces locale, permet de transformer la formule ci-dessus en une autre où n’apparaissent que des distributions invariantes. Le terme de droite de cette formule continue de distinguer les représentations π de G(F ) telles que m(ρ, π) = 1. Pour une représentation π de la série discrète, ou plus généralement pour π elliptique au sens de [7], le second théorème résulte facilement de cette formule « invariante » appliquée à un pseudo-coefficient de π. En étudiant le comportement des deux membres de l’égalité à démontrer relativement à l’induction en la variable π, le cas général se déduit de celui où π est elliptique par récurrence sur les dimensions de V et W . Expliquons encore deux points. Dans la formule non invariante, la fonction f est supposée très cuspidale, ce qui est assez restrictif. Cela parce que nous ne savons pas calculer la limite de JN (θρ , f ) pour une fonction qui ne vérifie pas cette hypothèse (le résultat rend d’ailleurs douteuse la possibilité d’étendre nos calculs à des fonctions ne vérifiant pas cette hypothèse). Mais, une fois la formule rendue invariante, on peut supposer f seulement cuspidale (c’est-à-dire les intégrales orbitales J G (x, f ) sont nulles pour tout élément x ∈ G(F ) qui est semi-simple, fortement régulier et non elliptique). Cela résulte du lemme suivant (lemme 2.7). Lemme. — Soit f ∈ Cc∞ (G(F )) une fonction cuspidale. Alors il existe une fonction très cuspidale f 0 ∈ Cc∞ (G(F )) telle que D(f ) = D(f 0 ) pour toute distribution D sur G(F ) invariante par conjugaison. Cet affaiblissement de la condition sur f est nécessaire pour achever la preuve (on prend pour f un pseudo-coefficient d’une représentation irréductible, tempérée et elliptique). Le deuxième point est l’apparition de la condition m(ρ, π) = 1 dans le terme Jspec (θρ , f ). Fixons ici une représentation π ∈ Temp(G). L’espace HomH,ξ (π, ρ) dont m(ρ, π) est la dimension est défini de façon abstraite. Il ne peut pas intervenir directement dans Jspec (θρ , f ) qui est une intégrale explicite. Ce qui intervient dans ce terme, c’est la forme sesquilinéaire L π,ρ sur Eρ ⊗C Eπ définie par Z ¯ L π,ρ (ε0 ⊗ e0 , ε ⊗ e) = (ρ(h)ε0 , ε)(e0 , π(hu)e)ξ(u)du dh, H(F )U (F )

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

175

pour ε, ε0 ∈ Eρ et e, e0 ∈ Eπ (les produits (., ) sont des produits hermitiens invariants sur Eρ et Eπ ). L’intégrale ci-dessus n’est pas absolument convergente, mais on peut la définir comme une limite d’intégrales absolument convergentes, cf. 5.1. Négligeons cette question de convergence. Fixons ε et e0 . Définissons une application l : Eπ → Eρ par l’égalité (ε0 , l(e)) = L π,ρ (ε0 ⊗ e0 , ε ⊗ e) pour tous ε0 ∈ Eρ et e ∈ Eπ . On vérifie que l ∈ HomH,ξ (π, ρ). Si L π,ρ n’est pas nulle, cet espace HomH,ξ (π, ρ) ne l’est pas non plus et m(ρ, π) = 1. On a besoin de la réciproque, qui s’avère vraie. Proposition. — Soient π ∈ Temp(G) et ρ ∈ Temp(H). Alors m(ρ, π) = 1 si et seulement si la forme sesquilinéaire L π,ρ est non nulle. Cf. proposition 5.7. Signalons que cette façon concrète de construire l’espace HomH,ξ (π, ρ) se trouve déjà dans l’article [13] de Ikeda et Ichino. La première section est consacrée aux notations et à des rappels sur les opérateurs d’entrelacement et la formule de Plancherel. La deuxième l’est aux propriétés des fonctions cuspidales ou très cuspidales et aux quasi-caractères qu’elles permettent de définir. Les sections 3 et 4 sont franchement pénibles. On y démontre diverses majorations nécessaires pour la suite (pour le groupe GLk dans la section 3, pour un groupe spécial orthogonal dans la section 4). On s’inspire ici plus que largement des travaux d’Harish-Chandra. Signalons à ce propos que l’on fait constamment référence à l’article [16]. Mais l’apparence est trompeuse puisque dans [16], on s’était contenté de rédiger des résultats non publiés d’Harish-Chandra. D’autre part, dans [16], on avait cru judicieux de modifier la définition de l’homomorphisme habituel HG en y glissant un signe −. On persiste à penser que, sur un corps de base p-adique, c’est une meilleure définition. Mais, pour utiliser les résultats d’Arthur, il vaut mieux reprendre ses définitions. C’est ce que l’on fait, mais cela induit des changements de signe dans les références que l’on fera à [16] : cela échange une chambre positive avec son opposée. La section 5 est consacrée à la définition et l’étude des formes sesquilinéaires L π,ρ évoquées ci-dessus. La preuve de l’égalité limN →∞ JN (θρ , f ) = Jspec (θρ , f ) se trouve dans la section 6. Il s’agit pour l’essentiel de recopier [6]. On en déduit dans la section 7 les deux théorèmes énoncés ci-dessus. Je remercie le rapporteur pour ses remarques et suggestions, et pour le tact avec lequel il les a faites.

1. Notations et rappels 1.1. Notations générales. — On utilise les notations introduites dans [17], qui sont la plupart du temps celles d’Arthur et d’Harish-Chandra. Soit F un corps local non archimédien de caractéristique nulle. On note oF son anneau d’entiers, pF l’idéal maximal de oF , q le nombre d’éléments du corps résiduel, valF et |.|F les valuation et valeur absolue usuelles et on fixe une uniformisante $F . Soit G un groupe réductif connexe défini sur F . On note g l’algèbre de Lie de G. On note AG le plus grand

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

176

J.-L. WALDSPURGER

tore déployé central dans G, X(G) le groupe des caractères de G définis sur F , A G = Hom(X(G), R) et A ∗G = X(G) ⊗Z R le dual de A G . On définit l’homomorphisme habituel HG : G(F ) → A G . On note A G,F , resp. ˜A G,F , l’image de G(F ), resp. AG (F ), ∗ ˜∨ par cet homomorphisme. On note A ∨ G,F , resp. A G,F , le sous-groupe des λ ∈ A G tels que λ(ζ) ∈ 2πZ pour tout ζ ∈ A G,F , resp. ζ ∈ ˜A G,F . On note aG la dimension de

AG. Soit K un sous-groupe compact spécial de G(F ). Soit P = M U un sous-groupe parabolique de G. Rappelons nos conventions : P est implicitement supposé défini sur F et la notation P = M U signifie que M est une composante de Levi de P , définie sur F , et U est le radical unipotent de P . Supposons que K soit en bonne position relativement à M . Précisément, il existe un sous-tore déployé maximal A0 de M tel que K fixe un point de l’appartement associé à A0 dans l’immeuble de G. On a l’égalité G(F ) = M (F )U (F )K. Pour tout g ∈ G(F ), on fixe des éléments mP (g) ∈ M (F ), uP (g) ∈ U (F ), kP (g) ∈ K tels que g = mP (g)uP (g)kP (g). On prolonge l’application HM : M (F ) → A M en une fonction HP : G(F ) → A M par HP (g) = HM (mP (g)). Ce dernier terme est bien défini puisque la classe mP (g)(M (F ) ∩ K) ne dépend pas de la décomposition choisie de g. Supposons fixé un Levi minimal Mmin de G. On pose W G = NormG(F ) (Mmin )/Mmin (F ). On note ΞG la fonction d’Harish-Chandra ([16, II.1]). Elle dépend de K. Mais elle ne nous sert qu’à résoudre des questions de majorations. Or changer de groupe K remplace ΞG par une fonction équivalente. Il est donc loisible d’utiliser cette fonction sans préciser le groupe K qui permet de la définir. On utilise aussi la fonction σ. Rappelons que, dans [17], on a légèrement modifié la définition d’Harish-Chandra en posant σ(g) = sup(1, log(||g||)). On a la relation σ(gg 0 ) ≤ σ(g) + σ(g 0 ) ≤ 2σ(g)σ(g 0 ) pour tous g, g 0 ∈ G(F ). Pour tout réel b ≥ 0, on note 1σ 0 tel que a ≤ cb pour tous x1 , . . . , xn . Cette notation est quelque peu imprécise mais nous évite d’introduire une kyrielle de constantes superflues. On introduit l’espace S (G(F )) des fonctions de Schwartz-Harish-Chandra sur G(F ). C’est l’ensemble des fonctions f : G(F ) → C qui sont biinvariantes par un sous-groupe ouvert compact et telles que, pour tout réel R ≥ 0, on ait une majoration |f (g)|  ΞG (g)σ(g)−R pour tout g ∈ G(F ). L’espace S (G(F )) contient l’espace Cc∞ (G(F )) des fonctions localement constantes à support compact. Soit π une représentation admissible de G(F ). On note sans plus de commentaire Eπ un espace complexe dans lequel elle se réalise. Si K est un sous-groupe de G(F ), on note EπK le sous-espace des éléments de Eπ invariants par K. Supposons π unitaire. On

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

177

fixe une forme hermitienne définie positive (., .) sur Eπ invariante par l’action de G(F ). On appelle une telle forme un produit scalaire invariant. Précisons notre convention ¯ 0 λ(e0 , e) sur les formes sesquilinéaires : la forme (., .) vérifie la relation (λ0 e0 , λe) = λ 0 0 pour tous λ, λ ∈ C, e, e ∈ Eπ . Nous dirons que π est tempérée si elle est unitaire, de longueur finie, et qu’il existe un entier D tel que, pour tous e, e0 ∈ Eπ , on ait une majoration |(e0 , π(g)e)|  ΞG (g)σ(g)D . En fait, l’entier D ne sert à rien : on peut prendre D = 0, cf. [16] lemme VI.2.2. Supposons G(F ) muni d’une mesure de Haar. Si π est tempérée, l’action de Cc∞ (G(F )) dans Eπ se prolonge en une action de S (G(F )). Pour f ∈ S (G(F )) et e, e0 ∈ Eπ , on a l’égalité Z (e0 , π(f )e) =

f (g)(e0 , π(g)e)dg.

G(F )

Cette intégrale est absolument convergente. On note Temp(G) l’ensemble des classes d’équivalence de représentations tempérées irréductibles de G(F ). On fixe un caractère ψ : F → C× , continu et non trivial. On note cψ le plus petit entier relatif c tel que ψ soit trivial sur pcF . 1.2. Mesures. — Dans la suite de l’article, la situation sera la suivante. Le groupe G est fixé, ainsi qu’un Levi minimal Mmin de G et un sous-groupe compact spécial K de G(F ), en bonne position relativement à Mmin . On munit K de la mesure de Haar de masse totale 1. On munit G(F ) d’une mesure de Haar (pour laquelle mes(K) n’est pas forcément égale à 1). Soit P = M U ∈ F (Mmin ) (les notations F (L), P (L), L (L) sont celles d’Arthur : pour un Levi L, elles désignent l’ensemble des sous-groupes paraboliques Q qui contiennent L, resp. de composante de Levi L, resp. l’ensemble des Levi qui contiennent L). On munit U (F ) de l’unique mesure de Haar telle que Z δP¯ (mP¯ (u))du = 1, U (F )

¯ est le parabolique opposé à P et δP¯ est le module usuel. On munit M (F ) où P¯ = M U de l’unique mesure de Haar telle que, pour toute f ∈ Cc∞ (G(F )), on ait l’égalité Z Z Z Z f (g)dg = f (muk)dm du dk. G(F )

K

U (F )

M (F )

Le point est que cette mesure sur M (F ) ne dépend pas du sous-groupe parabolique P ∈ P (M ) utilisé pour la définir ([6] 1.2). On munit l’espace i A ∗M ⊂ A ∗M ⊗R C de la mesure de Haar telle que le quotient ∨ ∗ i A M /i ˜A M,F soit de mesure 1. On pose i A ∗M,F = i A ∗M /i A ∨ M,F et on le munit de la ∗ ∗ mesure telle que l’application naturelle de i A M dans i A M,F préserve localement les mesures. Soit T un tore. Si T est déployé, on munit T (F ) de la mesure de Haar telle que le sous-groupe compact maximal de T (F ) soit de mesure 1. En général, on munit AT (F )

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

178

J.-L. WALDSPURGER

de la mesure que l’on vient de définir et T (F ) de la mesure telle que T (F )/AT (F ) soit de mesure 1 pour la mesure quotient. Remarques. — 1. Dans le cas où Mmin est un tore, les définitions précédentes peuvent entrer en conflit. On croit qu’en pratique, il n’y aura pas d’ambiguïté. 2. Dans les sections 3, 4 et 5, on se préoccupera de questions de convergence pour lesquelles les choix de mesures sont sans importance. On ne tiendra pas compte des normalisations ci-dessus. On supposera au contraire que les mesures sont choisies de telle sorte que toutes les constantes qui apparaissent à cause d’elles dans les calculs soient égales à 1. 1.3. Représentations induites, opérateurs d’entrelacement. — Soit P = M U un sous-groupe parabolique de G et τ une représentation admissible de M (F ). G On définit la représentation induite IndG P (τ ). On note EP,τ son espace. C’est celui des fonctions e : G(F ) → Eτ qui sont invariantes à droite par un sous-groupe ouvert compact de G(F ) et vérifient e(mug) = δP (m)1/2 τ (m)e(g) pour tous m ∈ M (F ), u ∈ U (F ) et g ∈ G(F ). Pour g ∈ G(F ), ou f ∈ Cc∞ (G(F )), ∗ G G on note IndG P (τ, g), ou IndP (τ, f ), l’action de g, ou f , dans EP,τ . Pour λ ∈ A M ⊗R C, on définit la représentation τλ de M (F ) par τλ (m) = exp(λ(HM (m)))τ (m) et la représentation induite IndG P (τλ ). Remarquons que ces représentations ne dépendent G que de l’image de λ dans ( A ∗M ⊗R C)/i A ∨ M,F . Supposons Mmin ⊂ M . Notons K P,τ l’espace des fonctions e : K → Eτ qui sont invariantes à droite par un sous-groupe ouvert compact de K et vérifient la même relation que ci-dessus, pour m ∈ K ∩M (F ), G u ∈ K ∩ U (F ) et g ∈ K. Par restriction à K, EP,τ s’identifie à K G P,τ , ce dernier λ G espace est donc un modèle commun à toutes les représentations IndP (τλ ). Supposons τ unitaire. On définit un produit hermitien sur K G P,τ par Z (e0 (k), e(k))dk. (e0 , e) = K

C’est un produit scalaire invariant pour la représentation IndG P (τλ ) pour tout λ ∈ i A ∗M,F . Laissons M fixé mais faisons varier P parmi les éléments de P (M ). Pour P = M U, P 0 = M U 0 ∈ P (M ) et λ ∈ A ∗M ⊗R C, on définit l’opérateur d’entrelacement G JP 0 |P (τλ ) : EP,τ → EPG0 ,τλ λ

Quand la partie réelle de λ est dans un certain cône, il est défini par la formule Z (JP 0 |P (τλ )e)(g) = e(u0 g)du0 . (U (F )∩U 0 (F ))\U 0 (F )

En général, il est défini par prolongement méromorphe (il est même rationnel, si l’on considère ( A ∗M ⊗R C)/i A ∨ M,F comme un tore algébrique complexe). Par restriction

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

179

G à K, on peut considérer JP 0 |P (τλ ) comme un homomorphisme de K G P,τ dans K P 0 ,τ . C’est ce point de vue que l’on adopte dans la suite. Supposons τ irréductible. L’opérateur JP |P¯ (τλ )JP¯ |P (τλ ) est une homothétie. Notons j(τλ ) le rapport d’homothétie. Il ne dépend pas de P . On peut normaliser l’opérateur d’entrelacement. On introduit une fonction rP 0 |P (τλ ) à valeurs complexes, qui est méromorphe et même rationnelle, de sorte qu’en posant

RP 0 |P (τλ ) = rP 0 |P (τλ )−1 JP 0 |P (τλ ), cet opérateur vérifie les conditions du théorème 2.1 de [5]. Les principales conditions sont — pour P, P 0 , P 00 ∈ P (M ), RP 00 |P 0 (τλ )RP 0 |P (τλ ) = RP 00 |P (τλ ) ; — supposons τ tempérée ; pour λ ∈ i A ∗M,F , RP 0 |P (τλ ) est holomorphe et son adjoint pour le produit scalaire est RP |P 0 (τλ ). La définition des opérateurs normalisés s’étend au cas où τ est semi-simple. En particulier, soit P M = M0 U0M un sous-groupe parabolique de M tel que Mmin ⊂ M0 , soit τ0 une représentation tempérée irréductible de M0 (F ), supposons que τ = M IndM P M (τ0 ). Pour P = M U ∈ P (M ), introduisons le groupe P0 = P U ∈ P (M0 ). ∗ G G L’espace K P,τ s’identifie à K P0 ,τ0 . Pour P, P 0 ∈ P (M ) et λ ∈ i A M,F , l’opérateur RP 0 |P (τλ ) s’identifie à RP00 |P0 (τ0,λ ). Cette propriété résulte de la construction des fonctions de normalisation rP 0 |P (τλ ), cf. [5, 2.2]. 1.4. Caractères pondérés. — On conserve les données M et τ du paragraphe précédent. On suppose que τ est tempérée, donc semi-simple d’après la définition que l’on a adoptée. Pour tous P, P 0 ∈ P (M ), l’opérateur RP 0 |P (τ ) est bien défini et inversible. Plus généralement, pour tout λ ∈ i A ∗M , l’opérateur RP 0 |P (τλ ) est bien défini et inversible. Fixons P . Pour tout P 0 ∈ P (M ), considérons la fonction R P 0 (τ ) sur i A ∗M définie par

RP 0 (τ, λ) = RP 0 |P (τ )−1 RP 0 |P (τλ ). Elle prend ses valeurs dans l’espace d’endomorphismes de K G P,τ . La famille ( R P 0 (τ ))P 0 ∈ P (M ) est une (G, M )-famille à valeurs opérateurs ([2] paragraphe 7). Cela entraîne que la fonction X λ 7→ RP 0 (τ, λ)θP 0 (λ)−1 P 0 ∈ P (M )

sur i A ∗M est C ∞ (la fonction θP 0 est définie en [7] p. 15). On note R M (τ ) la valeur de cette fonction en λ = 0. C’est un endomorphisme de K P,τ . Il dépend du choix de P puisque l’espace dans lequel il agit en dépend, mais il n’en dépend qu’à isomorphisme ˜ ∈ L (M ) et Q = LU ∈ F (M ˜ ). On définit une près. Plus généralement, soient M Q Q ˜ )-famille ( R ˜ L (τ )) ˜ L L ˜ de la façon suivante : R ˜ L (τ ) est la restriction à (L, M P ∈ P (M ) P P ∗ i A M˜ de la fonction R P 0 (τ ), où P 0 est un élément quelconque de P (M ) tel que P 0 ⊂ Q et P 0 ∩ L ⊂ P˜ L (les deux termes de cette dernière inclusion sont des sous-groupes

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

180

J.-L. WALDSPURGER

˜ pour le second). paraboliques de L, de composantes de Levi M pour le premier et M Q ˜ Comme ci-dessus, on associe à cette (L, M )-famille un opérateur R M˜ (τ ). G Le caractère pondéré de la représentation τ est la distribution f 7→ JM (τ, f )définie par G JM (τ, f ) = trace( R M (τ )IndG P (τ, f )) ∞ pour toute f ∈ Cc (G(F )). Cette distribution est définie à l’aide du sous-groupe parabolique P que nous avons fixé, mais on montre qu’elle ne dépend pas de ce choix ˜ (parce que R M (τ ) n’en dépend qu’à isomorphisme près). Plus généralement, pour M Q et Q comme ci-dessus, on définit une distribution f 7→ JM ˜ (τ, f ) en remplaçant R M (τ ) Q

par R M˜ (τ ). G Dans le cas où M = G, on pose simplement θτ (f ) = JG (τ, f ). La distribution f 7→ θτ (f ) est le caractère usuel de τ . 1.5. Le R-groupe. — Soient M un Levi de G et τ une représentation admissible de M (F ). Soit g ∈ G(F ). On définit la représentation gτ de gM (F )g −1 par (gτ )(gmg −1 ) = τ (m). Son espace Egτ est égal à Eτ . Sa classe d’isomorphie ne dépend que de l’image de g dans l’ensemble de classes G(F )/M (F ). La conjugaison par g induit un isomorphisme de A ∗M ⊗R C sur A ∗gM g−1 ⊗R C que l’on note λ 7→ gλ. On a l’égalité g(τλ ) = (gτ )gλ pour tout λ ∈ A ∗M ⊗R C. Supposons Mmin ⊂ M et τ irréductible et de la série discrète. Notons NormG(F ) (τ ) le sous-groupe des g ∈ NormG(F ) (M ) tels que gτ ' τ . Posons W (τ ) = NormG(F ) (τ )/M (F ). Simplifions la théorie en supposant vérifiées la condition : (1) la représentation τ se prolonge en une représentation τ N de NormG(F ) (τ ). Fixons ce prolongement τ N . Remarquons que τ N est nécessairement unitaire. Pour P ∈ P (M ) et w ∈ W (τ ), on définit un homomorphisme G AP (w) : K G w−1 P w,w−1 τ → K P,τ

de la façon suivante. Puisque K est en bonne position relativement à Mmin , a fortiori relativement à M , on peut choisir un relèvement w˙ de w qui appartient à K ∩ NormG(F ) (τ ). Pour e ∈ K G w−1 P w,w−1 τ et k ∈ K, on pose (AP (w)e)(k) = τ N (w)e( ˙ w˙ −1 k). Cela ne dépend pas du choix de w. ˙ Pour λ ∈ i A ∗M,F , G on définit un endomomorphisme RP (w, τλ ) de K P,τ par RP (w, τλ ) = AP (w)Rw−1 P w|P (τλ ) = RP |wP w−1 ((wτ )wλ )AwP w−1 (w). C’est un opérateur unitaire. On a la relation d’entrelacement G IndG P ((wτ )wλ , g)RP (w, τλ ) = RP (w, τλ )IndP (τλ , g)

et la relation de composition RP (w1 w2 , τλ ) = RP (w1 , τw2 λ )RP (w2 , λ). Apppliquons ceci pour λ = 0. Notons W 0 (τ ) le sous-groupe des w ∈ W (τ ) tels que RP (w, τ ) soit une homothétie. C’est le groupe de Weyl d’un système de racines Σ0 dont

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

181

tout élément est proportionnel à une racine de AM dans g. Ce système est conservé par l’action de W (τ ). Fixons un sous-ensemble Σ0 + de racines positives et notons R(τ ) le sous-groupe des éléments de W (τ ) qui conservent Σ0 + . On a la décomposition W (τ ) = W 0 (τ )oR(τ ). L’application w 7→ RP (w, τ ) se prolonge en un isomorphisme de l’algèbre de groupe C[R(τ )] sur l’algèbre commutante de la représentation IndG P (τ ). Ces propriétés forment la théorie du R-groupe, qui est due à Silberger dans le cas p-adique. Simplifions encore en supposant : (2) le groupe R(τ ) est abélien. On note R(τ )∨ le groupe dual de R(τ ). Pour tout caractère ζ ∈ R(τ )∨ , notons G KG P,τ,ζ le sous-espace des éléments e ∈ K P,τ tels que RP (w, τ )e = ζ(w)e pour tout G G w ∈ R(τ ). Alors K G P,τ,ζ est stable par la représentation IndP (τ ). Notons IndP (τ, ζ) la G restriction de IndG P (τ ) à ce sous-espace. Alors IndP (τ, ζ) est irréductible, sa classe ne G 0 0 dépend pas de P et IndP (τ, ζ) est isomorphe à IndG P (τ, ζ ) si et seulement si ζ = ζ . Soit π une représentation admissible de G(F ). On dit qu’elle est proprement induite s’il existe un élément Q = LU ∈ F (Mmin ) et une représentation admissible irréductible σ de L(F ) tels que Q 6= G et π ' IndG Q (σ). Remarquons que si π est tempérée, σ l’est forcément elle-aussi. Introduisons le groupe de Grothendieck à coefficients dans Q des représentations admissibles de longueur finie de G(F ). Soit π ∈ Temp(G). On dit que π est elliptique si son image dans ce groupe n’est pas combinaison linéaire de représentations proprement induites. En utilisant la classification de Langlands, on voit qu’il revient au même de dire que cette image n’est pas combinaison linéaire de représentations tempérées proprement induites. Il revient aussi au même de dire que le caractère de π n’est pas identiquement nul sur l’ensemble des éléments semisimples réguliers et elliptiques de G(F ). Revenons à la situation précédente. Notons W (M ) = NormG(F ) (M )/M (F ) et W (M )reg le sous-ensemble des éléments de W (M ) qui opèrent sans points fixes non nuls sur A M / A G . Les trois conditions suivantes sont équivalentes : — il existe ζ ∈ R(τ )∨ tel que IndG P (τ, ζ) soit elliptique ; — pour tout ζ ∈ R(τ )∨ , la représentation IndG P (τ, ζ) est elliptique ; — l’intersection R(τ ) ∩ W (M )reg n’est pas vide. Cela résulte de [7] proposition 2.1(c), compte tenu du fait qu’un caractère d’un groupe abélien ne s’annule en aucun point de ce groupe. Si ces conditions sont vérifiées, on a W 0 (τ ) = {1}. Inversement, pour toute représentation elliptique π ∈ Temp(G), il existe M , τ et ζ vérifiant toutes les conditions ci-dessus de sorte que π ' IndG P (τ, ζ). La classe de conjugaison par W G du triplet (M, τ, ζ) est bien déterminée. Remarque. — Les hypothèses simplificatrices (1) et (2) sont vérifiées si G est un groupe « classique », en particulier si G est spécial orthogonal, cf. [12] proposition 2.3 et [9]. Les auteurs de ces références imposent des hypothèses au groupe G, qui sont nécessaires pour déterminer entièrement les R-groupes, mais qui ne le sont pas pour les propriétés plus rudimentaires ci-dessus.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

182

J.-L. WALDSPURGER

1.6. La formule de Plancherel-Harish-Chandra. — Pour tout M ∈ L (Mmin ), fixons un élément P ∈ P (M ). Notons Π2 (M ) l’ensemble des classes d’isomorphie de représentations irréductibles et de la série discrète de M (F ). Cet ensemble se décompose en orbites pour l’action τ 7→ τλ de i A ∗M,F . Notons {Π2 (M )} l’ensemble des orbites. Pour chaque orbite O, fixons un élément τ de cette orbite. Notons i A ∨O le groupe des λ ∈ i A ∗M tels que les représentations τ et τλ soient équivalentes. Pour tout λ ∈ i A ∗M , on définit la mesure de Plancherel m(τλ ) = j(τλ )−1 d(τ ), où d(τ ) est le degré formel de τ . Soit f ∈ S (G(F )). La formule de Plancherel-HarishChandra affirme l’égalité X X −1 [i A ∨O : i A ∨ f (g) = |W M ||W G |−1 M,F ] M ∈ L (Mmin )

Z i A∗ M,F

O∈{Π2 (M )}

−1 m(τλ )trace(IndG )IndG P (τλ , g P (τλ , f ))dλ

pour tout g ∈ G(F ). Seules interviennent de façon non nulle les orbites O pour lesquelles une représentation IndG P (τλ ) admet des vecteurs non nuls invariants par un sous-groupe ouvert compact de G(F ) tel que f soit biinvariante par ce sous-groupe. Ces orbites sont en nombre fini. La formule ci-dessus est démontrée dans [16] théorème VIII.1.1. Dans cette référence, il y a quelques constantes supplémentaires dues aux normalisations différentes des mesures. Arthur a introduit les normalisations que nous utilisons et qui font disparaître ces constantes. Nous aurons aussi besoin d’une autre formule. Fixons P = M U ∈ F (Mmin ) et une représentation admissible irréductible τ de M (F ), de la série discrète. Soient ∞ e, e0 ∈ K G sur i A ∗M,F . Définissons une fonction fe,e0 ,ϕ sur P,τ et ϕ une fonction C G(F ) par Z 0 fe,e0 ,ϕ (g) = ϕ(λ)(IndG P (τλ , g)e , e)m(τλ )dλ. i A∗ M,F

Cette fonction appartient à S (G(F )). Identifions tout élément de W (M ) à un représentant dans K ∩ NormG(F ) (M ). Notons E(τ ) l’ensemble des couples (w, µ) ∈ W (M ) × i A ∗M,F tels que w−1 τ ' τµ . Pour (w, µ) ∈ E(τ ), fixons un automorphisme unitaire τ (w, µ) de K G P,τ tel que τ (w, µ)τµ (m) = (w−1 τ )(m)τ (w, µ) G pour tout m ∈ M (F ). Définissons l’homomorphisme A(w, µ) : K G w−1 P w,τ → K P,τ par

(A(w, µ)e)(g) = τ (w, µ)e(w−1 g). Pour λ ∈ i A ∗M,F , définissons l’endomorphisme R(w, µ, λ) de K G P,τ par R(w, µ, λ) = A(w, µ)Rw−1 P w|P (τλ+µ ).

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

183

Il vérifie la relation d’entrelacement G R(w, µ, λ)IndG P (τλ+µ , g) = IndP (τwλ , g)R(w, µ, λ).

Posons simplement R(w, µ) = R(w, µ, 0). Soient e0 , e00 ∈ K G P,τ . Alors on a l’égalité Z X fe,e0 ,ϕ (g)(e00 , IndG ϕ(µ)(R(w, µ)e0 , e0 )(e00 , R(w, µ)e). P (τ, g)e0 )dg = G(F )

(w,µ)∈ E(τ )

C’est une autre façon d’écrire la proposition VII.2 de [16]. Ici encore, les constantes disparaissent grâce aux normalisations d’Arthur. L’ensemble E(τ ) est fini, on peut donc choisir un voisinage ω de 0 dans i A ∗M,F tel que (w, µ) ∈ E(τ ) et µ ∈ ω entraînent µ = 0. Evidemment l’application w 7→ (w, 0) est un isomorphisme de W (τ ) sur le sous-ensemble des éléments de E(τ ) de la forme (w, 0). Supposons valides les hypothèses du paragraphe précédent. Pour w ∈ W (τ ), l’opérateur R(w, 0) est égal au RP (w, τ ) du paragraphe précédent (du moins, on peut effectuer les divers choix de sorte qu’il en soit ainsi). Pour w ∈ W 0 (τ ), c’est une homothétie dont le rapport est de module 1 par un argument d’unitarité. Seuls les éléments du R-groupe interviennent de façon non triviale dans la somme ci-dessus. On obtient alors (1) supposons le support de ϕ contenu dans ω ; alors on a l’égalité Z X 0 fe,e0 ,ϕ (g)(e00 , IndG (RP (w, τ )e0 , e0 )(e00 , RP (w, τ )e). P (τ, g)e0 )dg = |W (τ )|ϕ(0) G(F )

w∈R(τ )

2. Fonctions très cuspidales 2.1. Un lemme d’annulation. — Soit π une représentation admissible de G(F ). Introduisons sa contragrédiente π ˇ . Soient B une forme bilinéaire sur Eπˇ × Eπ et f ∈ Cc∞ (G(F )). Fixons un sous-groupe ouvert compact Kf de G(F ) tel que f soit K K biinvariante par Kf . Les sous-espaces Eπ f et Eπˇ f sont duaux l’un de l’autre par K l’accouplement naturel, que l’on note < ., . >. Fixons une base B f du premier et Kf introduisons la base duale {ˇ e; e ∈ B } du second. Posons X traceB (π(f )) = B(ˇ e, π(f )e). e∈ BKf

On vérifie que ce terme ne dépend ni du choix de Kf , ni de celui de la base. Remarquons que la trace usuelle θπ (f ) s’obtient comme cas particulier en prenant pour B l’accouplement naturel sur Eπˇ × Eπ . Soient P = M U ∈ F (Mmin ) et τ une représentation admissible de M (F ). Soient G G ∞ B une forme bilinéaire sur EP,ˇ τ × EP,τ et f ∈ Cc (G(F )). On impose les hypothèses suivantes (1) P 6= G ;

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

184

J.-L. WALDSPURGER

(2) f est très cuspidale (cf. [17] 5.1) ; G G 0 (3) soient e ∈ EP,τ et e0 ∈ EP,ˇ τ tels que e (g) ⊗ e(g) = 0 pour tout g ∈ G(F ) ; alors 0 B(e , e) = 0. Remarque. — e0 (g) ⊗ e(g) est un élément de Eτˇ ⊗C Eτ . Lemme. — Sous ces hypothèses, on a traceB (IndG P (τ, f )) = 0. Démonstration. — On fixe un sous-groupe ouvert compact Kf de K tel que f soit biinvariante par Kf . Fixons un ensemble de représentants Γ de l’ensemble de doubles K G Kf classes P (F )\G(F )/Kf . On peut choisir une base B f de (EP,τ ) telle que, pour K

tout e ∈ B f , il existe γ ∈ Γ de sorte que le support de e soit contenu dans P (F )γKf . L’élément correspondant eˇ de la base duale vérifie la même propriété, avec le même K γ. Pour tout e ∈ B f , on a X 0 IndG < eˇ0 , IndG P (τ, f )e = P (τ, f )e > e , e0 ∈ BKf

d’où traceB (IndG P (τ, f )) =

X

B(ˇ e, e0 ) < eˇ0 , IndG P (τ, f )e > .

e,e0 ∈ BKf K

Fixons e, e0 ∈ B f . Il suffit de prouver que le terme indexé par e, e0 dans cette somme est nul. Soient γ, γ 0 ∈ Γ tels que le support de e, resp. e0 , soit contenu dans P (F )γKf , resp. P (F )γ 0 Kf . Si γ 6= γ 0 , on a eˇ(g)⊗e0 (g) = 0 pour tout g ∈ G(F ), donc B(ˇ e, e0 ) = 0. Supposons γ = γ 0 . On a Z Z < eˇ0 , IndG (τ, f )e >= f (g) < eˇ0 (x), e(xg) > dg dx, P K

G(F )

où l’accouplement intérieur est celui sur Eτˇ ×Eτ . On effectue le changement de variable g 7→ x−1 g puis on décompose g en g = muk, avec m ∈ M (F ), u ∈ U (F ), k ∈ K. D’où Z Z Z Z (τ, f )e >= f (x−1 muk) < eˇ0 , IndG P K

K

M (F )

U (F )

< eˇ0 (x), τ (m)e(k) > δP (m)1/2 du dm dk dx. Fixons x, k, m et supposons < eˇ0 (x), τ (m)e(k) >6= 0. Cela entraîne x ∈ P (F )γKf et k ∈ P (F )γKf . Donc k ∈ P (F )xKf . Ecrivons k = m0 u0 xk 0 , avec m0 ∈ M (F ), u0 ∈ U (F ), k 0 ∈ Kf et considérons l’intégrale intérieure de la formule ci-dessus. Puisque f est invariante à droite par Kf , le k 0 disparaît. Par le changement de variable u 7→ m0 uu0 −1 m0 −1 , cette intégrale devient Z δP (m0 )1/2 f (x−1 mm0 ux)du. U (F )

Elle est nulle puisque f est très cuspidale. Donc < eˇ0 , IndG P (τ, f )e >= 0, ce qui achève la preuve.

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

185

Ce lemme admet plusieurs variantes. Supposons Mmin ⊂ M . Au lieu des modèles G G G G EP,τ et EP,ˇ τ et une forme bilinéaire τ , on peut aussi considérer les modèles K P,τ et K P,ˇ G G B sur K P,ˇτ × K P,τ . Le lemme reste valide si l’on remplace l’hypothèse (3) par G 0 0 (3’) soient e ∈ K G P,τ et e ∈ K P,ˇ τ tels que e (k) ⊗ e(k) = 0 pour tout k ∈ K ; alors 0 B(e , e) = 0. Dans le cas où τ unitaire, on peut aussi considérer une forme sesquilinéaire B sur K P,τ × K P,τ vérifiant la même condition (3’). Le lemme reste valide. 2.2. Caractères pondérés et fonctions très cuspidales. — Soient M ∈ G L (Mmin ) et f ∈ Cc∞ (G(F )). Comme on l’a dit, le caractère pondéré JM (τ, f ) est défini pour toute représentation τ tempérée (ce qui impose, d’après nos conventions, que τ semi-simple et de longueur finie). Plus généralement, on peut prolonger la définition par linéarité à une combinaison linéaire finie de représentations tempérées irréductibles. Fixons une telle combinaison linéaire τ . Proposition. — Supposons f très cuspidale. ˜ ∈ L (M ) et Q ∈ F (M ˜ ). Si M ˜ 6= M ou si Q 6= G, on a J Q (τ, f ) = 0. (i) Soient M ˜ M (ii) Supposons que τ est combinaison linéaire de représentations proprement inG (τ, f ) = 0. duites. Alors JM Démonstration. — Pour prouver (i), on peut supposer τ irréductible. Démontrons ˜ = M et Q 6= G. Soit donc Q = LU ∈ F (M ) tel que d’abord l’assertion pour M Q 6= G. Fixons P ∈ P (M ) tel que P ⊂ Q. Pour P 0 ∈ P (M ) tel que P 0 ⊂ Q, définissons une fonction cP 0 sur i A ∗M par cP 0 (λ) = trace(RP 0 |P (τ )−1 RP 0 |P (τλ )IndG P (τ, f )). Q Il s’agit de la trace d’un endomorphisme de K G P,τ . Par définition, JM (τ, f ) est la valeur en λ = 0 de la fonction X cP 0 (λ)θPL 0 ∩L (λ)−1 . P 0 ∈ P (M );P 0 ⊂Q Q Pour démontrer que JM (τ, f ) = 0, il suffit de prouver que cP 0 (λ) = 0 pour tous P 0 et L 0 λ. Fixons P 0 et λ. Introduisons les représentations π = IndL P ∩L (τ ) et π = IndP 0 ∩L (τ ), L G que l’on réalise dans les espaces K L P ∩L,τ et K P 0 ∩L,τ . On peut identifier K P,τ , resp. G G G K P 0 ,τ , à K Q,π , resp. K Q,π0 . On dispose de l’opérateur RPL 0 ∩L|P ∩L (τλ ) : K LP ∩L,τ →

K LP 0 ∩L,τ . Modulo les identifications précédentes, RP 0 |P (τλ ) s’identifie à l’opérateur G G L e 7→ RP 0 ∩L|P ∩L (τλ ) ◦ e de K Q,π dans K Q,π 0 . Il en est de même pour l’opérateur G RP 0 |P (τ ). Introduisons la forme sesquilinéaire B sur K G Q,π 0 × K Q,π définie par L −1 L B(e0 , e) = (e0 , RP RP 0 ∩L|P ∩L (τλ ) ◦ e). 0 ∩L|P ∩L (τ )

On a alors cP 0 (λ) = traceB (π(f )). Les conditions du lemme précédent sont vérifiées, si l’on remplace dans ce lemme P et τ par Q et π. Le lemme entraîne cP 0 (λ) = 0 comme on le voulait.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

186

J.-L. WALDSPURGER

G

Passons au cas général. On peut appliquer à la (G, M )-famille ( R P L (τ ))P L ∈ P L (M ) les formules de descente d’Arthur, en particulier le corollaire 7.2 de [4]. On en déduit l’égalité X 0 Q ˜ 0 Q JM dL M (M , L )JM (τ, f ). ˜ (τ, f ) = L0 ∈ L L (M )

Le sous-groupe parabolique Q0 appartient à P (L0 ) et est contenu dans Q. Si Q 6= G, ˜ 6= M car dans ce cas, tous ces Q0 sont aussi différents de G. Il en est de même si M L 0 0 ˜ la condition dM (M , L ) 6= 0 implique que L ( L. Alors le résultat précédent entraîne Q JM ˜ (τ, f ) = 0, ce qui prouve (i). Pour (ii), il suffit de considérer le cas où τ est proprement induite. Fixons P 0 = 0 0 M U ∈ F M (Mmin ) et une représentation irréductible τ 0 de M 0 (F ) tels que P 0 6= M 0 0 et τ = IndM P 0 (τ ). La représentation τ est tempérée. Il résulte des définitions (cf. la fin du paragraphe 1.3) que l’on a l’égalité G G JM (τ, f ) = JM (τ 0 , f ).

˜ par M 0 et M . On obtient la nullité du terme On applique le (i) en remplaçant M et M de droite ci-dessus, d’où celle du terme de gauche que l’on voulait démontrer. G Pour f ∈ Cc∞ (G(F )), le terme JM (τ, f ) dépend des facteurs rP 0 |P (τ, λ) utilisés pour définir les opérateurs d’entrelacement normalisés. En fait G (1) pour f très cuspidale, JM (τ, f ) ne dépend pas des facteurs de normalisation. En effet, considérons deux familles de facteurs, que l’on indexe par les nombres 1 et 2. On en déduit deux (G, M )-familles (cP 0 ,1 )P 0 ∈ P (M ) et (cP 0 ,2 )P 0 ∈ P (M ) comme dans la preuve précédente et il suffit de prouver que cM,1 = cM,2 . Or il existe une (G, M )-famille (dP 0 )P 0 ∈ P (M ) , construite à l’aide des rapports rP 0 |P,1 (τ, λ)rP 0 |P,2 (τ, λ)−1 , telle que cP 0 ,1 = cP 0 ,2 dP 0 pour tout P 0 . On a alors la formule de descente X 0 Q00 0 00 Q cM,1 = dG M (L , L )cM,2 dM , L0 ,L00 ∈ L (M ) 0

0

Q cf. [4] corollaire 7.4. Le terme Q0 est un élément de P (M 0 ) et cQ M,2 = JM (τ, f ), ce terme étant calculé à l’aide de la seconde famille de facteurs. Si L0 6= G, il est nul d’après le (i) du lemme. Dans la somme ci-dessus, il ne reste que la contribution du 0 Q00 couple (L0 , L00 ) = (G, M ). Pour ce couple, cQ M,2 dM = cM,2 , d’où l’égalité cherchée.

2.3. Induction de quasi-caractères. — Pour ce paragraphe, oublions les choix de mesures de Haar et de sous-groupe compact spécial que l’on a effectués. Soit M un Levi de G. On munit G(F ) et M (F ) de mesures de Haar. Soit ∆M une distribution sur M (F ) invariante par conjugaison. On sait définir la distribution induite M ∆ = IndG M (∆ ), qui est une distribution invariante sur G(F ). Rappelons sa définition. Fixons un élément P = M U ∈ P (M ) et un sous-groupe compact spécial K de G(F ), en bonne position relativement à M . Munissons K et U (F ) de mesures

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

187

de Haar, compatibles au sens habituel avec les mesures sur M (F ) et G(F ). Pour f ∈ Cc∞ (G(F )), on définit fP ∈ Cc∞ (M (F )) par Z Z fP (m) = δP (m)1/2 f (k −1 muk)du dk. K

U (F )

On pose ∆(f ) = ∆M (fP ). Cela ne dépend pas des choix de P et K. Soit maintenant θM une fonction définie presque partout sur M (F ), localement intégrable et invariante par conjugaison. Soit ∆M la distribution associée, c’est-à-dire que Z ∆M (ϕ) = ϕ(m)θM (m)dm. M (F )

A l’aide de la formule d’intégration de Weyl, on vérifie que ∆ est elle-aussi associée à une fonction θ sur G(F ), localement intégrable et invariante par conjugaison. Pour tout x ∈ G(F ), fixons un ensemble X M (x) de représentants des classes de conjugaison par M (F ) dans l’ensemble des éléments de M (F ) qui sont conjugués à x par un élément de G(F ). Pour x ∈ Greg (F ), on a l’égalité X (1) θ(x) = DG (x)−1/2 DM (x0 )1/2 θM (x0 ). x0 ∈ X M (x)

Les fonctions DG et DM sont les modules des fonctions discriminants de HarishChandra habituelles. Cette formule montre que θ est indépendante des choix de meM sures sur G(F ) et M (F ). On note IndG M (θ ) = θ. Rappelons que l’on a défini en [17] 4.1 la notion de quasi-caractère de G(F ). Soit θ une fonction définie presque partout sur G(F ) et invariante par conjugaison. On dit que c’est un quasi-caractère si et seulement si, pour tout élément semi-simple x de G(F ), il existe un bon voisinage ω de 0 dans gx (F ) et, pour tout O ∈ Nil(gx ), il existe cθ, O (x) ∈ C de sorte que l’on ait l’égalité X (2) θ(xexp(X)) = cθ, O (x)ˆ ( O, X) O∈Nil(gx )

presque partout pour X ∈ ω. Donnons quelques explications. On note Gx la composante neutre du centralisateur ZG (x) de x dans G et, comme toujours gx son algèbre de Lie. On renvoie à [17] 3.1 pour la notion de bon voisinage. On note Nil(gx ) l’ensemble des orbites nilpotentes dans gx . La fonction X 7→ ˆ( O, X) est la fonction associée à la distribution transformée de Fourier de l’intégrale orbitale J O associée à O, normalisée comme en [17] 1.2. M M D’autre part, soit O ∈ Nil(m). On sait définir « l’orbite induite » de O . Plus exactement, c’est la réunion d’un certain nombre d’éléments de Nil(g). Fixons P = M U ∈ P (M ). Une orbite O ∈ Nil(g) est incluse dans cette orbite induite si et seuleM M ment si l’intersection O ∩ ( O + u(F )) contient un ouvert non vide de O + u(F ). On M M M pose [ O : O ] = 1 si O est incluse dans l’orbite induite de O , [ O : O ] = 0 sinon. M Remarquons que si [ O : O ] = 1 et si l’une des deux orbites est régulière, l’autre l’est aussi.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

188

J.-L. WALDSPURGER

Pour un élément semi-simple x ∈ G(F ), on a fixé ci-dessus un ensemble X M (x). Pour tout élément x0 de cet ensemble, on note Γx0 l’ensemble des g ∈ G(F ) tels que gxg −1 = x0 . C’est un espace principal homogène pour l’action à droite de ZG (x)(F ). Pour tout g ∈ Γx0 , la conjugaison par g envoie Nil(gx ) sur Nil(gx0 ). On note O 7→ g O cette application. M Lemme. — Soient θM un quasi-caractère de M (F ) et θ = IndG M (θ ). Alors (i) θ est un quasi-caractère de G(F ) ; (ii) soient x un élément semi-simple de G(F ) et O ∈ Nil(gx ) une orbite régulière ; on a l’égalité X X X cθ, O (x) = DG (x)−1/2 DM (x0 )1/2 x0 ∈ X M (x) g∈Γx0 /Gx (F )

O0 ∈Nil(mx0 ) 0

[ZM (x0 )(F ) : Mx0 (F )]−1 [g O : O ]cθM , O0 (x0 ). Démonstration. — Soit x un élément semi-simple de G(F ). Considérons un bon voisinage ω de 0 dans gx . Pour x0 ∈ X M (x), posons ωx0 = gωg −1 , où g est un élément quelconque de Γx0 . C’est un bon voisinage de 0 dans gx0 . En prenant ω assez petit, on peut supposer que ωxM0 = ωx0 ∩ mx0 (F ) est un bon voisinage de 0 dans mx0 (F ) et que le quasi-caractère θM admet un développement de la forme (2) dans x0 exp(ωxM0 ). On définit θxM0 ,ωM : c’est la fonction sur mx0 (F ), à support dans ωxM0 et telle que x0

θxM0 ,ωM (Y ) = θM (x0 exp(Y )) pour tout Y ∈ ωxM0 . C’est un quasi-caractère de mx0 (F ). x0

En adaptant les définitions ci-dessus aux algèbres de Lie, on définit la fonction locaG lement intégrable φx0 ,ωx0 = IndMxx00 (θxM0 ,ωM ) sur gx0 (F ). On va prouver x0

(3) pour tout X ∈ ω ∩ gx,reg (F ), on a l’égalité X X θ(xexp(X)) = DG (x)−1/2 DM (x0 )1/2 x0 ∈ X M (x) g∈Γx0 /Gx (F )

[ZM (x0 )(F ) : Mx0 (F )]−1 φx0 ,ωx0 (gXg −1 ). Fixons X. Pour tout x0 ∈ X M (x) et tout g ∈ Γx0 , fixons un ensemble X Mx0 (gXg −1 ) de représentants des classes de conjugaison par Mx0 (F ) dans l’ensemble des éléments de Mx0 (F ) qui sont conjugués à gXg −1 par un élément de Gx0 (F ). Il est inclus dans ωxM0 et on peut supposer qu’il ne dépend que de l’image de g dans Γx0 /Gx (F ). En appliquant la formule (1) aux fonctions φx0 ,ωx0 , le membre de droite de la formule (3) est égal à X X DG (x)−1/2 DM (x0 )1/2 [ZM (x0 )(F ) : Mx0 (F )]−1 x0 ∈ X M (x) g∈Γx0 /Gx (F )

X Y∈X

DGx0 (Y )−1/2 DMx0 (Y )1/2 θM (x0 exp(Y )).

M 0 x (gXg −1 )

0

Soient x , g et Y apparaissant dans cette somme. On a DG (x)DGx0 (Y ) = DG (x0 )DGx0 (Y ) = DG (x0 exp(Y )),

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

189

DM (x0 )DMx0 (Y ) = DM (x0 exp(Y )). D’autre part, x0 exp(Y ) est un élément de M (F ) conjugué à xexp(X) par un élément de G(F ). Il est donc conjugué par un élément de M (F ) à un unique élément de X M (xexp(X)). Notons cet élément y(x0 , g, Y ). On a DG (x0 exp(Y ))−1/2 DM (x0 exp(Y ))1/2 θM (x0 exp(Y )) = DG (xexp(X))−1/2 DM (y(x0 , g, Y ))1/2 θM (y(x0 , g, Y )). La formule ci-dessus s’écrit donc X c(y)DG (xexp(X))−1/2 DM (y)1/2 θM (y), y∈ X M (xexp(X))

où c(y) =

X

[ZM (x0 )(F ) : Mx0 (F )]−1 .

x0 ,g,Y ;y(x0 ,g,Y )=y

En comparant avec la formule (1), on voit que, pour démontrer (3), il suffit de prouver que (4) c(y) = 1 pour tout y ∈ X M (xexp(X)). Soit y ∈ X M (xexp(X)). Fixons γ ∈ G(F ) tel que γxexp(X)γ −1 = y. Puisque y ∈ M (F ) ∩ Greg (F ), le centralisateur Gy de y est contenu dans M . Mais γxγ −1 appartient à Gy (F ). Il appartient donc à M (F ). Il existe donc x0 ∈ X M (x) et m ∈ M (F ) tel que γxγ −1 = mx0 m−1 . Posons g = m−1 γ. Alors g ∈ Γx0 et γ = mg. On a y = mx0 exp(gXg −1 )m−1 . Puisque y ∈ M (F ), on a aussi x0 exp(gXg −1 ) ∈ M (F ), donc gXg −1 ∈ mx0 (F ). Alors gXg −1 est conjugué par un élément de Mx0 (F ) à un élément Y ∈ X Mx0 (gXg −1 ) et y est conjugué par un élément de M (F ) à x0 exp(Y ). Pour ces choix de x0 , g, Y , on a y = y(x0 , g, Y ). Soit (x01 , g1 , Y1 ) un autre triplet, supposons y = y(x01 , g1 , Y1 ). Quitte à multiplier g à gauche par un élément de Gx0 (F ) (ce qui revient à le multiplier à droite par un élément de Gx (F )), on peut supposer Y = gXg −1 . De même, on peut supposer Y1 = g1 Xg1−1 . Soit µ ∈ M (F ) tel que µx0 exp(Y )µ−1 = x01 exp(Y1 ). Alors µgxexp(X)g −1 µ−1 = g1 xexp(X)g1−1 . En posant h = g1−1 µg, on a hxexp(X)h−1 = xexp(X). Puisque xexp(X) est régulier, cela entraîne h ∈ Gxexp(X) (F ). Cet ensemble est contenu dans Gx (F ) d’après les propriétés des bons voisinages. Donc h ∈ Gx (F ). On a alors µx0 µ−1 = µgxg −1 µ−1 = g1 hxh−1 g1−1 = x01 . Par définition de l’ensemble X M (x), cela entraîne x01 = x0 . A fortiori, la constante [ZM (x01 )(F ) : Mx01 (F )]−1 qui intervient dans la définition de c(y) est égale à [ZM (x0 )(F ) : Mx0 (F )]−1 et ne dépend pas du triplet. Puisque x01 = x0 , la relation ci-dessus entraîne µ ∈ ZM (x0 )(F ). En revenant à la définition de µ, on a µY µ−1 = Y1 . Le couple (g1 Gx (F ), Y1 ) appartient donc à l’orbite de (gGx (F ), Y ) pour l’action de ZM (x0 )(F ) ainsi définie : l’action de µ ∈ ZM (x0 )(F ) envoie (gGx (F ), Y ) sur le couple (g1 Gx (F ), Y1 ) tel que g1 Gx (F ) = µgGx (F ) et que Y1 soit l’unique élément de X Mx0 (g1 Xg1−1 ) conjugué à µY µ−1 par un élément de Mx0 (F ). Inversement, on vérifie que tout couple ainsi obtenu convient. L’action de ZM (x0 )(F ) que l’on

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

190

J.-L. WALDSPURGER

vient de définir se quotiente en une action de ZM (x0 )(F )/Mx0 (F ). Remarquons que ZM (x0 ) ∩ Gx0 = Mx0 car ces deux ensembles sont égaux au commutant de AM dans Gx0 . Il en résulte que l’action de ZM (x0 )(F )/Mx0 (F ) est libre : son action sur la première composante l’est. Le nombre de triplets est donc égal au nombre d’éléments de ce groupe, ce qui entraîne (4) et achève la preuve de (3). La formule (3) nous ramène au problème suivant. Soit maintenant θM un quasicaractère de m(F ), dont on écrit le développement à l’origine X M θM (Y ) = cθM , OM ˆM ( O , Y ). OM ∈Nil(m) M Soit θ = IndG M (θ ). On doit prouver que θ possède un développement à l’origine de la forme X (5) θ(X) = cθ, O ˆG ( O, X),

O∈Nil(g)

et prouver que, pour O régulière, on a l’égalité X (6) cθ, O =

M

[ O : O ].

OM ∈Nil(m)

Comme on l’a expliqué, l’induction ne dépend pas des mesures de Haar, si on la considère comme une application portant sur des fonctions localement intégrables. On peut donc supposer que les mesures sont normalisées comme en [17] 1.2. L’analogue pour les algèbres de Lie de l’application f 7→ fP « commute » à la transformation de M Fourier. On en déduit que l’induite d’une fonction Y 7→ ˆM ( O , Y ) est la fonction associée à la transformée de Fourier de la distribution induite de l’intégrale orbitale JM . Il est bien connu que cette distribution induite est combinaison linéaire des OM intégrales orbitales J G O pour des éléments O ∈ Nil(g) inclus dans l’orbite induite de

OM . En tout cas, l’induite d’une fonction Y 7→ ˆM ( OM , Y ) est combinaison linéaire de fonctions X 7→ ˆG ( O, X), ce qui prouve l’existence du développement (5). Pour M prouver (6), on voit qu’il suffit de prouver que, pour O régulière, la distribution induite de J M est égale à OM X

M

[ O : O ]J G O.

O∈Nil(g)

On peut supposer M et G quasi-déployés, sinon il n’y a pas d’orbites nilpotentes régulières et la question est vide. Toute orbite nilpotente régulière O de g(F ) apparaît M dans l’orbite induite d’une unique orbite nilpotente régulière O de m(F ). En effet, fixons P ∈ P (M ) et un sous-groupe de Borel B de G tel que B ⊂ P . Soient Y, Y 0 ∈ m(F ), N, N 0 ∈ u(F ), supposons que Y + N et Y 0 + N 0 appartiennent à O. Quitte à effectuer des conjugaisons par des éléments de M (F ), on peut supposer Y, Y 0 ∈ b(F ) ∩ m(F ). Soit g ∈ G(F ) tel que g(Y + N )g −1 = Y 0 + N 0 . L’élément Y 0 + N 0 appartient aux deux sous-algèbres de Borel b et gbg −1 . Mais Y 0 + N 0 est régulier donc n’appartient qu’à une seule telle sous-algèbre. Donc gbg −1 = b et g appartient à B(F ). En écrivant g = mu, avec m ∈ M (F ) et u ∈ U (F ), on a alors Y 0 = mY m−1 ,

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

191

c’est-à-dire que Y et Y 0 sont dans la même orbite. Cette unicité nous permet de transformer notre problème en le suivant : prouver que la distribution induite de X JM OM OM régulière

est égale à X

JG O.

O régulière

Introduisons un sous-groupe de Borel B comme ci-dessus et un sous-tore maximal T ⊂ B∩M . Fixons un élément X ∈ t(F )∩greg (F ). En utilisant un résultat de Shelstad, on a prouvé en [17] lemme 11.4 que la première distribution ci-dessus était la limite M simple des distributions f 7→ JM (zX, f ) sur M (F ) quand z ∈ F × tend vers 0 (dans [17], notre groupe était un groupe spécial orthogonal, mais la démonstration de cette propriété n’utilisait pas cette particularité). De même, la seconde distribution est la limite simple des distributions f 7→ JG (zX, f ) sur G(F ). Mais il résulte des définitions M (zX, f ). La que la distribution f 7→ JG (zX, f ) est l’induite de la distribution f 7→ JM conclusion s’ensuit.

2.4. Intégrales orbitales pondérées invariantes. — Soient f, f 0 ∈ Cc∞ (G(F )). Nous dirons que f et f 0 sont équivalentes si et seulement si D(f ) = D(f 0 ) pour toute distribution D sur G(F ) invariante par conjugaison. Comme on le sait, cette condition est équivalente à l’une ou l’autre des deux conditions suivantes (1) JG (x, f ) = JG (x, f 0 ) pour tout x ∈ G(F ) ; (2) θπ (f ) = θπ (f 0 ) pour toute représentation π ∈ Temp(G). Soient M ∈ L (Mmin ), x ∈ M (F ) ∩ Greg (F ) et f ∈ Cc∞ (G(F )). Arthur a défini l’intégrale orbitale pondérée JM (x, f ). On a rappelé la définition en [17] 2.3. Il a aussi défini l’intégrale pondérée invariante IM (x, f ). Conformément à la pratique d’Arthur, G G nous noterons aussi ces termes JM (x, f ) et IM (x, f ) s’il convient de préciser quel est le groupe ambiant. Rappelons la définition de IM (x, f ). Pour Z ∈ A G,F , notons 1HG =Z la fonction caractéristique de l’ensemble des x ∈ G(F ) tels que HG (x) = Z. Notons H ac (G(F )) l’ensemble des fonctions f : G(F ) → C qui vérifient les deux conditions suivantes (3) f est biinvariante par un sous-groupe ouvert compact de G(F ) ; (4) pour tout Z ∈ A G,F , la fonction f 1HG =Z appartient à Cc∞ (G(F )). Remarquons que plusieurs définitions posées pour les fonctions appartenant à ∞ Cc (G(F )) se généralisent aux éléments de H ac (G(F )). Par exemple les intégrales orbitales pondérées (on pose JM (x, f ) = JM (x, f 1HG =HG (x) )) ou la notion d’équivalence introduite ci-dessus. Soient L ∈ L (Mmin ) et f ∈ Cc∞ (G(F )). Arthur montre qu’il existe une fonction φL (f ) ∈ H ac (L(F )) telle que, pour toute représentation π ∈ Temp(L) et tout Z ∈

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

192

J.-L. WALDSPURGER

A L,F , on ait les égalités Z

Z

(5) i A∗ L,F

JL (πλ , f )exp(−λ(Z))dλ =

i A∗ L,F

θπλ (φL (f )1HL =Z )exp(−λ(Z))dλ

= mes(i A ∗L,F )θπ (φL (f )1HL =Z ). Cela résulte de [5] théorème 12.1 et [4] proposition 1.1. La fonction φL (f ) est bien définie à équivalence près. On définit IM (x, f ) par récurrence sur aM − aG par la formule X L IM (x, f ) = JM (x, f ) − IM (x, φL (f )1HL =HL (x) ). L∈ L (M ),L6=G

Bien sûr, IM (x, f ) ne dépend que de la classe de conjugaison par M (F ) de x. La distribution f 7→ IM (x, f ) est invariante par conjugaison par G(F ) et ne dépend pas du choix du groupe K. La propriété suivante en résulte, par simple transport de structure. Soit g ∈ G(F ) tel que gM g −1 ∈ L (Mmin ). Alors on a l’égalité IgM g−1 (gxg −1 , f ) = IM (x, f ). Pour f ∈ Cc∞ (G(F )), on définit une fonction θf sur Greg (F ) de la façon suivante. Soit x ∈ Greg (F ). Notons M (x) le commutant de AGx dans G. C’est un Levi de G et c’est le plus petit Levi contenant x. Choisissons g ∈ G(F ) tel que gM (x)g −1 ∈ L (Mmin ). On pose θf (x) = (−1)aM (x) −aG DG (x)−1/2 IgM (x)g−1 (gxg −1 , f ). Cela ne dépend pas du choix de g. La fonction θf est invariante par conjugaison et localement constante sur Greg (F ). Remarquons que θf = θf 0 si f et f 0 sont équivalentes, puisque les distributions f 7→ IM (x, f ) sont invariantes par conjugaison. 2.5. Fonctions cuspidales et quasi-caractères. — Soit f ∈ Cc∞ (G(F )). On dit que f est cuspidale si et seulement si, pour tout groupe de Levi M ( G et pour tout x ∈ Greg (F ) ∩ M (F ), on a JG (x, f ) = 0. Cette condition est équivalente à ce que θπ (f ) = 0 pour toute représentation π de G(F ) qui est tempérée et proprement induite. Une fonction très cuspidale est cuspidale. Lemme. — Soit f ∈ Cc∞ (G(F )), supposons f cuspidale. Alors θf est un quasicaractère de G(F ). Démonstration. — Arthur définit un ensemble de représentations virtuelles Tell (G). Tout élément π de Tell (G) est une combinaison linéaire à coefficients complexes de représentations elliptiques. Par linéarité, on définit la contragrédiente π ˇ , le caractère θπ et, pour λ ∈ i A ∗G , la représentation virtuelle πλ qui appartient aussi à Tell (G). On note {Tell (G)} l’ensemble des orbites dans Tell (G) pour l’action λ 7→ πλ . Si K 0 est un sous-groupe ouvert compact de G(F ), il n’y a qu’un nombre fini d’orbites O ∈ Tell (G) pour lesquelles il existe π ∈ O et une fonction f 0 ∈ Cc∞ (G(F )), biinvariante par K 0 , de sorte que θπ (f 0 ) 6= 0. Pour toute orbite O, on fixe un point-base π ∈ O. En [7] théorème 5.1, Arthur associe à O un coefficient que nous notons c( O) dont nous ne rappelons pas la définition (on explicitera toutefois ce coefficient pour les groupes

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

193

spéciaux orthogonaux en 7.3). On a c( O) > 0. Puis Arthur démontre que, pour toute fonction cuspidale f ∈ Cc∞ (G(F )), pour tout M ∈ L (Mmin ) et pour tout élément y ∈ M (F ) ∩ Greg (F ) qui est elliptique dans M (F ), on a l’égalité Z X c( O) θπλ (y)θ(πλ )ˇ(f )dλ. DG (y)−1/2 (−1)aM −aG IM (y, f ) = O∈{Tell (G)}

i A∗ G,F

Elle équivaut à X

DG (y)−1/2 (−1)aM −aG IM (y, f ) =

c0 ( O)θπ (y)θπˇ (f 1HG =HG (y) ),

O∈{Tell (G)} ∗ G,F )c(

où c0 ( O) = mes(i A O). Soit x un élément semi-simple de G(F ). Pour y dans un certain voisinage de x, on a HG (y) = HG (x). Nos définitions et la formule ci-dessus entraînent que, pour y ∈ Greg (F ) dans ce voisinage, on a l’égalité X (1) θf (y) = c0 ( O)θπ (y)θπˇ (f 1HG =HG (x) ). O∈{Tell (G)}

Comme on l’a dit, la somme est en fait finie. Donc θf coïncide dans ce voisinage de x avec une combinaison linéaire finie de caractères de représentations admissibles irréductibles. D’après Harish-Chandra ([11] théorème 16.2), tout tel caractère est un quasi-caractère. La notion de quasi-caractère étant de nature locale, la conclusion s’ensuit. On appelle θf le quasi-caractère associé à f . La notion de cuspidalité se généralise aux éléments de H ac (G(F )) : f ∈ H ac (G(F )) est cuspidale si et seulement si f 1HG =Z l’est pour tout Z ∈ A G,F . La définition de P θf aussi : θf = Z∈ A G,F θf 1HG =Z , cette somme étant localement finie. 2.6. Les deux quasi-caractères associés à une fonction très cuspidale. — Soit f ∈ Cc∞ (G(F )) une fonction très cuspidale. On vient de lui associer un quasicaractère θf sur G(F ). Dans [17] 5.3 et 5.9, on lui a associé un autre quasi-caractère, que l’on avait noté θf dans cette référence, et que nous noterons désormais θfJ . En fait, cette définition dépend des choix de mesures. Nous modifions la définition de [17] 5.6 en utilisant nos présentes mesures plutôt que celles de cette référence. Soit x ∈ Greg (F ). Notons M (x) le commutant de AGx dans G. Par définition G θfJ (x) = (−1)aM (x) −aG DG (x)−1/2 JM (x) (x, f ),

l’intégrale pondérée étant calculée à l’aide d’un sous-groupe compact spécial de G(F ) en bonne position relativement à M (x). Remarque. — L’exposant J de la notation θfJ rappelle que cette fonction est construite à l’aide des intégrales orbitales pondérées non invariantes. La définition n’a de sens que parce que ces intégrales, bien que n’étant pas invariantes, vérifient néanmoins une propriété d’invariance par conjugaison, quand on les restreint aux fonctions très cuspidales.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

194

J.-L. WALDSPURGER

Il convient de comparer θfJ et θf . Lemme. — Soit f ∈ Cc∞ (G(F )) une fonction très cuspidale. Alors (i) pour tout L ∈ L (Mmin ), la fonction φL (f ) est cuspidale ; (ii) on a l’égalité X L θfJ = |W L ||W G |−1 (−1)aL −aG IndG L (θφL (f ) ). L∈ L (Mmin )

Démonstration. — Pour prouver (i), on doit montrer que, pour tout Z ∈ A L,F et toute représentation tempérée proprement induite π de L(F ), on a θπ (φL (f )1HL =Z ) = 0. Cela résulte de l’égalité 2.4(5) et du lemme 2.2(ii) qui affirme que JLG (πλ , f ) = 0 pour tout λ ∈ i A ∗L . Soit ϕ ∈ Cc∞ (G(F )). On peut écrire la formule d’intégration de Weyl sous la forme Z X (1) θfJ (g)ϕ(g)dg = |W M ||W G |−1 G(F )

X

M ∈ L (Mmin )

Z

|W (M, T )|−1

θfJ (t)JG (t, ϕ)DG (t)1/2 dt,

T (F )

T ∈ T ell (M )

avec les notations d’Arthur que l’on a rappelées en [17] 2.4. Pour tout L ∈ L (Mmin ), fixons QL ∈ P (L). On a de même Z Z L IndG (θ )(g)ϕ(g)dg = θφLL (f ) (l)ϕQL (l)dl L φL (f ) G(F )

=

X

L(F )

X

L −1

M

|W ||W |

L

|W (M, T )|

Z T (F )

T ∈ T ell (M )

M ∈ L (Mmin )

−1

θφLL (f ) (t)JL (t, ϕQL )DL (t)1/2 dt.

Le terme JL (t, ϕQL ) intervenant ci-dessus est égal à JG (t, ϕ). Notons γf la fonction figurant dans le membre de droite du (ii) de l’énoncé. En sommant les égalités cidessus, on obtient Z (2)

γf (g)ϕ(g)dg G(F )

=

X

|W

M

X

G −1

||W |

M ∈ L (Mmin )

T ∈ T ell (M )

|W (M, T )|

−1

Z γM,T (t)JG (t, ϕ)dt, T (F )

où on a posé γM,T (t) =

X

(−1)aL −aG θφLL (f ) (t)DL (t)1/2 .

L∈ L (M )

Soient M ∈ L (Mmin ), T ∈ T ell (M ) et t ∈ T (F ) ∩ Greg (F ). Pour L ∈ L (M ), appliquons la définition de θφLL (f ) (t) donnée en 2.4. Puisque T est elliptique dans M , le Levi M (t) est égal à M (que le groupe ambiant soit G ou L). Donc L θφLL (f ) (t) = (−1)aM −aL DL (t)−1/2 IM (t, φL (f )).

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

195

On peut aussi bien remplacer φL (f ) par φL (f )1HL =HL (t) . Alors X L IM (t, φL (f )1HL =HL (t) ) γM,T (t) = (−1)aM −aG L∈ L (M ) G = (−1)aM −aG JM (t, f ).

En se reportant à la définition de [17] 5.3 rappelée ci-dessus, on obtient γM,T (t) = DG (t)1/2 θfJ (t). On conclut en comparant les égalités (1) et (2). 2.7. Fonctions cuspidales et fonctions très cuspidales Lemme. — Soit f ∈ Cc∞ (G(F )) une fonction cuspidale. Alors il existe une fonction très cuspidale f 0 ∈ Cc∞ (G(F )) qui est équivalente à f . Démonstration. — Par un procédé de partition de l’unité tel que celui de la preuve de [17] proposition 6.4, il suffit de prouver l’assertion suivante (1) soit x ∈ G(F ) un élément semi-simple ; alors il existe un G-domaine Ω dans G(F ) et une fonction très cuspidale f 0 ∈ Cc∞ (G(F )) tels que x ∈ Ω et JG (y, f 0 ) = JG (y, f ) pour tout y ∈ Ω ∩ Greg (F ). Supposons AGx 6= AG . Il existe un G-domaine Ω contenant x tel que, pour y ∈ Ω ∩ Greg (F ), on ait AGy 6= AG , autrement dit y n’est pas elliptique dans G(F ). Alors JG (y, f ) = 0 et il suffit de prendre f 0 = 0 pour vérifier l’assertion. Supposons maintenant AGx = AG . Fixons un bon voisinage ω de 0 dans gx (F ). Le quasi-caractère θf se descend en un quasi-caractère θf,x,ω sur gx (F ), cf. [17] 4.3, qui est évidemment à support compact modulo conjugaison et invariant par l’action de ZG (x)(F ). En combinant la proposition 6.4 et le lemme 6.2 de [17], on voit qu’il existe une fonction très cuspidale f 0 ∈ Cc∞ (G(F )) telle que θfJ0 ,x,ω = θf,x,ω . Posons Ω = {g −1 xexp(X)g; X ∈ ω, g ∈ G(F )} et soit y ∈ Ω ∩ Greg (F ). Si y n’est pas elliptique dans G(F ), on a JG (y, f 0 ) = 0 = JG (y, f ). Supposons y elliptique, écrivons y = g −1 xexp(X)g avec g ∈ G(F ) et X ∈ ω. D’après les définitions, on a JG (y, f 0 ) = θfJ0 ,x,ω (X) et JG (y, f ) = θf,x,ω (X). D’où l’égalité JG (y, f 0 ) = JG (y, f ). Cela prouve (1) et le lemme.

3. Majorations pour le groupe linéaire GLk 3.1. Le groupe linéaire. — Soient k ≥ 1 un entier, V un espace vectoriel sur F de dimension k et (vi )i=1,...,k une base de V . On note simplement GLk le groupe (algébrique) des automorphismes linéaires de V . Pour g ∈ GLk (F ), on note (gi,j )i,j=1,...,k sa matrice dans la base fixée. On note Bk le sous-groupe de Borel triangulaire supérieur de GLk , Uk son radical unipotent et Ak le sous-tore diagonal. Pour a ∈ Ak (F ), on note simplement ai = ai,i son coefficient diagonal, pour i = 1, . . . , k. On note Kk le sous-groupe compact spécial de GLk (F ) formé des éléments à coefficients entiers et de déterminant de valuation nulle.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

196

J.-L. WALDSPURGER

La théorie du R-groupe est « triviale » pour le groupe linéaire. C’est-à-dire que les représentations tempérées irréductibles et elliptiques de GLk (F ) sont de la série discrète. Ces notations seront utilisées pour divers espaces, parfois sans que l’on précise leur base. Ou bien le choix de cette base sera implicite, ou bien il n’aura pas d’importance. Dans la suite de cette section, on fixe un entier k ≥ 1, on pose G = GLk , et on utilise les notations ci-dessus dont on supprime l’indice k. 3.2. Une majoration. — Pour tout g ∈ G(F ), on note g = aB (g)uB (g)kB (g) une décomposition de g telle que aB (g) ∈ A(F ), uB (g) ∈ U (F ), kB (g) ∈ K. Pour un entier c ≥ 1, on note U (F )c le sous-groupe des éléments u ∈ U (F ) tels que valF (ui,i+1 ) ≥ −c pour tout i = 1, . . . , k − 1. Soient D ∈ R et g ∈ G(F ), posons Z I(c, D, g) = ΞG (ug)σ(ug)D du. U (F )c

Proposition. — Cette intégrale est convergente. Pour tout D, il existe un réel R tel que I(c, D, g)  cR σ(g)R δB (aB (g))1/2 pour tout c ≥ 1 et tout g ∈ G(F ). La preuve sera donnée en 3.4. 3.3. Un lemme auxiliaire. — Supposons k ≥ 2, notons P = M UP le sous-groupe parabolique des éléments de G qui conservent la droite F v1 . Soient c ≥ 1 un entier, D un réel et m ∈ M (F ). Posons UP (F )c = UP (F ) ∩ U (F )c et Z IP (c, D, m) = ΞG (um)σ(um)D du. UP (F )c

Lemme. — Cette intégrale est convergente. Pour tout D, il existe un réel R tel que IP (c, D, m)  cR σ(m)R δP (m)1/2 ΞM (m) pour tout c ≥ 1 et tout m ∈ M (F ). Démonstration. — Le réel D est fixé. Soit b ≥ 0 un réel. On a introduit en 1.1 la fonction 1σ≥b . Notons U\ le sous-groupe des éléments u ∈ UP tels que u1,2 = 0. Posons Z I\ (b, D) = 1σ≥b (u)δB¯ (aB¯ (u))1/2 σ(u)D du. U\ (F )

Montrons que (1) cette intégrale est convergente et il existe ε > 0 tel que I\ (b, D)  exp(−εb) pour tout b ≥ 0. Considérons le sous-espace V 00 de V engendré par les éléments v1 , v3 , . . . , vk . Soit 00 G = GLk−1 son groupe d’automorphismes, qui s’identifie à un sous-groupe de G. Le

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

197

groupe B 00 = G00 ∩ B est le sous-groupe de Borel standard de G00 , P 00 = G00 ∩ P est un sous-groupe parabolique de G00 et U\ n’est autre que le radical unipotent de P 00 . D’après [16] lemme II.4.2, il existe un entier d ≥ 0 tel que l’intégrale Z (2) δB¯ 00 (aB¯ 00 (u))1/2 σ(u)−d du U\ (F )

soit convergente. Soit u ∈ U\ (F ). On peut supposer aB¯ (u) = aB¯ 00 (u). Notons a1 , . . . , ak les coefficients diagonaux de cet élément. On peut supposer a2 = 1 et Q i=1,...,k ai = 1. On a Y i−(k+1)/2 δB¯ (aB¯ (u))1/2 = |ai |F , i=1,...,k

tandis que 1−k/2

δB¯ 00 (aB¯ 00 (u))1/2 = |a1 |F

Y

i−1−k/2

|ai |F

.

i=3,...,k

D’où

Ñ −1/2

δB¯ (aB¯ (u))1/2 = |a1 |F

é 1/2

Y

|ai |F

1/2 . δB¯ 00 (aB¯ 00 (u))1/2 = |a1 |−1 ¯ 00 (u)) ¯ 00 (aB F δB

i=3,...,k

L’égalité u = aB¯ 00 (u)uB¯ 00 (u)kB¯ 00 (u) et un calcul matriciel entraînent que valF (a1 ) = inf {valF (u1,j ); j = 1, 3, . . . , k}, d’où −valF (a1 )  σ(u). Il existe donc ε > 0 tel que |a1 |−1  exp(−2εσ(u)). Si F −1 1σ≥b (u) = 1, on transforme cette relation en |a1 |F  exp(−εb)exp(−εσ(u)). On obtient Z δB¯ 00 (aB¯ 00 (u))1/2 exp(−εσ(u))du.

I\ (b, D)  exp(−εb) U\ (F )

La dernière intégrale est convergente d’après la convergence de (2). D’où (1). Introduisons un réel b ≥ 0, que nous préciserons par la suite. On peut écrire (3)

IP (c, D, m) = I 0. D’autre part, d’après [16] lemmes II.1.1 et II.3.2, il existe un réel D00 tel que l’on ait une majoration 00

ΞG (g)  δB¯ (aB¯ (g))1/2 σ(g)D . Ces relations entraînent la majoration

I≥b (c, D, m, v)  exp(ασ(vm))σ(vm)

D+D00

Z

00

δB¯ (aB¯ (u))1/2 σ(u)D+D 1σ≥c1 b−c−σ(m) (u)du

U\ (F )

 exp(c2 c)exp(c2 σ(m))I\ (c1 b − c − σ(m), D + D00 ) pour un c2 > 0 convenable. Cette expression ne dépend plus de v. Le terme I≥b (c, D, m) est majoré par la même expression, multipliée par mes(U1,2 (F )c ). Cette mesure est majorée par exp(c3 c) pour un c3 > 0 convenable. D’où I≥b (c, D, m)  exp(c4 c)exp(c2 σ(m))I\ (c1 b − c − σ(m), D + D00 ),

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

199

où c4 = c2 + c3 . Il existe aussi c5 > 0 tel que l’on ait la minoration exp(−c5 σ(m))  δP (m)1/2 ΞM (m). D’où I≥b (c, D, m)  δP (m)1/2 ΞM (m)exp(c4 c)exp(c6 σ(m))I\ (c1 b − c − σ(m), D + D00 ), où c6 = c2 + c5 . Utilisons la relation (1). On voit qu’il existe c7 > 0 tel que, pour b = c7 (c + σ(m)), le produit des trois derniers termes ci-dessus est borné. Choisissons b ainsi. Alors I≥b (c, D, m)  δP (m)1/2 ΞM (m). Cette majoration et les relations (3) et (4) entraînent celle de l’énoncé. 3.4. Preuve de la proposition 3.2. — On démontre la proposition par récurrence sur k. Le cas k = 1 est évident. Supposons k ≥ 2. Remarquons tout d’abord que l’on peut se limiter à démontrer la majoration de l’énoncé pour g = a ∈ A(F ). En effet, pour g quelconque, on écrit g = vak, avec v ∈ U (F ), a ∈ A(F ), k ∈ K. Effectuons le changement de variable u 7→ uv −1 . Le nouveau domaine d’intégration est U (F )c v. Mais il existe c1 > 0 tel que cet ensemble soit inclus dans U (F )c+c1 σ(g) . Alors I(c, D, g) ≤ I(c + c1 σ(g), D, a). Si le deuxième terme vérifie une majoration comme dans l’énoncé, le premier terme aussi. Supposons donc g = a ∈ A(F ). Avec les notations du paragraphe précédent, on a Z Z I(c, D, a) = ΞG (uva)σ(uva)D du dv M (F )∩U (F )c

UP (F )c

Z =

IP (c, D, va)dv. M (F )∩U (F )c

En appliquant le lemme 3.3, on a Z I(c, D, a)  cR

δP (va)1/2 ΞM (va)σ(va)R dv.

M (F )∩U (F )c

Ecrivons M = GL1 × G0 , où G0 = GLk−1 et affectons d’un 0 les objets relatifs à G0 . Ecrivons aussi a = (a1 , a0 ), avec a1 ∈ F × et a0 ∈ A0 (F ). On a δP (va)1/2 = 0 δB (a)1/2 δB 0 (a0 )−1/2 , ΞM (va) = ΞG (va0 ) et σ(va)R  σ(a)R σ(va0 )R . On obtient I(c, D, a)  cR σ(a)R δB (a)1/2 δB 0 (a0 )−1/2 I 0 (c, R, a0 ). En utilisant la majoration du dernier terme fournie par l’hypothèse de récurrence, on obtient celle cherchée.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

200

J.-L. WALDSPURGER

3.5. Modèles de Whittaker et intégrales de coefficients. — On définit un caractère ξ de U (F ) par la formule X ξ(u) = ψ( ui,i+1 ). i=1,...,k−1

Soit µ ∈ Temp(G). On appelle fonctionnelle de Whittaker sur Eµ une application linéaire φ : Eµ → C telle que φ(µ(u)e) = ξ(u)φ(e) pour tous u ∈ U (F ) et e ∈ Eµ . Comme on le sait, l’espace des fonctionnelles de Whittaker sur Eµ est une droite. Soit c ≥ 1 un entier. Définissons une forme sesquilinéaire L µ,c sur Eµ (ce qui est un raccourci pour dire qu’il s’agit d’une forme sur Eµ × Eµ ) par Z 0 ¯ L µ,c (e , e) = (e0 , µ(u)e)ξ(u)du. U (F )c

Cette intégrale est absolument convergente d’après la proposition 3.2. Notons ω[1,k−1] (c) le sous-groupe des a ∈ A(F ) tels que ak = 1 et valF (1 − ai ) ≥ c pour tout i = 1, .., k − 1. Lemme. — Pour tous e, e0 ∈ Eµ , il existe un entier c0 ≥ 1 tel que L µ,c (e0 , e) soit indépendant de c pour c ≥ c0 . Plus précisément, soit c0 ≥ 1 un entier. Il existe c0 tel (c0 ) ω que cette conclusion soit vérifiée pour tous e, e0 ∈ Eµ [1,k−1] . (c0 )

ω

Démonstration. — Soient e, e0 ∈ Eµ [1,k−1] . Choisissons c0 tel que c0 ≥ 1 et −c0 + c0 ≤ cψ . Pour c ≥ c0 , notons U (F )c − U (F )c0 le complémentaire de U (F )c0 dans U (F )c . Il suffit de prouver que Z ¯ (e0 , µ(u)e)ξ(u)du = 0. U (F )c −U (F )c0 0

Soit a ∈ ω[1,k−1] (c ). Dans l’intégrale précédente, on peut remplacer e0 et e par µ(a)e0 et µ(a)e. Par le changement de variables u 7→ aua−1 , l’intégrale devient Z −1 ¯ (e0 , µ(u)e)ξ(aua )du. U (F )c −U (F )c0

Elle est donc aussi égale à mes(ω[1,k−1] (c0 ))−1

Z

Z

ω[1,k−1] (c0 )

−1 ¯ (e0 , µ(u)e)ξ(aua )du da.

U (F )c −U (F )c0

Cette expression est absolument convergente et on peut permuter les intégrales. Mais Z −1 ¯ ξ(aua )da = 0 ω[1,k−1] (c0 )

pour tout u ∈ U (F )c − U (F )c0 . Cela prouve la nullité cherchée et le lemme. On définit une forme sesquilinéaire L µ sur Eµ par

L µ (e0 , e) = limc→∞ L µ,c (e0 , e).

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

201

Cette forme vérifie les relations ¯ 0 ) L µ (e0 , e). L µ (µ(u0 )e0 , µ(u)e) = ξ(u)ξ(u Fixons une fonctionnelle de Whittaker φ sur Eµ , non nulle. Alors il existe Cµ ∈ C tel que L µ (e0 , e) = Cµ φ(e0 )φ(e) pour tous e, e0 . On montrera plus loin que Cµ 6= 0. Pour un entier c0 ∈ N, notons ιc0 la fonction caractéristique du sous-ensemble des a ∈ A(F ) tels que valF (ai ) ≥ valF (ai+1 ) − c0 pour tout i = 1, . . . , k − 1. On a (1) il existe un réel R et, pour tous e, e0 ∈ Eµ , il existe un entier c0 ∈ N tel que | L µ (µ(a0 )e0 , µ(a)e)|  ιc0 (a0 )δB (a0 )1/2 σ(a0 )R ιc0 (a)δB (a)1/2 σ(a)R pour tous a, a0 ∈ A(F ). Si Cµ = 0, c’est évident : L µ = 0. Sinon, fixons e0 ∈ Eµ tel que φ(e0 ) = 1. On a

L µ (µ(a0 )e0 , µ(a)e) = Cµ φ(µ(a0 )e0 )φ(µ(a)e) = Cµ φ(µ(a0 )e0 )φ(e0 )φ(e0 )φ(µ(a)e) −1

= Cµ L µ (e0 , µ(a0 )e0 ) L µ (e0 , µ(a)e). Cela nous ramène à majorer | L µ (e0 , µ(a)e)|. D’après le lemme ci-dessus, on peut remplacer L µ par L µ,c pour c assez grand. La majoration voulue résulte de la proposition 3.2 et du fait qu’il existe c0 tel que le support de la fonction a 7→ φ(µ(a)e) soit contenu dans celui de ιc0 . Indiquons la preuve bien connue de cette dernière propriété. On fixe un sous-groupe ouvert compact Γ de U (F ) qui fixe e. Pour tout u ∈ Γ, on a φ(µ(a)e) = φ(µ(au)e) = φ(µ(aua−1 a)e) = ξ(aua−1 )φ(µ(a)e). Si φ(µ(a)e) 6= 0, on doit donc avoir ξ(aua−1 ) = 1 pour tout u ∈ Γ. Cela entraîne la propriété voulue. 3.6. Quelques égalités d’intégrales. — Soit h un entier tel que 1 ≤ h ≤ k. Notons P h le sous-groupe parabolique de G formé des éléments qui conservent le drapeau F v1 ⊕ · · · ⊕ F vh ⊂ F v1 ⊕ · · · ⊕ F vh+1 ⊂ · · · ⊂ F v1 ⊕ · · · ⊕ F vk . Ecrivons P h = M h U h , où M h est la composante de Levi qui contient A. On a M h = GLh × GL1 × · · · × GL1 , avec k − h termes GL1 . Identifions GLh−1 au groupe des automorphismes linéaires du sous-espace de V engendré par v1 , . . . , vh−1 . Pour un entier c ≥ 1, notons ω[h,k−1] (c) le sous-groupe des γ ∈ A(F ) tels que γ1 = · · · = γh−1 = 1, γk = 1 et valF (1 − γi ) ≥ c pour i = h, . . . , k − 1. Posons U h (F )c = U (F )c ∩ U h (F ). rappelons que l’on a défini en 1.1 l’exposant cψ du conducteur de ψ. Supposons c + cψ ≥ 1. Pour µ ∈ Temp(G) et e, e0 ∈ Eµ , posons Z Z h 0 Ic (e , e) = L µ (µ(γg)e0 , µ(γg)e)|d´et(g)|Fh−k dg dγ, ω[h,k−1] (c+cψ )

Uh−1 (F )\GLh−1 (F )

Jch (e0 , e) =

Z

¯ (e0 , µ(u)e)ξ(u)du.

U h (F )c

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

202

J.-L. WALDSPURGER

Lemme. — Soient h un entier tel que 1 ≤ h ≤ k et c un entier tel que c ≥ 1 et c + cψ ≥ 1. Les intégrales ci-dessus sont absolument convergentes. Il existe C > 0 tel que Jch (e0 , e) = CIch (e0 , e) pour tous e, e0 ∈ Eµ . Démonstration. — Prouvons la convergence de Ich (e0 , e). L’intégrale sur le groupe compact ω[h,k−1] (c + cψ ) est insignifiante, on peut l’oublier. On décompose g ∈ Uh−1 (F )\GLh−1 (F ) en g = ak, avec a ∈ Ah−1 (F ), k ∈ K. La mesure devient δBh−1 (a)−1 dadk. De nouveau, on peut oublier l’intégrale sur K et on doit majorer Z h−k −1 | L µ (µ(a)e0 , µ(a)e)||d´et(a)|F δBh−1 (a)da. Ah−1 (F )

Grâce à 3.5(1), c’est majoré par Z h−k −1 ιc0 (a)δB (a)|d´et(a)|F δBh−1 (a)da Ah−1 (F )

0

pour un entier c convenable. Pour a ∈ Ah−1 (F ), on a −1 δB (a)|d´et(a)|h−k δB (a) = |d´et(a)|F F h−1

et il est immédiat que l’intégrale Z ιc0 (a)|d´et(a)|F da Ah−1 (F )

est convergente. Prouvons la convergence de Jch (e0 , e). Il suffit de prouver que Z ΞG (u)du U h (F )c

est convergente. Puisqu’on en aura besoin plus loin, prouvons la propriété plus forte suivante. On s’autorise pour un instant à faire varier l’entier c. Alors (1) il existe un réel R tel que Z ΞG (m−1 um)du  cR σ(m)R δP h (m) U h (F )c

pour tous c ≥ 1 et tout m ∈ M h (F ) Notons X(m) l’intégrale ci-dessus. On peut écrire m = vak, avec v ∈ U (F ) ∩ M h (F ), a ∈ A(F ) et k ∈ K ∩ M h (F ). La conjugaison par v conserve U h (F )c , ce qui permet de faire disparaître v. Le terme k disparaît également. Puisque σ(a)  σ(m) et δP h (a) = δP h (m), on est ramené au cas où m = a. Effectuons le changement de variable u 7→ aua−1 . Cela remplace du par δP h (a)du et le domaine d’intégration U h (F )c par l’ensemble des u ∈ U h (F ) tels que valF (ui,i+1 ) ≥ −c + valF (ai+1 ) − valF (ai )

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

203

pour tout i = h, . . . , k − 1. Il existe c1 > 0 tel que valF (ai+1 ) − valF (ai ) ≥ −c1 σ(a). L’ensemble ci-dessus est donc contenu dans U h (F )c+c1 σ(a) on obtient Z ΞG (u)du. X(a) ≤ δP h (a) U h (F )c+c1 σ(a)

Pour u0 ∈ M h (F ) ∩ U (F ) ∩ K, on a ΞG (u) = ΞG (uu0 ). On peut donc remplacer ci-dessus u par uu0 , puis intégrer en u0 . D’où Z Z ΞG (uv)du dv. X(a)  δP h (a) M h (F )∩U (F )∩K

U h (F )c+c1 σ(a)

L’ensemble d’intégration est contenu dans U (F )c+c1 σ(a) . Donc Z X(a)  δP h (a) ΞG (u)du. U (F )c+c1 σ(a)

Il ne reste plus qu’à faire appel à la proposition 3.2 pour obtenir (1). Le groupe Ah−1 (F ) agit à gauche sur Uh−1 (F )\Gh−1 (F ). Introduisons le sousgroupe ω[1,h−1] (c + cψ ) de Ah−1 (F ). Dans la définition de Ich (e0 , e), on peut remplacer g par γ 0 g pour γ 0 ∈ ω[1,h−1] (c + cψ ). On peut ensuite intégrer en γ 0 , à condition de diviser le tout par mes(ω[1,h−1] (c + cψ )). Puisque ω[1,h−1] (c + cψ )ω[h,k−1] (c + cψ ) = ω[1,k−1] (c + cψ ), on obtient Z Z Ich (e0 , e) = mes(ω[1,h−1] (c + cψ ))−1 ω[1,k−1] (c+cψ )

Uh−1 (F )\GLh−1 (F )

L µ (µ(γg)e0 , µ(γg)e)|d´et(g)|Fh−k dg dγ. Le même calcul qu’en 3.5 montre que Z L µ (µ(γg)e0 , µ(γg)e)dγ = mes(ω[1,k−1] (c + cψ )) L µ,c (µ(g)e0 , µ(g)e). ω[1,k−1] (c+cψ )

D’où

(2)

Ich (e0 , e) = mes(ω[h,k−1] (c + cψ ))

Z

L µ,c (µ(g)e0 , µ(g)e)|d´et(g)|h−k F dg.

Uh−1 (F )\GLh−1 (F )

On démontre maintenant le lemme par récurrence sur h. Pour h = 1, l’intégrale ci-dessus disparaît et Ic1 (e0 , e) = C L µ,c (e0 , e), où C > 0. Mais U h = U et Jc1 (e0 , e) = L µ,c (e0 , e) par définition. Supposons maintenant h ≥ 2 et le lemme vrai pour h − 1. Notons Y˜ le sous-groupe des éléments y ∈ GLh−1 (F ) qui vérifient — pour i = 1, . . . , h − 2, yi,i = 1 ; — pour i, j = 1, . . . , h − 1, avec i 6= j et i 6= h − 1, yi,j = 0. Par l’application y 7→ (yh−1,1 , . . . , yh−1,h−1 ), Y˜ s’identifie au complémentaire d’un hyperplan (l’hyperplan yh−1,h−1 = 0) dans un espace vectoriel Y de dimension h − 1 sur F . On note dy la mesure de Haar sur Y et sa restriction à Y˜ (ce n’est pas une

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

204

J.-L. WALDSPURGER

mesure de Haar sur cet ensemble). On vérifie qu’il existe C0 > 0 tel que, pour toute fonction intégrable ϕ sur Uh−1 (F )\GLh−1 (F ), on ait l’égalité Z Z Z ϕ(g)dg = C0 ϕ(g 0 y)|d´et(g 0 )|−1 dg 0 |yh−1,h−1 |−1 F dy. Y˜

Uh−1 (F )\GLh−1 (F )

Uh−2 (F )\GLh−2 (F )

On déduit de cette égalité et de (2) la relation Z h 0 Ic (e , e) = C1 Ich−1 (µ(y)e0 , µ(y)e)|yh−1,h−1 |h−k−1 dy F Y˜

pour un C1 > 0 convenable. Grâce à l’hypothèse de récurrence, on en déduit qu’il existe C2 > 0 tel que Z h−k−1 Ich (e0 , e) = C2 Jch−1 (µ(y)e0 , µ(y)e)|yh−1,h−1 |F dy Y˜

Z Z = C2

Y˜

U h−1 (F )c

h−k−1 ¯ (µ(y)e0 , µ(uy)e)ξ(u)du|y dy. h−1,h−1 |F

Pour n ∈ N, notons Yn le sous-ensemble des y ∈ Y tels que valF (yh−1,j ) ≥ −n pour tout j = 1, . . . , h − 1. On a (3)

Ich (e0 , e) = C2 limn→∞ Xn ,

où Z Xn =

Z

Yn ∩Y˜

U h−1 (F )c

h−k−1 ¯ (µ(y)e0 , µ(uy)e)ξ(u)du|y dy. h−1,h−1 |F

Cette dernière expression est absolument convergente. En effet, remplaçons tous les termes par leurs valeurs absolues. D’après (1), l’intégrale intérieure est majorée par k+1−h . L’expression totale est donc majorée par σ(y)R δP h−1 (y) = σ(y)R |yh−1,h−1 |F Z σ(y)R dy Yn h−1

qui est convergente. Posons Z = U (F ) ∩ M h (F ). Pour y ∈ Y et z ∈ Z, posons P x(y, z) = i=1,...,h−1 yh−1,i zi,h . Pour y ∈ Y , notons Z(y) le sous-ensemble des z ∈ Z tels que valF (x(y, z)) ≥ −c et , pour z ∈ Z, notons Y (z) le sous-ensemble des y ∈ Y vérifiant la même condition. Dans Xn , effectuons le changement de variable u 7→ yuy −1 . Cela remplace le domaine d’intégration U h−1 (F )c par U h (F )c Z(y), donc la dz dv variable u par vz, avec v ∈ U h (F )c et z ∈ Z(y), la mesure du par |yh−1,h−1 |k+1−h F et ξ(u) par ξ(v)ψ(x(y, z)). On obtient Z Z Z ¯ ψ(x(y, ¯ Xn = (e0 , µ(vz)e)ξ(v) z))dz dv dy. Yn ∩Y˜

U h (F )c

Z(y)

D’après l’absolue convergence de cette expression, on peut permuter les intégrales et on obtient Z ¯ Xn = Xn (v)ξ(v)dv, U h (F )c

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

205

où Z

(e0 , µ(vz)e)

Xn (v) = Z

¯ ψ(x(y, z))dy dz

Yn ∩Y (z)∩Y˜

Z

=

Z

(e0 , µ(vz)e)

Z

¯ ψ(x(y, z))dy dz.

Yn ∩Y (z)

Z

Fixons z ∈ Z. L’ensemble Yn ∩ Y (z) est un oF -réseau dans Y et l’application y 7→ ¯ ψ(x(y, z)) est un caractère de Y . Son intégrale sur le réseau est nulle si le caractère y est non trivial et vaut la mesure du réseau si le caractère y est trivial. On a ¯ (4) le caractère y 7→ ψ(x(y, z)) est trivial sur Yn ∩ Y (z) si et seulement si valF (zi,h ) ≥ n + cψ pour tout i = 1, . . . , h − 1. En effet, ce caractère est trivial si et seulement si valF (x(y, z)) ≥ cψ pour tout y ∈ Yn ∩ Y (z). Cette condition est satisfaite si z vérifie les conditions de (3). Inversement, s’il existe i tel que valF (zi,h ) < n + cψ , soit y ∈ Y dont la seule coordonnée non nulle soit yh−1,i de valuation cψ − valF (zi,h) − 1. Ce nombre est ≥ −n, donc y ∈ Yn . On a valF (x(y, z)) = cψ − 1. On a supposé c + cψ ≥ 1, donc valF (x(y, z)) ≥ −c, ce qui entraîne y ∈ Y (z). La condition valF (x(y, z)) ≥ cψ n’est pas vérifiée pour cet y. D’où (4). Notons Z n l’ensemble des z ∈ Z vérifiant les conditions de (4). Alors Z (e0 , µ(vz)e)mes(Yn ∩ Y (z))dz. Xn (v) = Zn

L’inégalité c + cψ ≥ 1 entraîne que, pour z ∈ Z n , on a Yn ∩ Y (z) = Yn . D’autre part, si n est assez grand, on a µ(z)e = e pour tout z ∈ Z n . Alors Xn (v) = mes(Yn )mes(Z n )(e0 , µ(v)e). Le produit des mesures est une constante positive, disons C3 . Pour n assez grand, on a donc Z ¯ Xn = C3 (e0 , µ(v)e)ξ(v)dv = C3 Jch (e0 , e). U h (F )c

En reportant cette égalité dans (3), on obtient Ich (e0 , e) = C2 C3 Jch (e0 , e), ce qui achève la preuve. 3.7. Propriétés des fonctionnelles de Whittaker. — Soient µ ∈ Temp(G). Appliquons le lemme précédent pour h = k. On obtient une égalité Z L µ (µ(g)e0 , µ(g)e)dg = C(e0 , e) Uk−1 (F )\GLk−1 (F ) 0

pour tous e, e ∈ Eµ , où C est une constante positive. Il en résulte que L µ est non nulle, autrement dit que la constante Cµ du paragraphe 3.5 est non nulle. On peut alors récrire la relation 3.5(1) et le lemme 3.6 sous la forme suivante.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

206

J.-L. WALDSPURGER

Lemme. — Soient µ ∈ Temp(G) et φ une fonctionnelle de Whittaker non nulle sur Eµ . (i) Il existe un réel R et, pour tout e ∈ Eµ , il existe c0 ∈ N tel que |φ(µ(a)e)|  ιc0 (a)δB (a)1/2 σ(a)R pour tout a ∈ A(F ). (ii) Pour tout h = 1, . . . , k et tout entier c tel que c ≥ 1 et c + cψ ≥ 1, il existe C > 0 tel que l’on ait l’égalité Z Z h−k φ(µ(ag)e0 )φ(µ(ag)e)|d´et(g)|F dg da ω[h,k−1] (c+cψ )

Uh−1 (F )\GLh−1 (F )

Z =C

¯ (e0 , µ(u)e)ξ(u)du

U h (F )c

pour tous e, e0 ∈ Eµ .

4. Majorations pour un groupe spécial orthogonal 4.1. Les groupes spéciaux orthogonaux. — Soit (V, qV ) un espace quadratique sur F , c’est-à-dire que V est un espace vectoriel de dimension finie sur F et qV est une forme bilinéaire symétrique non dégénérée sur V (on dira souvent que V est un espace quadratique, la forme qV étant sous-entendue). On note aussi qV la forme quadratique définie par qV (v) = qV (v, v)/2. On note dV la dimension de V et G le groupe spécial orthogonal de V . Considérons un système hyperbolique maximal (v±i )i=1,...,l dans V (« système hyperbolique » signifie que qV (vi , vj ) = δi,−j pour tous i, j, où δi,−j est le symbole de Kronecker). Notons Z le sous-espace de V engendré par ce système et Van l’orthogonal de Z dans V . La restriction qVan de qV à Van est anisotrope. On note dan,V la dimension de Van . Fixons un réseau spécial Ran ⊂ Van ([17] 7.1). On peut choisir un réseau RZ de Z ayant une base formée de vecteurs proportionnels aux vi , de sorte que R = RZ ⊕ Ran soit spécial. On note K le stabilisateur de R dans G(F ). C’est un sous-groupe compact spécial de G(F ). Considérons une suite P d’entiers (k1 , . . . , ks ) telle que kj ≥ 1 pour tout j et j=1,...,s kj ≤ l. Pour tout j, notons Zj , resp. Z−j , le sous-espace de V engendré par les vi , resp. v−i , pour i = k1 + · · · + kj−1 + 1, . . . , k1 + · · · + kj . Notons V˜ l’orthogonal dans V de la somme ˜ le groupe spécial orthogonal de V˜ . Notons P le sous-groupe des Z±j et notons G parabolique de G formé des éléments qui conservent le drapeau de sous-espaces Z1 ⊂ Z1 ⊕ Z2 ⊂ · · · ⊂ Z1 ⊕ · · · ⊕ Zs . Notons M la composante de Levi de P formée des éléments qui conservent chaque sous-espace Z±j . On a (1)

ASTÉRISQUE 346

˜ M ' GLk1 × · · · × GLks × G.

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

207

On sait que K est en bonne position relativement à M . Inversement, si P = M U est un sous-groupe parabolique de G, si K est un sous-groupe compact spécial de G(F ) en bonne position relativement à M , on peut trouver un système hyperbolique, un réseau spécial et une suite d’entiers de sorte que P , M et K soient déterminés comme ci-dessus (ces données ne sont pas uniques). Si s = l et kj = 1 pour tout j, M est un Levi minimal et inversement, si M est un Levi minimal, on peut supposer ces égalités vérifiées. Dans la situation ci-dessus, supposons M minimal et notons-le plutôt Mmin . L’application naturelle K ∩ NormG(F ) (Mmin ) → W G est surjective et il est utile de remarquer qu’elle possède une section ι : W G → K ∩ NormG(F ) (Mmin ) qui est un homomorphisme de groupes. En effet, pour tout i = ±1, . . . , ±l, fixons un élément vi0 ∈ F vi de sorte que (vi0 )i=±1,...,±l soit une base sur oF de RZ . Parce que R est un réseau 0 0 0 ) = qV (vi00 , v−i spécial, on peut supposer que qV (vi0 , v−i 0 ) pour tous i, i = 1, . . . , l. Si Van 6= {0}, fixons un élément gan du groupe orthogonal de Van tel que d´et(gan ) = −1 2 = 1. L’action de tout élément de ce groupe orthogonal, en particulier l’acet gan tion de gan , conserve le réseau Ran . Notons W K le sous-ensemble des éléments g ∈ G(F ) qui agissent par permutation sur l’ensemble {v±i ; i = 1, . . . , l} et agissent sur Van , soit par l’identité, soit comme gan . On vérifie que W K est un sous-groupe de K ∩ NormG(F ) (Mmin ) et que l’application naturelle de W K dans W G est un isomorphisme. Les hypothèses de 1.5 sont vérifiées pour le groupe G. Soient M un Levi que l’on écrit sous la forme (1) et τ une représentation admissible irréductible et de la série discrète de M (F ). On a τ ' µ1 ⊗ · · · ⊗ µs ⊗ τ˜, ˜ ). où µj , resp. τ˜, est une représentation de la série discrète de GLkj (F ), resp. G(F s L’espace A M s’identifie naturellement à R . Supposons R(τ ) ∩ W (M )reg 6= ∅. Alors le groupe R(τ ) s’identifie à un sous-groupe de {±1}s . Un élément ε = (ε1 , . . . , εs ) de ce groupe agit sur Rs par (x1 , . . . , xs ) 7→ (ε1 x1 , . . . , εs xs ). Le groupe R(τ ) contient l’élément t = (−1, . . . , −1) de {±1}s , qui est l’unique élément de R(τ ) ∩ W (M )reg . On a |d´et(t − 1)| A M | = 2aM . Soit π une représentation tempérée irréductible et elliptique de G(F ). On peut trouver M , τ comme ci-dessus, et ζ ∈ R(τ )∨ de sorte que π = IndG P (τ, ζ), où P est un élément de P (M ). Puisque la classe de conjugaison du couple (M, τ ) est bien déterminée, on peut poser r(π) = |R(τ )| et t(π) = 2aM . 4.2. Espaces quadratiques compatibles. — Soient (V, qV ) et (W, qW ) deux espaces quadratiques. Notons G et H leurs groupes spéciaux orthogonaux, dV et dW les dimensions de V et W . Supposons par exemple dW ≤ dV . On dit que les deux espaces quadratiques sont compatibles si V est isomorphe (comme espace quadratique) à la somme orthogonale de W , d’une droite D0 et d’un espace hyperbolique Z. On peut alors identifier W à un sous-espace de V et H à un sous-groupe de G. D’après le

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

208

J.-L. WALDSPURGER

théorème de Witt, cette identification est unique à conjugaison près par un élément de G(F ). Soulignons que la compatibilité de V et W entraîne que dV et dW sont de parités distinctes. Supposons que V soit la somme directe orthogonale de deux sous-espaces V 0 et Z 0 , avec Z 0 hyperbolique. Alors V et W sont compatibles si et seulement si V 0 et W le sont. Soient V et W deux espaces quadratiques compatibles, avec dW < dV . On fixe un isomorphisme V = W ⊕ D0 ⊕ Z avec les propriétés ci-dessus, une base hyperbolique (v±i )i=1,...,r de Z et un élément non nul v0 ∈ D0 . On pose V0 = W ⊕ D0 et on note G0 son groupe spécial orthogonal. On note A le sous-tore maximal du groupe spécial orthogonal de Z qui conserve chaque droite F v±i . Pour a ∈ A(F ) et i = ±1, . . . , ±r, on note ai la valeur propre de a sur le vecteur vi . On note P le sous-groupe parabolique de G formé des éléments qui conservent le drapeau F vr ⊂ F vr ⊕ F vr−1 ⊂ · · · ⊂ F vr ⊕ · · · ⊕ F v1 de V . On note U le radical unipotent de P et M sa composante de Levi qui contient A. On a l’égalité M = AG0 . On définit un caractère ξ de U (F ) par X ξ(u) = ψ( qV (uvi , v−i−1 )). i=0,...,r−1

On fixe un réseau spécial R0 ⊂ V0 , cf. [17] 7.1. On choisit, ainsi qu’il est loisible, un réseau RZ ⊂ Z possédant une base sur oF formée de vecteurs proportionnels aux v±i et tel que le réseau R = R0 ⊕ RZ soit spécial. On note K le stabilisateur de R dans G(F ). Pour un entier N ≥ 1, on définit une fonction κN sur G(F ) de la façon suivante. Elle est invariante à gauche par U (F ) et à droite par K. Sa restriction à M (F ) est la fonction caractéristique des éléments ag0 , avec a ∈ A(F ) et g0 ∈ G0 (F ), qui vérifient les conditions |valF (ai )| ≤ N pour tout i = 1, . . . , r et g0−1 v0 ∈ p−N F R0 . Remarque. — Les constructions et notations ci-dessus seront utilisées sans plus de commentaires chaque fois que l’on se donnera des espaces quadratiques compatibles V et W avec dW < dV .

4.3. Les résultats. — On énonce ici toutes les majorations que la section est destinée à prouver. On fixe pour toute cette section deux espaces quadratiques compatibles (V, qV ) et (W, qW ) tels que dV > dW . (1) Il existe un réel R tel que Z 1σ 0 tel que I(c, c0 , N, D)  N −R pour tout N ≥ 2 et tout c0 ≥ αlog(N ). (8) l’intégrale I(c, c0 , N, C, D) est convergente ; les termes c et D étant fixés, pour tout réel R, il existe α > 0 tel que I(c, c0 , N, C, D)  N −R pour tout N ≥ 1, tout c0 ≥ 1 et tout C ≥ α(log(N ) + c0 ). 4.4. Preuve de la majoration 4.3(1). — Fixons un Levi minimal Mmin de G tel que K soit en bonne position relativement à Mmin . Soit Pmin = Mmin Umin ∈ P (Mmin ). On a Z Z Z Z 1σ 0 tel que Ir,\ (b, D)  exp(−εb) pour tout b ≥ 0. Démonstration. — Notons V[ l’orthogonal dans V de l’espace de dimension 4 engendré par vr , vr−1 , v1−r et v−r . Pour x ∈ V[ et y ∈ F , notons u(x, y) l’unique élément de U\ (F ) tel que u(x, y)v−r = v−r + x + yvr−1 − qV (x)vr . Alors (x, y) 7→ u(x, y) est un isomorphisme de V[ × F sur U\ (F ). On pose simplement u(x) = u(x, 0) et y = u(0, y). Une description analogue vaut sur le corps de base F¯ . On note Ur,] , resp. Y , le sousgroupe de Ur,\ formé des u(x), resp. y. On a Ur,\ = Ur,] Y . Notons V] l’orthogonal dans V du plan engendré par vr−1 et v1−r . Notons G] le groupe spécial orthogonal de V] et affectons d’un indice ] les intersections avec G] des groupes que l’on a introduits. En particulier Pr,] = G] ∩ Pr . On voit que Ur,] n’est autre que le radical unipotent de Pr,] . Soit x ∈ V[ , introduisons l’élément mP¯] (u(x)), que l’on écrit a(x)g0 (x), avec a(x) ∈ A(F ), g0 (x) ∈ G0 (F ). On a a(x)r−1 = 1 puisque a(x) ∈ G] (F ). Pour tout v ∈ V , notons valR (v) le plus grand entier n ∈ Z tel que v ∈ pnF R. On a (1) il existe c1 , c2 ∈ Z tel que valF (a(x)r ) = inf (0, c1 + valR (x), c2 + valF (qV (x))) ; il existe ε1 > 0 tel que |a(x)r |−1 F  exp(−ε1 σ(u(x))). En effet, posons k = u(x)−1 mP¯] (u(x))uP¯] (u(x)). On a k = kP¯] (u(x))−1 ∈ K. Donc valR (kv−r ) = valR (v−r ). Mais kv−r = a(x)−1 r (v−r − x − qV (x)vr ), d’où valR (kv−r ) = −valF (a(x)r ) + valR (v−r − x − qV (x)vr ). D’après la définition de R, on a valR (v−r − x − qV (x)vr ) = inf (valR (v−r ), valR (x), valF (qV (x) + valR (vr )). En utilisant toutes ces égalités, on obtient valF (a(x)r ) = inf (0, −valR (v−r ) + valR (x), valR (vr ) − valR (v−r ) + valF (qV (x))). D’où la première assertion de (1). La seconde s’en déduit immédiatement. On a l’égalité mP¯ (u(x)) = mP¯] (u(x)). On calcule Y 1−i−dV0 /2 |a(x)i |F , δP¯ (mP¯ (u(x)))1/2 ΞM (mP¯ (u(x))) = ΞG0 (g0 (x)) i=1,...,r

δP¯] (mP¯] (u(x)))1/2 ΞM] (mP¯] (u(x))) =

2−r−dV0 /2 ΞG0 (g0 (x))|a(x)r |F

Y

1−i−dV0 /2

|a(x)i |F

,

i=1,...,r−2

d’où 1/2 M] (2) δP¯ (mP¯ (u(x)))1/2 ΞM (mP¯ (u(x))) = |a(x)r |−1 Ξ (mP¯] (u(x))). F δP¯] (mP¯] (u(x)))

Soient x ∈ V[ et y ∈ F . On a mP¯ (u(x)y) = mP¯ (u(x))mP¯ (kP¯ (u(x)y). Il existe donc R1 > 0 tel que (3) δP¯ (mP¯ (u(x)y))1/2 ΞM (mP¯ (u(x)y))  δP¯ (mP¯ (u(x)))1/2 ΞM (mP¯ (u(x))exp(R1 σ(y)),

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

212

J.-L. WALDSPURGER

où σ(y) = sup(1, −valF (y)). Introduisons un réel µ > 0 que nous fixerons plus tard et posons Z Z 1 1σ≥b (u(x)y)δP¯ (mP¯ (u(x)y))1/2 Ir,\ (b, D) = y∈F ;valF (y)≥−µb

V[

M

Ξ (mP¯ (u(x)y))σ(u(x)y)D dx dy. Pour y tel que valF (y) ≥ −µb, on a |σ(u(x)y) − σ(u(x))|  µb. Si µ est assez petit, la condition 1σ≥b (u(x)y) = 1 entraîne 1σ≥b/2 (u(x)) = 1. Grâce à (3), on obtient Z 1 1σ≥b/2 (u(x))δP¯ (mP¯ (u(x)))1/2 ΞM (mP¯ (u(x)))σ(u(x))D dx Ir,\ (b, D)  V[

Z

(µb)D exp(R1 σ(y))dy.

y∈F ;valF (y)≥−µb

Il existe R2 tel la dernière intégrale soit bornée par exp(R2 µb). Dans la première intégrale, on utilise (1) et (2). Pour 1σ≥b/2 (u(x)) = 1, on a δP¯ (mP¯ (u(x)))1/2 ΞM (mP¯ (u(x))  exp(−ε1 b/4 − ε1 σ(u(x))/2)δP¯ (mP¯] (u(x)))1/2 ΞM] (mP¯] (u(x))). Alors 1 Ir,\ (b, D)  exp(R2 µb − ε1 b/4)

Z

exp(−ε1 σ(u(x))/2)δP¯ (mP¯] (u(x)))1/2

V[

ΞM] (mP¯] (u(x)))σ(u(x))D dx. D’après [16] lemme II.4.3, cette intégrale est convergente. Choisissons µ tel que ε1 /4− R2 µ > 0 et notons ε2 ce dernier terme. On obtient alors 1 Ir,\ (b, D)  exp(−ε2 b).

Le réel µ étant maintenant fixé, posons Z Z 2 Ir,\ (b, D) = y∈F ;valF (y) 0 que nous préciserons par la suite. On décompose Xr (c, D, x) en Xr, 0 convenable. D’après [16], lemmes II.1.1 et II.3.2, il existe un réel D00 tel que 00 ΞG (g)  δP¯ (mP¯ (g))1/2 ΞM (mP¯ (g))σ(g)D pour tout g ∈ G(F ). Alors l’intégrale ci-dessus est bornée par l’intégrale Ir,\ (b/2, D + D00 ) de 4.5. En utilisant les lemmes 4.5 ou 4.6 selon la valeur de r, elle est essentiellement bornée par exp(−εb) pour un ε > 0 convenable. Enfin, on vérifie qu’il existe c2 > 0 tel que l’on ait la majoration exp(−c2 σ(x))  δPr (x)1/2 ΞMr (x). Alors Xr,≥b (c, D, x)  exp(αc1 c + (α + c2 )σ(x) − εb)δPr (x)1/2 ΞMr (x). On choisit maintenant b = 2c1 c + ε−1 αc1 c + ε−1 (α + c2 )σ(x). Alors Xr,≥b (c, D, x)  δPr (x)1/2 ΞMr (x). Cette majoration et (2) entraînent (1). 4.8. Majoration d’intégrales doubles sur U (F ). — Soient D un réel, c et c0 deux entiers. Supposons c0 ≥ c > 0. Remarquons que l’ensemble U (F ) − U (F )c0 est invariant par translation par U (F )c . Pour m, m0 ∈ M (F ), posons Z Z Z 0 X(c, D, m, m ) = ΞG (uvm)ΞG (uv 0 m0 )σ(uvm)D σ(uv 0 m0 )D dv 0 dv du, U(F )/U(F )c

U(F )c

X(c, c0 , D, m, m0 ) =

U(F )c

Z (U (F )−U (F )c0 )/U (F )c

ASTÉRISQUE 346

Z U (F )c

Z U (F )c

ΞG (uvm)

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

219

ΞG (uv 0 m0 )σ(uvm)D σ(uv 0 m0 )D dv 0 dv du, Z 0 X(D, m, m ) = ΞG (um)ΞG (um0 )σ(um)D σ(um0 )D du. U (F )

Lemme. — (i) Ces intégrales sont convergentes. (ii) Le réel D étant fixé, il existe un réel R tel que X(c, D, m, m0 )  cR σ(m)R δP (m)1/2 ΞM (m)σ(m0 )R δP (m0 )1/2 ΞM (m0 ) pour tous m, m0 ∈ M (F ) et tout c ≥ 1. (iii) Les termes c et D étant fixés, il existe un réel R et un réel ε > 0 tels que X(c, c0 , D, m, m0 )  exp(−εc0 )σ(m)R δP (m)1/2 ΞM (m)σ(m0 )R δP (m0 )1/2 ΞM (m0 ) pour tous m, m0 ∈ M (F ) et tout c0 ≥ c. (iv) Le réel D étant fixé, il existe un réel R tel que X(D, m, m0 )  σ(m)R δP (m)1/2 ΞM (m)σ(m0 )R δP (m0 )1/2 ΞM (m0 ) pour tous m, m0 ∈ M (F ). Démonstration. — L’application r (F/p−c F ) −c (ur,r−1 + pF , . . . , u1,0 + p−c F )

→ 7→

U (F )/U (F )c u

est un isomorphisme. Définissons une fonction valcF sur F par valcF (x) = 0 si x ∈ p−c F , −c . Elle se quotiente en une fonction sur F/p . Pour valcF (x) = valF (x) + c si x 6∈ p−c F F P u ∈ U (F )/U (F )c , posons valcF (u) = i=1,...,r valcF (ui,i−1 ). Montrons que : (1) il existe un réel D1 tel que l’on ait une majoration Z c ΞG (uvm)σ(uvm)D dv  (c − valcF (u))D1 q valF (u) σ(m)D1 δP (m)1/2 ΞM (m) U (F )c

pour tout m ∈ M (F ), tout c ≥ 1 et tout u ∈ U (F )/U (F )c . Soit a ∈ A(F ) ∩ K. On peut remplacer ΞG (uvm)σ(uvm)D

par ΞG (auvma−1 )σ(auvma−1 )D .

On peut ensuite intégrer en a. Puisque a commute à m et normalise U (F )c , on obtient Z Z Z ΞG (uvm)σ(uvm)D dv  ΞG (aua−1 vm)σ(aua−1 vm)D dv da. U (F )c

A(F )∩K

U (F )c

Considérons l’application A(F ) ∩ K a

→

U (F )/U (F )c

7→ aua−1 U (F )c .

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

220

J.-L. WALDSPURGER

c

On vérifie que son jacobien est borné par q valF (u) . Son image est contenue dans U (F )−valcF (u)+c /U (F )c . D’où Z Z Z c ΞG (uvm)σ(uvm)D dv du ΞG (uvm)σ(uvm)D dv  q valF (u) U(F )c

U(F )−valc

F

c

 q valF (u)

Z

(u)+c /U(F )c

U(F )c

ΞG (vm)σ(vm)D dv.

U (F )−valc

F

(u)+c

Il reste à faire appel à 4.3(3) pour obtenir l’assertion (1). Grâce à (1), on a X(c, D, m, m0 )  σ(m)D1 δP (m)1/2 ΞM (m)σ(m0 )D1 δP (m0 )ΞM (m0 ) Z c (c − valcF (u))2D1 q 2valF (u) du. U (F )/U (F )c

Cette dernière intégrale est produit d’intégrales du type Z c (c − valcF (x))2D1 q 2valF (x) dx. (2) F/p−c F

On vérifie qu’il existe un réel D2 tel que cette expression soit essentiellement bornée par cD2 . On en déduit une majoration similaire pour l’intégrale intervenant dans la formule ci-dessus. Alors X(c, D, m, m0 ) vérifie la majoration du (ii) de l’énoncé. Grâce à (1), on a X(c, D, m, m0 )  σ(m)D1 δP (m)1/2 ΞM (m)σ(m0 )D1 δP (m0 )ΞM (m0 ) Z c (c − valcF (u))2D1 q 2valF (u) du. (U (F )−U (F )c0 )/U (F )c

Cette dernière intégrale est combinaison linéaire de termes qui sont des produits d’intégrales du type (2) et d’au moins une intégrale du type Z c (c − valcF (x))2D1 q 2valF (x) dx. (3) 0 −c (F −pF )/p−c F

On vérifie qu’il existe ε > 0 tel que cette expression soit essentiellement bornée par exp(−εc0 ). On en déduit le (iii) de l’énoncé. Soit v ∈ U (F ) ∩ K. Dans l’intégrale X(D, m, m0 ), on peut remplacer ΞG (um) par G Ξ (vum), puis intégrer sur v. Donc Z Z 0 X(D, m, m )  ΞG (vum)ΞG (um0 )σ(vum)D σ(um0 )D du dv. U (F )∩K

U (F )

Choisissons c tel que U (F ) ∩ K ⊂ U (F )c . On peut remplacer l’intégrale sur U (F ) ∩ K par l’intégrale sur U (F )c . Ce groupe étant distingué dans U (F ), on peut remplacer vu par uv. On peut ensuite décomposer l’intégrale sur U (F ) en la composée d’une intégrale sur U (F )c et d’une intégrale sur U (F )/U (F )c . On obtient alors X(D, m, m0 )  X(c, D, m, m0 ), et l’assertion (iv) résulte de (ii).

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

221

4.9. Comparaison de ΞG et ΞH Lemme. — Supposons r = 0. Il existe ε > 0 tel que ΞG (h)  exp(−εσ(h))ΞH (h) pour tout h ∈ H(F ). Démonstration. — On a nécessairement dan,V = dan,W ± 1. Si dan,V = dan,W + 1, fixons un système hyperbolique maximal (e±j )j=1,...,n de W . C’est aussi un système hyperbolique maximal de V . Si dan,V = dan,W − 1, fixons un système hyperbolique maximal (e±j )j=2,...,n de W . Notons Wan l’orthogonal dans W du sous-espace engendré par ces vecteurs. Fixons un système hyperbolique maximal (e±1 ) de Wan ⊕ D0 . Alors (e±j )j=1,...,n est un système hyperbolique maximal de V . On pose ι = 1 dans le premier cas, ι = 2 dans le second. Notons Amin le sous-tore déployé maximal de G formé des éléments qui conservent chaque droite F e±j pour j = 1, . . . , n et qui agissent trivialement sur l’orthogonal du sous-espace engendré par ces vecteurs. Pour a ∈ Amin (F ), et j = 1, . . . , n, on note aj la valeur propre de a sur le vecteur ej . Posons AH min = Amin ∩ H. C’est un sous-tore déployé maximal de H. On sait qu’il existe un sous-ensemble compact Γ de H(F ) tel que H(F ) = ΓAH min (F )Γ. Il suffit donc de H démontrer le lemme pour les éléments de Amin (F ). On va démontrer (1) il existe un réel D tel que P − |valF (hj )|/2 j=ι,...,n ΞG (h)  ΞH (h)σ(h)D q pour tout h ∈ AH min (F ). Cette relation implique la majoration de l’énoncé. Notons S le groupe des permutations s de {±1, . . . , ±n} telles que s(−j) = −s(j) pour tout j ∈ {±1, . . . , ±n}. Si dan,V = dan,W + 1, on pose SH = S ; si dan,V = dan,W − 1, on note SH le sousgroupe des s ∈ S qui fixent 1 et −1. Pour tout s ∈ S, notons Amin (F )− s l’ensemble des a ∈ Amin (F ) tels que valF (asn ) ≥ · · · ≥ valF (as1 ) ≥ 0. H − Le groupe AH min (F ) est contenu dans la réunion des Amin (F )s quand s décrit S . H − On peut fixer s et se limiter à prouver (1) pour h ∈ Amin (F ) ∩ Amin (F )s . Fixons donc s. Quitte à réindexer notre système hyperbolique, on peut supposer s = 1. On abandonne les indices s, en conservant les exposants −. Notons Pmin le sous-groupe parabolique de G formé des éléments qui conservent le drapeau

F en ⊂ F en ⊕ F en−1 ⊂ · · · ⊂ F en ⊕ · · · ⊕ F e1 . H − Posons Pmin = Pmin ∩ H. D’après [16] lemme II.1.1, pour h ∈ AH min (F ) ∩ Amin (F ) , on a 1/2 H (h) ΞH (h)  δPmin .

Ce dernier terme est égal à Y

j−ι+dan,W /2

|hj |F

j=ι,...,n

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

222

J.-L. WALDSPURGER

D’après la même référence, il existe un réel D tel que 1/2 G (h) ΞG (h)  σ(h)D δPmin .

Ce dernier terme s’écrit Y

j−1+dan,V /2

|hj |F

.

j=1,...,n

Remarquons que dan,V /2 − 1 = 1/2 + dan,W /2 − ι et h1 = 1 si ι = 2. Ces relations entraînent (1).

4.10. Preuve des relations 4.3(4) et 4.3(5). — On veut prouver la convergence de Z (1) ΞH (h)ΞG (hu)σ(hu)D du dh. H(F )U (F )c

On utilise 4.3(3) pour majorer l’intégrale sur U (F )c . En remarquant que δP (h) = 1 et ΞM (h) = ΞG0 (h), on obtient qu’il existe un réel D0 tel que l’intégrale ci-dessus soit essentiellement bornée par Z 0 ΞH (h)ΞG0 (h)σ(h)D dh. H(F )

En appliquant le lemme 4.9 à G0 , il existe ε > 0 tel que cette expression soit essentiellement bornée par Z ΞH (h)2 exp(−εσ(h))dh. H(F )

Or cette intégrale est convergente d’après [16] lemme II.1.5. D’où la relation 4.3(4). On veut prouver la convergence de Z Z (2) ΞG (hu)ΞH (h0 h)ΞG (h0 u0 )σ(hu)D σ(h0 u0 )D du0 dh0 du dh. H(F )U (F )c

H(F )U (F )c

Soit K H le sous-groupe compact spécial de H(F ) sous-jacent à la définition de ΞH . On peut remplacer h par kh, avec k ∈ K H , puis intégrer sur k, tout en divisant par mes(K H ). Or ΞG (khu)  ΞG (hu) et σ(khu)  σ(hu). Le procédé ci-dessus revient donc à remplacer le terme ΞH (h0 h) par Z mes(K H )−1 ΞH (h0 kh)dk. KH

D’après [16] lemme II.1.3, ceci n’est autre que ΞH (h0 )ΞH (h). Alors l’expression (2) apparaît comme le carré de l’expression (1). Elle est convergente puisque (1) l’est. Cela prouve la relation 4.3(5).

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

223

4.11. Majoration d’une intégrale de fonctions d’Harish-Chandra, cas r = 0. — On suppose dans ce paragraphe r = 0. Soit D un réel. Pour h ∈ H(F ) et N ≥ 1 un entier, posons Z χ(h, N, D) = ΞG (hx)ΞG (x)κN (x)σ(x)D dx. G(F )

Lemme. — Cette intégrale est convergente. Le réel D étant fixé, il existe un réel R tel que χ(h, N, D)  ΞG (h)N R σ(h)R pour tout h ∈ H(F ) et tout entier N ≥ 1. Démonstration. — Si V est anisotrope, le groupe G(F ) est compact et l’assertion est évidente. On suppose que V n’est pas anisotrope. Comme dans la preuve de 4.9, dont on reprend les notations, on introduit un système hyperbolique maximal (e±j )j=1,...,n de V . On note Van l’orthogonal dans V du sous-espace engendré par ces vecteurs. Si dan,V = dan,W + 1, v0 appartient à Van . Si dan,V = dan,W − 1, on peut choisir e1 et e−1 de sorte que v0 = e1 + ν0 e−1 , où ν0 = qV (v0 ). Il suffit de démontrer le lemme H pour h ∈ AH de H(F ) tel que min (F ). En effet, il existe un sous-ensemble compact Γ H H H 0 0 H H(F ) = Γ Amin (F )Γ . Ecrivons h = γ aγ, avec γ, γ ∈ Γ et a ∈ AH min (F ). On effectue le changement de variable x 7→ γ −1 x. Puisque la fonction κN est invariante à gauche par H(F ), on obtient Z χ(h, N, D) = ΞG (γ 0 ax)ΞG (γ −1 x)κN (x)σ(γ −1 x)D dx. G(F )

Mais cette expression est essentiellement majorée par χ(a, N, D), d’où l’assertion. On suppose donc h ∈ AH min (F ). Fixons un sous-groupe d’Iwahori I de G(F ) en bonne position relativement à Amin . Il est loisible de supposer que ΞG est biinvariante par I (par contre, on ne suppose pas que I soit inclus dans le sous-groupe compact spécial K que l’on a fixé, c’est-à-dire dans le fixateur du réseau R). D’après BruhatTits, il existe un sous-ensemble ouvert compact Γ de G(F ) tel que G(F ) = IAmin (F )Γ. On fixe Γ et on suppose I ⊂ Γ. Le groupe I ∩ Amin (F ) est le sous-groupe compact maximal de Amin (F ). L’application Zn

Amin (F ) → a

7→ (valF (a1 ), . . . , valF (an ))

se quotiente en un isomorphisme de Amin (F )/(I ∩ Amin (F )) sur Zn . Fixons un sousensemble Λ ⊂ Amin (F ) qui s’envoie bijectivement sur Zn . Alors X χ(h, N, D) ≤ χ(h, N, D, a), a∈Λ

où Z χ(h, N, D, a) =

ΞG (hx)ΞG (x)κN (x)σ(x)D dx.

IaΓ

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

224

J.-L. WALDSPURGER

Comme en 4.9, introduisons le groupe de permutations S et, pour tout s ∈ S, le − − − sous-ensemble Amin (F )− s . Posons Λs = Λ ∩ Amin (F )s . Alors Λ est réunion des Λs . − Il nous suffit de fixer s et de majorer χ(h, N, D)s , où X χ(h, N, D)− χ(h, N, D, a). s = a∈Λ− s

Fixons donc s. Quitte à réindexer notre système hyperbolique, on peut supposer s = 1. On abandonne les indices s dans les notations précédentes, en conservant seulement les exposants −. Il faut prendre garde au fait que, dans le cas où dan,V = dan,W − 1, la réindexation n’a pas de raison de conserver les vecteurs e1 et e−1 . Ceux-ci se transforment en des vecteurs que nous notons e±t , avec t ∈ {1, . . . , n}. On a e0 = et + ν0 e−t ou e0 = ν0 et + e−t . On introduit le sous-groupe parabolique Pmin = Mmin Umin de G comme en 4.9. Soit a ∈ Λ− . On a l’égalité I = (I ∩ Umin (F ))(I ∩ P¯min (F )) et a−1 (I ∩ P¯min (F ))a ⊂ I ⊂ Γ. Donc IaΓ = (I ∩ Umin (F ))aΓ. On vérifie que la mesure de cet ensemble est essentiellement bornée par δPmin (a)−1 . Donc Z Z χ(h, N, D, a)  δPmin (a)−1 ΞG (hyaγ)ΞG (yaγ)κN (yaγ)σ(yaγ)D dγ dy. I∩Umin (F )

Γ

1 Il existe un entier c1 ≥ 0 tel que ΓR ⊂ p−c F R. De la définition de la fonction κN résulte que κN (yaγ) ≤ κN +c1 (ya). Alors Z χ(h, N, D, a)  δPmin (a)−1 ΞG (a)σ(a)D ΞG (hya)κN +c1 (ya)dy.

I∩Umin (F )

Grâce au lemme II.1.1 de [16], il existe un réel D1 tel que δPmin (a)−1 ΞG (a)σ(a)D  δPmin (a)−1/2 σ(a)D1 . D’autre part, pour le résultat que l’on veut obtenir, le changement de N en N + c1 est insignifiant. Il nous suffit de majorer X (1) δPmin (a)−1/2 σ(a)D1 Y (h, N, a), a∈Λ−

où

Z

ΞG (hya)κN (ya)dy.

Y (h, N, a) = I∩Umin (F )

Définissons des entiers N (h) et b(h, N, a) par N (h) = sup(N, N − valF (hn )), b(h, N, a) = sup(0, valF (an ) − N (h)). Montrons que (2) il existe ε0 > 0 tel que Z Y (h, N ; a)  exp(−ε0 b(h, N, a)) ΞG (hya)dy I∩Umin (F )

AH min (F ),

−

pour tous h ∈ N ≥ 1, a ∈ Λ . Si valF (an ) ≤ N , il suffit de majorer κN (ya) par 1. Supposons valF (an ) > N . Supposons d’abord que v0 est orthogonal à en et e−n , c’est-à-dire que dan,V = dan,W + 1 ou dan,V = dan,W −1 et t 6= n. Introduisons le sous-groupe de G(F ) formé des éléments

ASTÉRISQUE 346

225

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

2

z u(z), pour z ∈ F , définis ainsi : u(z) envoie v0 sur v0 +zen , e−n sur e−n − 2νz0 v0 − 2ν en , 0 et fixe en ainsi que l’orthogonal du sous-espace engendré par e−n , v0 et en . Il existe un entier c2 tel que u(z) ∈ I ∩ Umin (F ) pour valF (z) ≥ c2 . On a h−1 u(z)h = u(h−1 n z). sup(c ,val (h )+c ) Pour z ∈ pF 2 F n 2 , les deux éléments u(z) et h−1 u(z)h appartiennent à I ∩ Umin (F ). Pour un tel z, on ne modifie pas Y (h, N, a) en remplaçant h par u(z)h. sup(c ,val (h )+c ) On peut ensuite intégrer en z, tout en divisant par mes(pF 2 F n 2 ). On effectue le changement de variable y 7→ h−1 u(−z)hy et on obtient Z (3) Y (h, N, a) = ΞG (hya)κN,h (ya)dy, I∩Umin (F )

où κN,h (ya) =

sup(c ,val (h )+c ) mes(pF 2 F n 2 )−1

Z sup(c2 ,valF (hn )+c2 )

κN (u(−h−1 n z)ya)dz.

pF

−1 −1 −N Supposons κN (u(−h−1 y u(h−1 R, c’est-à-dire n z)ya) = 1. Alors a n z)v0 ∈ p −N −1 −1 −1 a y (v0 + hn zen ) ∈ p R. Il existe c3 tel que valF (qV (e−n , v)) ≥ c3 pour tout v ∈ R. Donc valF (qV (e−n , a−1 y −1 (v0 + h−1 n zen ))) ≥ c3 − N. Puisque y ∈ Umin (F ), on a y −1 en = en , donc

−1 −1 −1 −1 −1 −1 qV (e−n , a−1 y −1 (v0 + h−1 n zen )) = qV (ae−n , y v0 + hn zen ) = an qV (e−n , y v0 ) + an hn z.

En posant z(h, y) = −hn qV (e−n , y −1 v0 ) et c(h, N, a) = valF (an ) + valF (hn ) + c3 − N , c(h,N,a) on a donc z ∈ z(h, y) + pF . Alors sup(c2 ,valF (hn )+c2 ) −1

κN,h (ya) ≤ mes(pF

)

sup(c2 ,valF (hn )+c2 )

mes(pF

sup(c ,val (h )+c )

c(h,N,a)

∩ (z(h, y) + pF

))

c(h,N,a)

≤ inf (1, mes(pF 2 F n 2 )−1 mes(pF )). 0 On vérifie qu’il existe ε > 0 tel que cette dernière expression soit essentiellement bornée par exp(−ε0 b(h, N, a)). Alors (3) entraîne la majoration (2). Supposons maintenant que v0 n’est pas orthogonal aux deux vecteurs en et e−n . C’est-à-dire que dan,V = dan,W − 1 et t = n. On peut écrire v0 = νn en + ν−n e−n . Introduisons le sous-groupe de Amin (F ) formé des éléments a(z), pour z ∈ 1 + pF , définis ainsi : a(z)n = z et a(z)j = 1 pour tout j = 1, . . . , n − 1. Il existe un entier c4 > 0 tel que a(z) ∈ I ∩ K pour tout z ∈ 1 + pcF4 . Un tel a(z) normalise I ∩ Umin (F ), on peut donc effectuer dans Y (h, N, a) le changement de variable y 7→ a(z)−1 ya(z), puis intégrer en z, à condition de diviser par mes(1 + pcF4 ). Puisque a(z) commute à a et h, on obtient Z (4) Y (h, N, a) = ΞG (hya)κ∗N (ya)dy, I∩Umin (F )

où κ∗N (ya)

= mes(1 +

pcF4 )−1

Z c

κN (a(z)−1 ya)dz.

1+pF4

Supposons κN (a(z)−1 ya) = 1. Alors a−1 y −1 a(z)v0 ∈ p−N F R et, comme ci-dessus, valF (qV (e−n , a−1 y −1 a(z)v0 )) ≥ c3 − N.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

226

J.-L. WALDSPURGER

On a qV (e−n , a−1 y −1 a(z)v0 ) = qV (ae−n , y −1 a(z)(ν−n e−n + νn en )) −1 −1 −1 = a−1 (z ν−n e−n + zνn en )) = a−1 ν−n qV (e−n , y −1 e−n ) + a−1 n qV (e−n , y n z n zνn .

Posons z(y) = ν−n νn−1 qV (e−n , y −1 e−n ) et c(N, a) = valF (an ) + c3 − N − valF (νn ). Alors valF (z(y) + z 2 ) ≥ c(N, a). Notons Z (y, N, a) l’ensemble des z qui vérifient cette condition. Alors, comme ci-dessus, κ∗N (ya) ≤ inf (1, mes(1 + pcF4 )−1 mes( Z (y, N, a))). c(N,a)

On a mes( Z (y, N, a)  mes(pF ). Puisque t = n, on a hn = 1 par définition de notre système hyperbolique. On vérifie qu’il existe ε00 > 0 tel que l’expression cidessus soit essentiellement bornée par exp(−ε00 b(h, N, a)). Alors (4) entraîne (2), ce qui achève la preuve de cette relation. Montrons (5) il existe des réels D2 et D3 tels que Z ΞG (hya)dy  σ(h)D2 ΞG (h)σ(a)D3 δPmin (a)1/2 I∩Umin (F )

pour tout h ∈ Amin (F ) et tout a ∈ Λ− . On peut fixer s ∈ S et supposer h ∈ Amin (F )− s . L’élément s détermine un sousgroupe parabolique minimal Pmin,s = Mmin Umin,s formé des éléments de G qui conservent le drapeau F esn ⊂ F esn ⊕ F es(n−1) ⊂ · · · ⊂ F esn ⊕ · · · ⊕ Fs1 . ¯min,s le radical unipotent de P¯min,s . On a l’égalité I ∩ Umin (F ) = (I ∩ On note U ¯min,s (F )). Pour y ∈ I ∩ Umin (F ) ∩ Umin,s (F ), Umin (F ) ∩ Umin,s (F ))(I ∩ Umin (F ) ∩ U on a hyh−1 ∈ I. Donc Z Z G Ξ (hya)dy  ΞG (hya)dy ¯min,s (F ) I∩Umin (F )∩U

I∩Umin (F )

Z  δ0 (h)

ΞG (yha)dy,

¯min,s (F ))h−1 h(I∩Umin (F )∩U

où δ0 (h) est la valeur absolue du déterminant de ad(h−1 ) agissant sur umin (F ) ∩ ¯min,s (F ). Pour v ∈ I ∩ Umin (F ) ∩ Umin,s (F ), on a ΞG (vyha) = ΞG (yha). Donc u Z Z Z ΞG (hya)dy  δ0 (h) ΞG (vyha)dy dv. I∩Umin (F )

I∩Umin (F )∩Umin,s (F )

¯min,s (F ))h−1 h(I∩Umin (F )∩U

Dans le domaine d’intégration, on a σ(vy)  σ(h). Pour tout réel D3 > 0, on a donc Z Z ΞG (hya)dy  δ0 (h) I∩Umin (F )

Z ¯min,s (F ))h−1 h(I∩Umin (F )∩U

ASTÉRISQUE 346

I∩Umin (F )∩Umin,s (F )

ΞG (vyha)σ(h)D3 σ(vy)−D3 dy dv

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

 δ0 (h)σ(h)D3

Z

227

ΞG (uha)σ(u)−D3 du.

Umin (F )

D’après [16], proposition II.4.5, il existe un réel D3 ≥ 0 tel que la dernière intégrale soit convergente et essentiellement bornée par σ(ha)D3 δPmin (ha)1/2 pour tout ha ∈ Amin (F ). Fixons un tel D3 . On obtient Z (6) ΞG (hya)dy  σ(h)2D3 σ(a)D3 δ0 (h)δPmin (ha)1/2 . I∩Umin (F )

On calcule δ0 (h)δPmin (h)1/2 = δPmin,s (h)1/2 . D’après [16], lemme II.1.1, on a δPmin,s (h)1/2  ΞG (h) puisque h ∈ Amin (F )− s . Alors (6) entraîne (5). Grâce à (2) et (5), l’expression (1) est bornée par X σ(h)D2 ΞG (h) σ(a)D1 +D3 exp(−ε0 b(h, N, a)). a∈Λ−

On peut identifier Λ− à l’ensemble M des familles m = (mn , . . . , m1 ) d’entiers telles que mn ≥ mn−1 ≥ · · · ≥ m1 ≥ 0. Pour une telle famille, posons b(h, N, m) = b(h, N, a) où a ∈ Λ− correspond à la famille m. C’est-à-dire que b(h, N, m) = sup(0, mn −N (h)). On vérifie que X exp(−ε0 b(h, N, m)) m∈ M

est convergente et qu’il existe un réel D4 tel que cette expression soit essentiellement majorée par N (h)D4 . De plus N (h)D4  N D4 (1+|valF (hn )|)D4  N D4 σ(h)D4 . Alors l’expression (1) est bornée par N D4 σ(h)D2 +D4 ΞG (h). C’est ce qu’on voulait démontrer.

4.12. Majoration d’une intégrale de fonctions d’Harish-Chandra, cas r > 0. — Soient D un réel, C > 0 un réel et c ≥ 1, N ≥ 1 deux entiers. Posons Z Z χ(c, C, N, D) = 1σ≥C (u)ΞM (m)ΞG (um)κN (m)δP (m)−1/2 σ(u)D σ(m)D du dm. M (F )

U (F )c

Lemme. — Cette expression est convergente. Le réel D étant fixé, pour tout réel R, il existe α > 0 tel que χ(c, C, N, D)  exp(−cR)N −R pour tout c ≥ 1, N ≥ 1 et tout C tel que C ≥ α(log(N ) + c).

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

228

J.-L. WALDSPURGER

Démonstration. — Pour i = 1, . . . , r, on a introduit en 4.5 le sous-groupe parabolique Pi = Mi Ui . Posons Ui0 = Mi+1 ∩ Ui , avec la convention Mr+1 = G. Le groupe U est produit de ces groupes Ui0 . D’où Z Z Z 1σ≥C (ur · · · u1 )ΞM (m)ΞG (ur · · · u1 m) ··· χ(c, C, N, D) = M (F )

U10 (F )∩U (F )c

Ur0 (F )∩U (F )c

κN (m)δP (m)−1/2 σ(ur · · · u1 )D σ(m)D dur · · · du1 dm. Introduisons une fonction b sur {1, . . . , r}, à valeurs réelles strictement positives, que nous préciserons par la suite. Si nous supposons X (1) C≥ b(i), i=1,...,r

la condition 1σ≥C (ur · · · u1 ) = 1 entraîne qu’il existe i tel que 1σ≥b(i) (ui ) = 1. Donc χ(c, C, N, D) est majorée par la somme sur les sous-ensembles non vides J de {1, . . . , r} des χ(c, C, N, D; J), où, dans ce dernier terme, on restreint l’intégration aux ui vérifiant les conditions — si i ∈ J, σ(ui ) ≥ b(i) ; — si i 6∈ J, σ(ui ) < b(i). On peut fixer J et majorer χ(c, C, N, D; J). Notons j le plus petit élément de J. On a Z Z Z Z Z Y ··· (2) χ(c, C, N, D; J)  ( 1σ c1 c pour tout i, et posons b2 (i) = b1 (i) − c1 c. Alors Z Z Z Z Z Y χ(c, C, N, D; {r})  ··· ( 1σ 0 tel que ΞG (ur−1 · · · u1 vum)  exp(c2 σ(ur−1 · · · u1 v))ΞG (um). P Dans le domaine d’intégration, on a σ(ur−1 · · · u1 v)  c + i=1,...,r−1 b(i). Il existe donc c3 > 0 tel que X ΞG (ur−1 · · · u1 vum)  exp(c3 (c + b(i)))ΞG (um). i=1,...,r−1

P Alors χ(c, C, N, D; {r}) est borné par le produit de exp(c3 (c + i=1,...,r−1 b(i))), de l’intégrale Z Z Z Y ··· ( 1σ 0 tel que X (4) χ(c, C, N, D; {r})  exp(c4 (c + b(i)))Z(b2 (r), N, D). i=1,...,r−1

On doit majorer Z(b2 (r), N, D). On commence par changer la variable u en u−1 . D’après [16], lemme II.1.1 et II.3..2, il existe un réel D3 tel que, pour tout g ∈ G(F ), on ait ΞG (g) = ΞG (g −1 )  δP¯ (mP¯ (g −1 ))1/2 ΞM (mP¯ (g −1 ))σ(g)D3 . On applique cette relation à g = u−1 m. On a mP¯ (g −1 ) = m−1 mP¯ (u) et δP¯ (m−1 ) = δP (m), d’où Z Z Z(b2 (r), N, D)  1σ≥b2 (r) (u)ΞM (m)δP¯ (mP¯ (u))1/2 ΞM (m−1 mP¯ (u))κN (m) M (F )

Ur,\ (F )

σ(u)D+D3 σ(m)D+D3 du dm. On décompose M en AG0 . Comme plus haut, on peut majorer l’intégrale sur A(F ) par une puissance de N et on obtient qu’il existe D4 tel que Z Z D4 Z(b2 (r), N, D)  N 1σ≥b2 (r) (u)ΞG0 (x)δP¯ (a(u))1/2 ΞG0 (x−1 g0 (u))κN (x) G0 (F )

Ur,\ (F )

σ(u)D+D3 σ(x)D+D3 du dx, où, pour tout u ∈ Ur,\ (F ), on a écrit mP¯ (u) = a(u)g0 (u), avec a(u) ∈ A(F ) et g0 (u) ∈ G0 (F ). Supposons d’abord r ≥ 2. On remarque qu’alors Ur,\ (F ) est invariant par conjugaison par G0 (F ), a fortiori par K ∩ G0 (F ). Soit k ∈ K ∩ G0 (F ). On peut remplacer la variable u par kuk −1 , puis intégrer en k. Changer u en kuk −1 ne modifie qu’une seule des fonctions que l’on intègre, à savoir ΞG0 (x−1 g0 (u)). On a g0 (kuk −1 ) = kg0 (u)k −1 d’où, puisque ΞG0 est invariante par K ∩ G0 (F ), ΞG0 (x−1 g0 (kuk −1 )) = ΞG0 (x−1 kg0 (u)). Or, d’après [16], lemme II.1.3, l’intégrale de ce terme sur k ∈ K ∩ G0 (F ) est égale à ΞG0 (x−1 )ΞG0 (g0 (u)), ou encore à ΞG0 (x)ΞG0 (g0 (u)). On voit alors que Z(b2 (r), N, D)  N D4 χG0 (1, N, D + D3 )Ir,\ (b2 (r), D + D3 ), avec les notations introduites en 4.5 et 4.11 (l’exposant G0 indiquant que le groupe ambiant est G0 au lieu de G). D’après les lemmes de ces paragraphes, il y a un réel D5 et un réel ε > 0 tel que (5)

Z(b2 (r), N, D)  N D5 exp(−εb2 (r)).

Supposons maintenant r = 1. Dans ce cas, on introduit le sous-espace V] de V orthogonal à D0 et son groupe spécial orthogonal G] . On fixe un sous-groupe compact spécial K] de G] (F ). Le groupe U1,\ est contenu dans le groupe G] . Ecrivons mP¯] (u) =

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

231

a] (u)g0,] (u), avec a] (u) ∈ A(F ) et g0,] (u) ∈ G0 (F ) ∩ G] (F ) = H(F ). Comme dans la preuve du lemme 4.6, les éléments a(u)a] (u)−1 et g0 (u)g0,] (u)−1 restent dans des compacts. En utilisant les relations ΞG0 (x−1 g0 (u))  ΞG0 (x−1 g0,] (u)) = ΞG0 (g0,] (u)−1 x), on voit que l’on a Z Z(b2 (r), N, D)  N D4 1σ≥b2 (r) (u)δP¯ (a(u))1/2 χG0 (g0,] (u)−1 , N, D + D3 )σ(u)D+D3 du. Ur,\ (F )

En utilisant les lemmes 4.11 puis 4.6, on obtient encore une majoration de la forme (5). Les formules (4) et (5) fournissent une majoration de χ(c, C, N, D; {r}). Revenons au terme plus général χ(c, C, N, D; J). On doit remplacer r par j. On doit aussi multiplier la majoration issue de (4) et (5) par cD1 N D2 . Mais le terme cD1 est absorbé par le facteur exp(c4 c) qui figure dans (4), quitte à augmenter c4 . On obtient qu’il existe des réels c5 , D5 , ε, tous strictement positifs, tels que X (6) χ(c, C, N, D; J)  N D5 exp(c5 (c + b(i)) − εb2 (j)). i=1,...,j−1

Soit R un réel. Fixons, indépendamment de c, C et N , une fonction b∗ sur {1, . . . , r}, à valeurs réelles strictement positives. On en déduit comme ci-dessus une fonction b∗1 . Supposons que b∗1 vérifie b∗1 (i) > c1 , X εb∗1 (i) − c5 b∗ (j) > sup(R + c5 + εc1 , R + D5 ) i0 =1,...,i−1

pour tout i. Une telle fonction existe, ces conditions pouvant être imposées par récurrence sur i. Prenons pour fonction b la fonction b(i) = (c + log(N ))b∗ (i). Cette fonction vérifie (3). On a X c5 (c + b(i)) − εb2 (j) < −(R + D5 )log(N ) − Rc. i=1,...,j−1

Alors la majoration (6) devient χ(c, C, N, D; J)  N −R exp(−Rc), c’est-à-dire celle que l’on voulait. La seule condition que l’on a imposée à C est la P condition (1), qui s’écrit C ≥ α(c + log(N )), où α = i=1,...,r b∗ (i). 4.13. Preuve de la relation 4.3(2). — On veut majorer Z I(N, D) = ΞG (g)2 κN (g)σ(g)D dg. G(F )

Par la décomposition usuelle de la mesure dg, on a Z Z Z I(N, D) = ΞG (umk)2 κN (umk)σ(umk)D δP (m)−1 du dm dk. K

M (F )

U (F )

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

232

J.-L. WALDSPURGER

Le k disparaît et l’intégrale sur K également. Puisque κN est invariante à gauche par U (F ), l’intégrale en u est celle notée X(D, m, m) en 4.8. D’après le (iv) du lemme de ce paragraphe, on obtient Z 0 I(N, D)  ΞM (m)2 κN (m)σ(m)D dm, M (F ) 0

pour un réel D convenable. On décompose M en AG0 . Comme dans la preuve précédente, on obtient Z Z 0 0 σ(a)D κN (a)da σ(g0 )D ΞG0 (g0 )2 κN (g0 )dg0 . I(N, D)  A(F )

G0 (F )

La première intégrale est convergente et bornée par une puissance de N . La seconde intégrale n’est autre que χ(1, N, D0 /2), avec la notation de 4.11 appliquée au groupe G0 . Par le lemme de ce paragraphe, elle est convergente et bornée par une puissance de N . D’où la relation 4.3(2). 4.14. Preuve de la relation 4.3(7). — La conclusion que l’on veut obtenir nous autorise à supposer c0 ≥ c. Alors l’ensemble U (F )−U (F )c0 est invariant par translation par U (F )c et on peut décomposer l’intégrale en u0 ∈ U (F )−U (F )c0 en composée d’une intégrale sur U (F )c et d’une intégrale sur (U (F ) − U (F )c0 )/U (F )c . C’est-à-dire Z Z Z Z 0 I(c, c , N, D) = φ(m, h, u, u0 v1 ; D)dv1 du0 du dh dm. M(F ) H(F )U(F )c 0 −1

(U(F )−U(F )c0 )/U(F )c

U(F )c

−1 −1 0

Posons v2 = u u h u v1 h. Le groupe H(F )U (F )c normalise U (F )c et agit trivialement sur U (F )/U (F )c . Quand u décrit U (F )c , v2 décrit le même ensemble. On remplace la variable u par v2 . Le jacobien de cette transformation est 1 et on obtient Z Z Z 0 H D1 D −1 I(c, c , N, D)  Ξ (h)σ(h) κN (m)σ(m) δP (m) M (F )

Z U (F )c

Z

H(F )

(U (F )−U (F )c0 )/U (F )c

ΞG (u0 v1 m)ΞG (u0 v2 h−1 m)σ(u0 v1 )D σ(u0 v2 )D1 dv1 dv2 du0 dh dm

U (F )c

pour un réel D1 convenable. La triple intégrale intérieure est essentiellement X(c, c0 , D, m, h−1 m), avec la notation de 4.8. En appliquant le (iii) du lemme de ce paragraphe, et en remarquant que δP (h) = 1, on obtient Z Z 0 0 I(c, c , N, D)  exp(−εc ) ΞH (h)ΞM (m)ΞM (h−1 m)κN (m)σ(h)D2 σ(m)D2 M (F )

H(F )

pour des réels D2 et ε > 0 convenables. Comme dans le paragraphe précédent, on décompose l’intégrale sur M (F ) en produit d’intégrales sur A(F ) et G0 (F ). L’intégrale sur A(F ) est bornée par une puissance de N . D’après le lemme 4.11, l’intégrale sur G0 (F ) est bornée par exp(−ε0 σ(h))ΞH (h)N D3 pour des réels D3 et ε0 > 0 convenables. Il reste une intégrale Z ΞH (h)2 σ(h)D2 exp(−ε0 σ(h))dh, H(F )

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

233

qui est convergente. Finalement, on a une majoration I(c, c0 , N, D)  exp(−εc0 )N D4 , pour un réel D4 convenable. Soit R un réel. Il existe α > 0 tel que, si c0 ≥ αlog(N ), l’expression ci-dessus est majorée par N −R . C’est ce qu’il fallait démontrer. 4.15. Preuve de la relation 4.3(8). — On a I(c, c0 , N, C, D)  I(sup(c, c0 ), sup(c, c0 ), N, C, D). Puisque c est fixé, on peut aussi bien majorer le membre de droite, autrement dit supposer c = c0 . Introduisons un réel b > 0 que nous préciserons plus tard. On a I(c0 , c0 , N, C, D) = I≥b (c0 , c0 , N, C, D) + I r. On peut alors supposer vi = y−k+r−i pour i = 1, . . . , r et v0 = yr−k + ν0 yk−r , où ν0 = qV (v0 ). Alors UQ¯ ⊂ P et QP est un ouvert de Zariski de G. On a l’égalité Z (5) (e0 (g), e(ghu))dg = Q(F )\G(F )

Z

Z

(e0 (x0 m), e(x0 mhu))dx0 dm.

(M (F )∩Q(F ))\M (F ) (U (F )∩Q(F ))\U (F ) Y0+ = Y + ∩ V0 , Q0 = Q ∩ G0 . L’espace

Posons Y0+ a pour base (yj )j=1,...,k−r et le groupe Q0 est le sous-groupe parabolique des éléments de G0 qui conservent Y0+ . + − Posons YW = Y + ∩ W , YW = Y − ∩ W , notons QH le sous-groupe parabolique de + H formé des éléments qui conservent YW , LH son sous-groupe de Levi formé des − + éléments qui conservent de plus YW et UQH le radical unipotent de QH . L’espace YW + − a pour base (yj )j=1,...,k−r−1 . Notons W0 l’orthogonal de YW ⊕ YW dans W , H0 son

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

240

J.-L. WALDSPURGER

+ groupe spécial orthogonal et GLk−r−1 le groupe des automorphismes linéaires de YW . 1 1 On a l’égalité LH = GLk−r−1 × H0 . Posons w0 = − 2ν0 yr−k + 2 yk−r et DH = F w0 . Alors W0 est la somme directe orthogonale de V˜ et de DH . Le groupe Q0 ∩ H est le sous-groupe des éléments de H qui conservent Y0+ (disons que, pour quelques instants, + on étend les scalaires à F¯ ). Un tel élément conserve Y0+ ∩ W = YW , donc appartient + à QH . Soit h ∈ QH . Pour qu’il conserve Y0 , il doit envoyer yk−r dans Y0+ . Mais il fixe v0 et on a w0 = − 2ν10 v0 + yk−r . On a donc hw0 ∈ w0 + Y0+ . Puisque hw0 et + w0 appartiennent à W , cela force hw0 ∈ w0 + YW . La réciproque est similaire. Donc + H Q0 ∩ H est le sous-groupe des h ∈ Q tels que hw0 ∈ w0 + YW . Autrement dit

˜ QH . Q0 ∩ H = GLk−r−1 GU On vérifie que la restriction du module δQ0 au groupe Q0 (F ) ∩ H(F ) est égale au module de ce groupe : si on écrit un élément h ∈ Q0 (F ) ∩ H(F ) sous la forme ˜ ), n ∈ UQH (F ), ces modules coïncident avec δ˜ g n, avec δ ∈ GLk−r−1 (F ), g˜ ∈ G(F dV˜ +k−r−1 |d´et(δ)|F . L’application naturelle Q0 (F )\G0 (F ) → (M (F ) ∩ Q(F ))\M (F ) est un isomorphisme. Le groupe H(F ) agit sur l’ensemble des sous-espaces isotropes de V0 de dimension k − r. Il y a deux orbites : l’orbite ouverte des sous-espaces dont l’intersection avec W est de dimension k − r − 1 et l’orbite fermée des sous-espaces contenus dans W . L’espace Y0+ appartient à l’orbite ouverte. Il en résulte que l’application naturelle (Q0 (F ) ∩ H(F ))\H(F ) → Q0 (F )\G0 (F ) est injective et a pour image un ouvert de l’espace d’arrivée dont le complémentaire est de mesure nulle. On en déduit aisément l’assertion suivante. Soit ϕ : G0 (F ) → C une fonction telle que ϕ(qg) = δQ0 (q)ϕ(g) pour tous q ∈ Q0 (F ), g ∈ G0 (F ). Supposons ϕ absolument intégrable sur Q0 (F )\G0 (F ). Alors la restriction de ϕ à H(F ) est absolument intégrable sur (Q0 (F ) ∩ H(F ))\H(F ) et on a l’égalité Z Z ϕ(g)dg = ϕ(h)dh. Q0 (F )\G0 (F )

(Q0 (F )∩H(F ))\H(F )

Dans l’égalité (5), on peut donc remplacer l’intégration sur (M (F ) ∩ Q(F ))\M (F ) par une intégration sur (Q0 (F ) ∩ H(F ))\H(F ). On obtient Z Z Z (e0 (g), e(ghu))dg = (e0 (x0 h0 ), e(x0 h0 hu))dx0 dh0 , Q(F )\G(F )

(Q0 (F )∩H(F )\H(F ) (U(F )∩Q(F ))\U(F )

puis

L π,ρ (ε0 ⊗ e0 , ε ⊗ e) =

Z H(F )U (F )c

Z

Z (Q0 (F )∩H(F ))\H(F )

0 ¯ (ρ(h)ε0 , ε)(e0 (x0 h0 ), e(x0 h0 hu))ξ(u)dx dh0 du dh.

(U (F )∩Q(F ))\U (F )

On effectue les changements de variables u 7→ (h0 h)−1 uh0 h, puis h 7→ h0 −1 h, on décompose ensuite l’intégrale sur H(F ) en une intégrale composée d’une intégrale sur Q0 (F ) ∩ H(F ) et d’une intégrale sur (Q0 ∩ H(F ))\H(F ) (la mesure sur Q0 (F ) ∩ H(F )

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

doit être une mesure de Haar à gauche). On obtient Z Z 0 0 L π,ρ (ε ⊗ e , ε ⊗ e) = ((Q0 (F )∩H(F ))\H(F ))2

Z

241

Z

Q0 (F )∩H(F )

(U (F )∩Q(F ))\U (F )

¯ (ρ(qh)ε0 , ρ(h0 )ε)(e0 (x0 h0 ), e(x0 uqh))ξ(u)du dx0 dq dh dh0 ,

U (F )c

On effectue le changement de variable u 7→ x0 −1 u puis on décompose l’intégrale en u ∈ U (F ) en composée d’une intégrale sur u ∈ U (F ) ∩ Q(F ) et d’une intégrale sur x ∈ (U (F ) ∩ Q(F ))\U (F ). Remarquons que U (F ) ∩ Q(F ) = U (F ) ∩ L(F ) = U (F ) ∩ GLk (F ). La condition initiale u ∈ U (F )c est remplacée par x0 −1 ux ∈ U (F )c . Notons ϕc (u, x, x0 ) la fonction caractéristique de l’ensemble des (u, x, x0 ) vérifiant cette condition. On obtient Z Z Z L π,ρ (ε0 ⊗ e0 , ε ⊗ e) = ((Q0 (F )∩H(F ))\H(F ))2

Z

Q0 (F )∩H(F )

((U (F )∩Q(F ))\U (F ))2

¯ 0 −1 ux)du dx dx0 dq dh dh0 . (ρ(qh)ε0 , ρ(h0 )ε)(e0 (x0 h0 ), e(uxqh))ϕc (u, x, x0 )ξ(x

U (F )∩GLk (F )

On effectue le changement de variable x 7→ qxq −1 : cette conjugaison préserve à la ¯ fois U (F ) et U (F ) ∩ Q(F ). Les termes ξ(x) et ϕc (u, x, x0 ) ne dépendent de x que par l’intermédiaire des coefficients qV (xvi , v−i−1 )pour i = 1, . . . , r − 1. Puisque q fixe les vecteurs vi , la conjugaison par q ne change pas ces termes. Par contre, elle introduit un module. Pour l’exprimer commodément et pour poursuivre notre calcul, ˜ ). Le module on décompose q en δn˜ g , où δ ∈ GLk−r−1 (F ), n ∈ UQH (F ) et g˜ ∈ G(F −r est alors |d´et(δ)|F . On obtient Z Z Z Z Z 0 0 L π,ρ (ε ⊗ e , ε ⊗ e) = ((Q0 (F )∩H(F ))\H(F ))2

Z

((U (F )∩Q(F ))\U (F ))2

˜ ) G(F

UQH (F )

GLk−r−1 (F )

(ρ(δn˜ g h)ε0 , ρ(h0 )ε)(e0 (x0 h0 ), e(uδn˜ g xh))ϕc (u, x, x0 )

U (F )∩GLk (F )

¯ 0 −1 ux)|d´et(δ)|−r du dδ dn d˜ ξ(x g dx dx0 dh dh0 . F On a l’égalité U (F ) = (U (F ) ∩ Q(F )) × (U (F ) ∩ UQ¯ (F )) qui permet de remplacer l’ intégration sur (U (F ) ∩ Q(F ))\U (F ) par une intégration sur U (F ) ∩ UQ¯ (F ). On va légèrement modifier cet ensemble de représentants. Soit x ∈ U (F ) ∩ UQ¯ (F ). Pour i = 1, . . . , r − 1, on a qV (xvi , v−i−1 ) = qV (xy−k+r−i , yk−r+i+1 ) = qV (y−k+r−i , yk−r+i+1 ) = 0 puisque x fixe y−k+r−i . Par contre, qV (xv0 , v−1 ) n’est en général pas nul. Exprimons matriciellement les éléments de GLk dans la base (yj )j=1,...,k de Y + . Remarquons que le groupe U ∩ GLk est le radical unipotent du sous-groupe parabolique de GLk triangulaire supérieur par blocs, de blocs k − r, 1,. . .,1. Pour u ∈ U (F ) ∩ GLk (F ), on calcule aisément qV (uvi , v−i−1 ) = −uk−r+i,k−r+i+1

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

242

J.-L. WALDSPURGER

pour i = 0, . . . , r − 1. Pour x ∈ U (F ) ∩ UQ¯ (F ), notons u(x) l’élément de U (F ) ∩ GLk (F ) dont toutes les coordonnées non diagonales sont nulles, sauf u(x)k−r,k−r+1 qui vaut qV (xv0 , v−1 ). Posons x∗ = u(x)x. Alors qV (x∗ vi , v−i−1 ) = 0 pour tout i = 0, . . . , r − 1 et {x∗ ; x ∈ U (F ) ∩ UQ¯ (F )} est encore un ensemble de représentants de (U (F ) ∩ Q(F ))\U (F ). Pour x, x0 ∈ U (F ) ∩ UQ¯ (F ) et u ∈ U (F ) ∩ GLk (F ), on a ¯ 0∗−1 ux∗ ) = ξ(u) ¯ ξ(x et ϕc (u, x∗ , x0∗ ) = 1 si et seulement si u ∈ Uk (F )c . On obtient Z Z 0 0 (6) L π,ρ (ε ⊗ e , ε ⊗ e) = ((Q0 (F )∩H(F ))\H(F ))2 0

0

0

(U (F )∩UQ ¯ (F ))2

0

I(x , h , x, h)dx dx dh dh , où 0

0

Z

Z

Z

Z

(ρ(δn˜ g h)ε0 , ρ(h0 )ε)

I(x , h , x, h) = ˜ ) G(F

UQH (F )

GLk−r−1 (F )

U (F )∩Uk (F )c

¯ (e0 (x0∗ h0 ), e(uδn˜ g x∗ h))ξ(u)|d´ et(δ)|−r g. F du dδ dn d˜ Toutes ces expressions sont absolument convergentes d’après (1) : on n’a jusqu’ici effectué que des changements de variables et des permutations d’intégrales. On va calculer I(x0 , h0 , x, h). Fixons un sous-groupe compact spécial K H de H(F ) en bonne position relativement au sous-groupe parabolique QH . L’application naturelle de K H dans QH (F )\H(F ) est surjective. D’après la description que l’on a donnée ci-dessus du groupe Q0 ∩ H, tout élément de(Q0 (F ) ∩ H(F ))\H(F ) a un représentant qui appartient à H0 (F )K H . On peut donc se limiter à calculer I(x0 , h0 , x, h) pour des éléments x, x0 ∈ U (F ) ∩ UQ¯ (F ) et h, h0 ∈ H0 (F )K H . Dans l’expression de I(x0 , h0 , x, h), on peut remplacer e(uδn˜ g x∗ h) par µ(u)e(δn˜ g x∗ h). D’après les formules P ¯ écrites ci-dessus, ξ(u) = ψ(− j=k−r,...,k−1 uj,j+1 ). Fixons une fonctionnelle de Whittaker φ non nulle sur Eµ et notons comme plus haut Φ : Eµ ⊗C Eπ˜ → Eπ˜ l’application φ ⊗ id. On peut appliquer le lemme 3.7(ii) : il existe une constante C 6= 0 telle que Z ¯ (e0 (x0∗ h0 ), µ(u)e(δn˜ g x∗ h))ξ(u)du = U (F )∩Uk (F )c

Z

(Φµ(γa)e0 (x0∗ h0 ), Φµ(γa)e(δn˜ g x∗ h))|d´et(γ)|−r F da dγ,

C Uk−r−1 (F )\GLk−r−1 (F )×ω[k−r] (c+cψ )

pourvu que c + cψ ≥ 1. On peut remplacer µ(γa)e(δn˜ g x∗ h) par (d +k−1)/2

µ(γδa)e(n˜ g x∗ h)δQ (δ)1/2 = µ(γδa)e(n˜ g x∗ h)|d´et(δ)|F V˜ On obtient I(x0 , h0 , x, h) = C

Z ˜ ) G(F

Z

Z

Z UQH (F )

GLk−r−1 (F )

(ρ(δn˜ g h)ε0 , ρ(h0 )ε)(Φµ(γa)e0 (x0∗ h0 ), Φµ(γδa)e(n˜ g x∗ h))

Uk−r−1 (F )\GLk−r−1 (F )×ω[k−r] (c+cψ ) −r+(dV˜ +k−1)/2

|d´et(γ)|−r et(δ)|F F |d´ Montrons que

ASTÉRISQUE 346

.

da dγ dδ dn d˜ g.

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

243

(7) pour g˜ et n fixés, l’intégrale intérieure sur GLk−r−1 (F ) × (Uk−r−1 (F )\GLk−r−1 (F )) × ω[k−r] (c + cψ ) est absolument convergente. La variable a ∈ ω[k−r] (c + cψ ) disparaît tout de suite : la fonction que l’on intègre est localement constante en cette variable et le domaine d’intégration est compact. On a une majoration |(ρ(δn˜ g h)ε0 , ρ(h0 )ε)|  ΞH (δ). On peut décomposer e0 (x0∗ h0 ) et e(n˜ g x∗ h) en combinaisons linéaires de produits η ⊗ e˜, où η ∈ Eµ et e˜ ∈ Eπ˜ . Cela nous ramène à montrer que, pour η, η 0 ∈ Eµ , l’intégrale Z Z ΞH (δ)|φµ(γ)η 0 ||φµ(γδ)η| GLk−r−1 (F )

Uk−r−1 (F )\GLk−r−1 (F ) −r+(dV˜ +k−1)/2

|d´et(γ)|−r et(δ)|F F |d´

dγ dδ

est convergente. On effectue le changement de variable δ 7→ γ −1 δ. On remplace ensuite la variable γ par t0 k 0 , avec t0 ∈ Ak−r−1 (F ) et k 0 ∈ Kk−r−1 et δ par tuk, avec t ∈ Ak−r−1 (F ), u ∈ Uk−r−1 (F ), k ∈ Kk−r−1 . Cela remplace dγ dδ par δBk−r−1 (t0 )−1 dt0 dk 0 dt du dk. De nouveau, les intégrales en k et k 0 sont inessentielles et on est ramené à l’intégrale Z Z (8) ΞH (t0 −1 tu)|φµ(t0 )η 0 ||φµ(t)η|δBk−r−1 (t0 )−1 Uk−r−1 (F )

Ak−r−1 (F )2 (1−dV˜ −k)/2

|d´et(t0 )|F

−r+(dV˜ +k−1)/2

|d´et(t)|F

dt dt0 du.

Montrons que (9) pour tous réels R > 0 et ε avec 0 < ε < 1/2, on a une majoration ε+(r+1−dV˜ −k)/2

ΞH (g)  ΞGLk−r−1 (g)σ(g)−R |d´et(g)|F

pour tout g ∈ GLk−r−1 (F ). On peut supposer g = a ∈ Ak−r−1 (F ). Notons aj , pour j = 1, . . . , k − r − 1, les coefficients diagonaux de a. Choisissons un sous-groupe de Levi minimal Mmin de H contenant Ak−r−1 et un sous-groupe parabolique minimal Pmin ∈ P (Mmin ) tel que a soit « négatif » pour Pmin , c’est-à-dire que |α(a)| ≤ 1 pour toute racine α de AMmin dans uPmin . D’après [16] lemme II.1.1, on a des inégalités (10)

δPmin (a)  ΞH (a)2  δPmin (a)σ(a)D

où D est un certain entier. On énumère les valeurs de α(a) pour toutes les racines 0 −1 α de AMmin dans h : ce sont aj a−1 pour j < j 0 , j 0 pour j 6= j , aj aj 0 et (aj aj 0 ) −1 qui interviennent avec multiplicité 1, et aj et aj qui interviennent avec multiplicité dW0 = dV˜ +1 (évidemment, les j, j 0 parcourent {1, . . . , k −r −1}). Le module δPmin (a) est le produit de celles des valeurs absolues de ces termes qui sont inférieures ou égales à 1. Donc δPmin (a) = I1 I2 I3 ,

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

244

J.-L. WALDSPURGER

où Y

I1 =

|aj a−1 j 0 |F ;

j6=j 0 ;valF (aj )≥valF (aj 0 )

Y

I2 = (

j bi + l. Si le plus petit des entiers ci-dessus est valF (xj ) pour un j tel que i + 1 ≤ j ≤ 0 n, on introduit l’élément k = exp(c(v−j , vi0 )). Il appartient à K 0 . On a a−1 ka = −1 0 −1 exp(aj ai c(v−j , vi0 )). Puisque j > i, valF (aj a−1 ka ∈ K 0 et k ∈ i ) ≥ 0, donc a K 0 ∩ aK 0 a−1 . Posons y = kx. Les coordonnées de y sont les mêmes que celles de x, sauf yi qui vaut xi + xj . Alors valF (yi ) = bi et on conclut en appliquant encore un élément convenable de Amin (F ) ∩ K 0 ∩ aK 0 a−1 . Si le plus petit des entiers ci-dessus est valF (xj ) − αj + αi , pour un j tel que 1 ≤ j ≤ i − 1, on introduit l’élément 0 0 k = exp(ai a−1 j c(v−j , vi )) et on pose y = kx. On vérifie de même que k appartient 0 0 −1 à K ∩ aK a et que les coordonnées de y sont les mêmes que celles de x, sauf yi qui vérifie valF (yi ) = bi . On conclut comme précédemment. Reste le cas où le plus petit des entiers ci-dessus est αi . On peut même supposer que tous les autres sont strictement plus grands. En particulier valF (xn ) > αn ≥ 0. D’après la définition de E, cela entraîne ι = 1. On introduit alors l’élément k = exp(c(e, ai vi0 )) et on pose y = kx. On vérifie encore que k ∈ K 0 ∩ aK 0 a−1 et que y a les mêmes coordonnées que x, sauf yi qui vérifie valF (yi ) = bi + valF (2). On conclut comme précédemment. Cela prouve (7). P Notons X 00 l’ensemble des x ∈ X 0 de la forme x = e + i=1,...,n xi vi si ι = 1, P x = x−1 i=1,...,n xi vi si ι = 2, où e ∈ E dans le premier cas et où, dans les 1 v−1 + deux cas et pour tout i = 1, . . . , n, xi = $Fci , avec 0 ≤ ci ≤ (n+ 1)l. C’est un ensemble fini. Pour tout x ∈ X 00 , fixons γx ∈ G(F ) tel que γx−1 v0 = x. Posons Γ = {γx ; x ∈ X 00 }. Pour b = (b1 , . . . , bn ) ∈ B, introduisons l’élément a(b) ∈ Amin (F ) tel que a(b)i = $Fbi pour tout i = 1, . . . , n. Un élément y vérifiant la conclusion de l’assertion (7) est de la forme a(b)x00 pour un x00 ∈ X 00 , autrement dit de la forme a(b)γ −1 v0 pour un γ ∈ Γ. On a donc (8) pour tout x ∈ X(b), il existe k 0 ∈ K 0 ∩ aK 0 a−1 et γ ∈ Γ tels que k 0 x = a(b)γ −1 v0 . Pour tout x ∈ X 00 , notons H x le sous-groupe des éléments de G qui fixent x. Introduisons le sous-groupe parabolique Pmin = Mmin Umin ∈ P (Mmin ) formé des

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

258

J.-L. WALDSPURGER

éléments qui conservent le drapeau de sous-espaces F vn ⊂ F vn ⊕ F vn−1 ⊂ · · · ⊂ F vn ⊕ · · · ⊕ F v1 . Introduisons aussi le sous-groupe parabolique P 0 = M 0 U 0 ∈ F (Mmin ) formé des éléments qui conservent le drapeau des n + 1 − ι premiers sous-espaces ci-dessus. On a P 0 = Pmin si ι = 1 ou si Van = {0}. Montrons que (9) pour tout x ∈ X 00 , l’application (P 0 (F ) ∩ H x (F )) × P¯min (F ) → G(F ) (p0 , p¯)

7→

p0 p¯

est submersive à l’origine. Il suffit de prouver l’égalité ¯min (F ) = g(F ). (p0 (F ) ∩ hx (F )) + p Notons hx,⊥ l’orthogonal de hx dans g pour la forme (X, Y ) 7→ trace(XY )/2 et notons ¯min le radical unipotent de P¯min . Il suffit encore de prouver l’égalité U ¯min (F ) = {0}. (u0 (F ) + hx,⊥ (F )) ∩ u Notons W x l’orthogonal de x dans V . Tout élément de hx,⊥ (F ) est de la forme c(x, y), pour un y ∈ W x . On doit donc prouver que, pour y ∈ W x , N 0 ∈ u0 (F ), ¯ ∈ u ¯ entraîne N ¯ = 0. Considérons de tels élé¯min (F ), l’égalité N 0 + c(x, y) = N N ments. Soit i = ι, . . . , n. On a c(x, y)vi = qV (vi , x)y − qV (vi , y)x = −qV (vi , y)x. ¯ vi , v−i ) = 0. Donc qV (c(x, y)vi , v−i ) = 0, c’est-à-dire On a qV (N 0 vi , v−i ) = qV (N −xi qV (vi , y) = 0. Puisque xi 6= 0, on a qV (vi , y) = 0, donc c(x, y)vi = 0. Alors ¯ vi . Ces éléments appartiennent à des sous-espaces de V d’intersection N 0 vi = N ¯ vi = 0. Si ι = 1, ou si Van = {0}, cela suffit pour conclure : un nulle, donc N ¯(F ) qui annule tous les vi , pour i = ι, . . . , n, est nul. Supposons ι = 2 et élément de u ¯ annule Van . Soit van ∈ Van . On a c(x, y)van = Van 6= {0}. Il faut montrer de plus que N ¯ van , v−1 ) = 0, qV (van , x)y −qV (van , y)x = −qV (van , y)x. On a qV (N 0 van , v−1 ) = qV (N donc qV (c(x, y)van , v−1 ) = 0, c’est-à-dire −x1 qV (van , y) = 0. Puisque x1 6= 0, on a ¯ van . Ces éléments appartiennent qV (van , y) = 0, donc c(x, y)van = 0. Alors N 0 van = N ¯ van = 0. Cela démontre encore à des sous-espaces de V d’intersection nulle, donc N (9). Après ces préparatifs, passons à la majoration de IK (ε, e, a). Pour tout k ∈ K, on a k −1 v0 ∈ X. Pour b ∈ B, posons K(b) = {k ∈ K; k −1 v0 ∈ X(b)}. On a l’égalité X IK(b) (ε, e, a), IK (ε, e, a) = b∈ B

où

Z |(ε, l(π(ka)e))|dk.

IK(b) (ε, e, a) = K(b)

Fixons b = (b1 , . . . , bn ) ∈ B. Soit k ∈ K(b), appliquons (8) à x = k −1 v0 . Il y a k 0 ∈ K 0 ∩aK 0 a−1 et γ ∈ Γ tels que k 0 k −1 v0 = a(b)γ −1 v0 . Fixons de tels éléments et posons h = kk 0 −1 a(b)γ −1 . Alors hv0 = v0 , c’est-à-dire h ∈ H(F ). On a k = hγa(b)−1 k 0 ,

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

259

donc π(ka)e = π(hγa(b)−1 a)e0 , où e0 = π(a−1 k 0 a)e. Notons x l’élément de X 00 tel que γ = γx . Montrons que (10) il existe des sous-groupes ouverts compacts K 1 ⊂ P 0 (F ) ∩ H x (F ) et K 2 ⊂ ¯ P (F ), indépendants des variables a, b et k, tels que la fonction (k1 , k2 ) 7→ (ε, l(π(hγk1 k2 a(b)−1 a)e0 )) soit constante sur K 1 × K 2 . Puisque a−1 k 0 a ∈ K 0 , le vecteur e0 appartient à un ensemble fini indépendant 0 des variables. D’autre part, par définition de B , on a a(b)−1 a ∈ Amin (F )− . La −1 conjugaison par a(b)a contracte P¯ . Il existe donc K 2 comme ci-dessus tel que π(k2 a(b)−1 a)e0 = π(a(b)−1 a)e0 pour tout k2 ∈ K 2 . Considérons l’application k1 7→ hγk1 γ −1 h−1 sur P 0 (F ) ∩ H x (F ). Elle prend ses valeurs dans H(F ) car γH x γ −1 = H. D’autre part, elle est « bornée » en un sens facile à préciser. En effet, on a hγ = kk 0 −1 a(b) ; les racines de a(b) dans p0 sont bornées par définition de B et k et k 0 appartiennent à des compacts. Il existe donc K 1 comme dans l’énoncé tel que ρ(hγk1−1 γ −1 h−1 )ε = ε pour tout k1 ∈ K 1 . D’où (10). Grâce à (9) et (10), et à la finitude de Γ, il existe un sous-groupe ouvert compact K 00 de G(F ), indépendant des variables, tel que la fonction k 00 7→ (ε, l(π(hγk 00 a(b)−1 a)e0 )) soit constante sur K 00 . Fixons un tel K 00 . On a alors (ε, l(π(ka)e)) = (ε, l(π(hγa(b)−1 a)e0 )) = (ε, l(π(hγ)e00 )), où e00 = mes(K 00 )−1

Z

π(k 00 a(b)−1 a)e0 dk 00 .

K 00 K 00

Fixons une base orthonormée (ej )j=1,...,k du sous-espace Eπ . On a X e00 = (ej , e00 )ej , j=1,...,k

d’où (ε, l(π(ka)e)) =

X

(ε, l(π(hγ)ej ))(ej , e00 ).

j=1,...,k

Pour tout j, on a (ej , e ) = (ej , π(a(b)−1 a(a−1 k 0 a))e). Rappelons que a−1 k 0 a ∈ K 0 . D’où une majoration |(ej , e00 )|  ΞG (a(b)−1 a). 00

On a aussi |(ε, l(π(hγ)ej ))| = |(ρ(h−1 )ε, l(π(γ)ej ))|  ΞH (h). Rappelons que h = kk 0 −1 a(b)γ −1 et que l’on a noté x l’élément de X 00 tel que γ = γx . P Ecrivons x = y + i=ι,...,n xi vi avec y ∈ E si ι = 1, y = x−1 1 v−1 + x1 v1 si ι = 2. On a x = exp(c(y, (y − x)/2))y. Posons γ0 = γexp(c(y, (x − y)/2)). Alors γ0−1 v0 = y.

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

260

J.-L. WALDSPURGER

Notons H y le sous-groupe des éléments de G qui fixent y. On a γ0−1 Hγ0 = H y et une majoration y ΞH (h)  ΞH (γ0−1 hγ0 ). On a γ0−1 hγ0 = γ0−1 kk 0 −1 a(b)exp(c(y, (x − y)/2)). Introduisons l’élément a(b)0 ∈ Amin (F ) tel que — si ι = 1, a(b)0i = $Fbi +nl pour i = 1, . . . , n ; — si ι = 2, a(b)0i = $Fbi −b1 pour i = 1, . . . , n. On a γ0−1 hγ0 = k1 a(b)00 u0 a(b)0 , où k1 = γ0 kk 0 −1 , a(b)00 = a(b)a(b)0 −1 , u0 = a(b)0 exp(c(y, (x − y)/2))a(b)0 −1 . Remarquons que a(b)0 appartient à H y (F ), donc aussi k1 a(b)00 u0 ∈ H y (F ). L’élément k1 reste dans un compact. D’après la définition de B, l’élément a(b)00 reste lui-aussi dans un compact. Enfin c(y, (x − y)/2) appartient à u0 (F ) et la conjugaison par a(b)0 contracte cet ensemble. Donc u0 reste dans un compact. On en déduit une majoration y

y

ΞH (γ0−1 hγ0 )  ΞH (a(b)0 ), puis y

|(ε, l(π(ka)e))|  ΞH (a(b)0 )ΞG (a(b)−1 a). Les éléments a(b)−1 a et a(b)0 sont « négatifs » pour Pmin , resp. P 0 ∩ H y . D’après [16] lemme II.1.1, il existe un réel D tel que le membre de droite ci-dessus soit essentiellement borné par δP 0 ∩H y (a(b)0 )1/2 δPmin (a(b)−1 a)1/2 σ(a(b)0 )D σ(a(b)a)D . Les coefficients de a(b)0 sont essentiellement les mêmes que ceux de a(b). En calculant explicitement l’expression ci-dessus, on obtient la majoration P b /2 |(ε, l(π(ka)e))|  q i=1,...,n i σ(a(b))D ΞG (a)σ(a)D . L’application k 7→ k −1 v0 de (K ∩ H(F ))\K(b) dans X(b) est injective et préserve les mesures. D’après (4), on a donc P − b i=1,...,n i , mes(K(b))  q puis IK(b) (ε, e, a)  q

−

P i=1,...,n

bi /2

σ(a(b))D ΞG (a)σ(a)D ,

et enfin IK (ε, e, a)  ΞG (a)σ(a)D

X

q

−

P i=1,...,n

bi /2

σ(a(b))D .

b∈ B

L’ensemble B dépend de a mais est contenu dans l’ensemble des (b1 , . . . , bn ) ∈ Zn tels que bi ≥ −nl pour tout i. On peut remplacer B par cet ensemble, la série ci-dessus y est convergente et on obtient la majoration IK (ε, e, a)  ΞG (a)σ(a)D que l’on voulait démontrer. Cela achève la preuve.

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

261

5.6. Le cas r = 0 : tout entrelacement est tempéré. — Soient (V, qV ) et (W, qW ) deux espaces quadratiques compatibles. Soient π ∈ Temp(G) et ρ ∈ Temp(H). Proposition. — Supposons dV = dW + 1. Alors m(π, ρ) = 1 si et seulement si L π,ρ n’est pas nulle. Démonstration. — Pour une raison qui va apparaître, modifions la notation en notant π 0 plutôt que π la représentation de G(F ). Un sens de l’équivalence (« si ») est clair d’après 5.1. On doit prouver que, si HomH,ξ (π 0 , ρ) n’est pas nul, L π0 ,ρ ne l’est pas non plus. Puisque π 0 est tempérée, on peut fixer des données comme en 5.3, avec de plus π ˜ , µ1 ,. . .,µs de la série discrète, de sorte que π 0 soit une sous-représentation de la G 0 représentation induite π = π0 . On peut supposer que Eπ0 ⊂ K G Q,τ . Soient e, e ∈ K Q,τ ∗ ∞ et ϕ une fonction C sur i A L,F . Comme en 1.6, définissons une fonction f = fe,e0 ,ϕ sur G(F ) par Z ϕ(λ)(πλ (g)e0 , e)m(τλ )dλ. f (g) = i A∗ L,F

Elle appartient à l’espace de Schwartz-Harish-Chandra S (G(F )) et agit donc dans π 0 . Par définition de cette action, on a Z (e00 , π 0 (f )e0 ) = (e00 , π 0 (g)e0 )f (g)dg G(F )

e0 , e00

pour tous ∈ Eπ0 . Soit l ∈ HomH,ξ (π 0 , ρ), supposons l 6= 0. Soient e0 ∈ Eπ0 et ε ∈ Eρ . Posons Z I(ε, e0 , f ) = (ε, l(π(g)e0 ))f (g)dg. G(F )

Grâce à la proposition 5.5, cette intégrale est absolument convergente. On peut la calculer de deux façons. La première consiste à fixer un sous-groupe ouvert compact Kf de G(F ) tel que f soit biinvariante par Kf et une suite (Ωn )n≥1 de sous-ensembles ouverts compacts de G(F ) biinvariants par Kf , telle que [ Ωn ⊂ Ωn+1 , et Ωn = G(F ). n≥1

Alors Z I(ε, e0 , f ) = limn→∞

(ε, l(π 0 (g)e0 ))f (g)dg.

Ωn

Considérons cette dernière intégrale. Puisqu’elle est limitée à un compact, on a Z (ε, l(π 0 (g)e0 ))f (g)dg = (ε, l(en )), Ωn

où Z en =

π 0 (g)e0 f (g)dg.

Ωn

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

262

J.-L. WALDSPURGER

K

Ces vecteurs restent dans le sous-espace de dimension finie Eπ0f . De plus, dans ce sous-espace, limn→∞ en = π 0 (f )e0 . On en déduit limn→∞ (ε, l(en )) = (ε, l(π 0 (f )e0 )), puis (1)

I(ε, e0 , f ) = (ε, l(π 0 (f )e0 )).

D’autre part, on a Z I(ε, e0 , f, g)dg,

I(ε, e0 , f ) = H(F )\G(F )

où

Z

(ε, l(π 0 (hg)e0 ))f (hg)dh

I(ε, e0 , f, g) = H(F )

Z

Z

0

=

(ε, l(π (hg)e0 )) H(F )

i A∗ L,F

ϕ(λ)(πλ (hg)e0 , e)m(τλ )dλ dh.

Fixons g. La dernière intégrale est absolument convergente. En effet, quand on remplace tous les termes par leurs valeurs absolues, il existe un entier D tel que l’intégrale intérieure soit  ΞG (h)σ(h)D . Le premier terme est  ΞH (h) et l’assertion résulte de 4.3(4). On permute les deux intégrales, on utilise l’égalité (ε, l(π 0 (hg)e0 )) = (ρ(h)−1 ε, l(π(g)e0 )) et on change h en h−1 . On obtient Z Z ϕ(λ)m(τλ ) (ρ(h)ε, l(π 0 (g)e0 ))(πλ (g)e0 , πλ (h)e)dh dλ. I(ε, e0 , f, g) = i A∗ L,F

H(F )

On reconnaît l’intégrale intérieure : c’est L πλ ,ρ (ε ⊗ πλ (g)e0 , l(π 0 (g)e0 ) ⊗ e). On obtient Z Z ϕ(λ)m(τλ ) L πλ ,ρ (ε ⊗ πλ (g)e0 , l(π 0 (g)e0 ) ⊗ e)dλ dg. (2) I(ε, e0 , f ) = H(F )\G(F )

i A∗ L,F

Fixons un voisinage ω de 0 dans i A ∗L,F tel que la relation 1.6(1) soit vérifiée. Fixons ε ∈ Eρ et e0 ∈ Eπ0 tels que (ε, l(e0 )) 6= 0. Appliquons les constructions cidessus à e = e0 = e0 et à une fonction ϕ à support dans ω telle que ϕ(0) 6= 0. La relation 1.6(1) nous dit que π 0 (f )(e0 ) est un multiple non nul de e0 . D’après (1), on a donc I(ε, e0 , f ) 6= 0. Alors (2) implique qu’il existe λ tel que L πλ ,ρ ne soit pas nul. D’après le lemme 5.3(ii), L π,ρ n’est pas nul. Si π est irréductible, on a π 0 = π et c’est terminé. Sinon, d’après le lemme 5.4, il y a une sous-représentation irréductible π 00 de π telle que L π00 ,ρ ne soit pas nul. D’après 5.1(2) on peut fixer e1 ∈ Eπ00 ⊂ K G Q,τ et ε1 ∈ Eρ de sorte que L π,ρ (ε1 ⊗ e1 , ε1 ⊗ e1 ) 6= 0. Notons c la valeur non nulle de ce terme. D’après le lemme 5.3(i), quitte à restreindre ω, on peut supposer que | L πλ ,ρ (ε1 ⊗ e1 , ε1 ⊗ e1 )| ≥ |c|/2 pour λ ∈ ω. Soit ϕ0 la fonction à support dans ω telle que, pour λ ∈ ω, ϕ0 (λ) = ϕ(λ) L πλ ,ρ (ε ⊗ e1 , ε1 ⊗ e0 ) L πλ ,ρ (ε1 ⊗ e1 , ε1 ⊗ e1 )−1 . C’est une fonction C ∞ . Posons f 0 = fe1 ,e0 ,ϕ0 . Montrons que l’on a l’égalité (3)

ASTÉRISQUE 346

I(ε, e0 , f ) = I(ε1 , e0 , f 0 ).

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

263

D’après (2), il suffit de prouver que, pour tous λ et g, on a l’égalité ϕ(λ) L πλ ,ρ (ε ⊗ πλ (g)e0 , l(π 0 (g)e0 ) ⊗ e0 ) = ϕ0 (λ) L πλ ,ρ (ε1 ⊗ πλ (g)e0 , l(π 0 (g)e0 ) ⊗ e1 ). Tous les termes étant C ∞ , on peut supposer λ en position générale, donc πλ irréductible. On peut aussi supposer λ ∈ ω. Fixons un tel λ et un élément non nul l ∈ HomH (πλ , ρ). D’après 5.1, il existe c ∈ C× tel que

L πλ ,ρ (ε0 ⊗ e0 , ε ⊗ e) = c(ε0 , l(e))(l(e0 ), ε) 0 pour tous e, e0 ∈ K G Q,τ et ε, ε ∈ Eρ . Les deux membres de l’égalité à prouver valent

cϕ(λ)(ε, l(e0 ))(l(πλ (g)e0 ), l(π 0 (g)e0 )). Cela prouve cette égalité et (3). D’après (3), I(ε1 , e0 , f 0 ) 6= 0. D’après (1), π 0 (f 0 )e0 6= 0. Alors, d’après 1.6(1), le produit scalaire (e0 , e1 ) n’est pas nul. Puisque e0 ∈ Eπ0 et e1 ∈ Eπ00 , on a donc π 0 = π 00 et L π0 ,ρ est non nul par définition de π 00 . 5.7. Tout entrelacement est tempéré. — Soient (V, qV ) et (W, qW ) deux espaces quadratiques compatibles. Soit π ∈ Temp(G) et ρ ∈ Temp(H). Proposition. — On a m(π, ρ) = 1 si et seulement si L π,ρ est non nul. Démonstration. — On peut supposer dV > dW . Le cas dV = dW + 1 est traité par la proposition précédente. Supposons dV ≥ dW + 3. Comme dans la preuve précédente, on peut supposer m(π, ρ) = 1 et on doit montrer que L π,ρ n’est pas nul. Posons k = (dV − dW + 1)/2. Soit (Z 0 , qZ 0 ) un espace hyperbolique de dimension 2k, notons (V 0 , qV 0 ) la somme directe orthogonale de W et Z 0 . Alors (V 0 , qV 0 ) et (V, qV ) sont compatibles et dV 0 = dV + 1. Le groupe spécial orthogonal G0 de V 0 contient un groupe de Levi L0 isomorphe à GLk × H. Soient P 0 ∈ P (L0 ) et γ une représentation 0 admissible irréductible et cuspidale de GLk (F ). Posons ρ0 = IndG P 0 (γ ⊗ ρ). D’après le théorème 15.1 de [8], on peut choisir γ de sorte que d’une part ρ0 soit irréductible, d’autre part l’hypothèse m(π, ρ) = 1 entraîne m(ρ0 , π) = 1. On fixe un tel γ. Grâce à la proposition 5.6, la forme L ρ0 ,π n’est pas nulle. Grâce au lemme 5.3(ii), la forme L π,ρ est elle-aussi non nulle. C’est ce qu’il fallait prouver.

6. Expression spectrale de la limite d’une intégrale 6.1. Le théorème. — Soient (V, qV ) et (W, qW ) deux espaces quadratiques compatibles. On suppose dV > dW . On utilise les constructions et notations de 4.2. On fixe un Levi minimal Mmin de G contenu dans M . On suppose, ainsi qu’il est loisible, que le groupe K est en bonne position relativement à Mmin . On fixe des mesures de Haar sur G(F ) et H(F ). Les autres mesures que l’on utilisera sont normalisées

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

264

J.-L. WALDSPURGER

comme en 1.2. On fixe une représentation ρ ∈ Temp(H) et on note θρ son caractère. Soit f ∈ Cc∞ (G(F )). Pour g ∈ G(F ), on définit une fonction g f ξ sur H(F ) par Z g ξ f (g −1 hug)ξ(u)du f (h) = U (F )

et on pose Z

θρ (h)g f ξ (h)dh.

J(θρ , f, g) = H(F )

Pour un entier N ≥ 1, on pose Z J(θρ , f, g)κN (g)dg.

JN (θρ , f ) = U (F )H(F )\G(F )

Pour L ∈ L (Mmin ), notons Πell (L) l’ensemble des classes d’isomorphie de représentations admissibles irréductibles tempérées et elliptiques de L(F ). Cet ensemble se décompose en orbites pour l’action π 7→ πλ de i A ∗L . On note {Πell (L)} l’ensemble des orbites. Soient O une telle orbite et π ∈ O. Ecrivons ˜ L = GLk1 × · · · × GLks × G, π = µ1 ⊗ · · · ⊗ µs ⊗ π ˜. ˜ est le La représentation π ˜ ne dépend pas du choix de π dans O. D’autre part, G ˜ groupe spécial orthogonal d’un sous-espace V de V qui est compatible à W . On définit comme en 4.1 et 5.1 les nombres t(˜ π ) et m(˜ π , ρ). On pose t(π) = t(˜ π ) et m( O, ρ) = m(π, ρ) = m(˜ π , ρ). Notons aussi i A ∨O le groupe des λ ∈ i A ∗L tels que, pour tout π ∈ O, πλ soit équivalente à π. On pose X X Jspec (θρ , f ) = |W L ||W G |−1 (−1)aL L∈ L (Mmin ) −1 [i A ∨O : i A ∨ t(π)−1 L,F ]

O∈{Πell (L)};m( O,ρ)=1

Z i A∗ L,F

JLG (πλ , f )dλ,

où, pour toute orbite O, on a fixé un élément π ∈ O. La fonction λ 7→ JLG (πλ , f ) est C ∞ . Si l’on fixe un sous-groupe ouvert compact Kf de G(F ) tel que f soit biinvariante par Kf , elle n’est non nulle que si π admet des invariants non nuls par Kf . Il n’y a qu’un nombre fini d’orbites O vérifiant cette condition. L’expression ci-dessus est donc absolument convergente. Théorème. — Soit f ∈ Cc∞ (G(F )). Si f est très cuspidale, on a l’égalité limN →∞ JN (θρ , f ) = Jspec (θρ , f ). Toute la section est consacrée à la preuve de ce théorème. Pour toute cette section, on fixe une fonction f ∈ Cc∞ (G(F )), que l’on suppose très cuspidale. On fixe un sousgroupe ouvert compact Kf de K, distingué dans K, tel que f soit biinvariante par Kf .

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

265

6.2. Utilisation de la formule de Plancherel. — Exprimons f à l’aide de la formule de Plancherel. Comme on l’a dit en 1.6, pour tout L ∈ L (Mmin ), on peut fixer un sous-ensemble fini Π2 (L)f ⊂ Π2 (L) de sorte que pour tout g ∈ G(F ), X X f O (g), f (g) = |W L ||W G |−1 L∈ L (Mmin )

O∈Π2 (L)f

où ∨

f O (g) = [i A O : i A

∨ −1 L,F ]

Z i A∗ L,F

−1 m(τλ )trace(IndG )IndG Q (τλ , g Q (τλ , f )) dλ.

On a remplacé M et P par L et Q dans la formule de 1.6. Pour g ∈ G(F ) et h ∈ H(F ), on a donc Z X X g ξ f O (g −1 hug)ξ(u) du. f (h) = |W L ||W G |−1 U (F ) L∈ L (M min )

O∈Π2 (L)f

On fixe un produit scalaire invariant sur Eρ . Pour g ∈ G(F ), fixons un sousgroupe ouvert compact Kg0 de H(F ) tel que g −1 Kg0 g ⊂ Kf . La fonction g f ξ sur H(F ) Kg0

K0

est biinvariante par Kg0 . Fixons une base orthonormée Bρ g du sous-espace Eρ éléments de Eρ invariants par Kg0 . Alors on a l’égalité X Z (ε, ρ(h)ε)g f ξ (h)dh, J(θρ , f, g) = K0

ε∈ Bρ g

des

H(F )

d’où J(θρ , f, g) =

X Z ε∈ B

0 Kg ρ

Z (ε, ρ(h)ε)

H(F )

X

X

|W L ||W G |−1

U (F ) L∈ L (M min )

f O (g −1 hug)ξ(u) du dh.

O∈Π2 (L)f

Pour ε ∈ Eρ , L ∈ L (Mmin ), O ∈ Π2 (L)f , et g ∈ G(F ), posons Z (ε, ρ(h)ε)f O (g −1 hug)ξ(u) du dh. (1) JL, O (ε, f, g) = H(F )×U (F )

On a (2) cette expression est absolument convergente. D’après Harish-Chandra ([16] proposition VI.3.1), la fonction f O appartient à l’espace de Schwartz-Harish-Chandra S (G(F )). D’après [16] proposition II.4.5, pour tout entier D, on a une majoration Z |f O (g −1 hug)|du  δP (h)−1/2 ΞM (h)σ(h)−D U (F )

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

266

J.-L. WALDSPURGER

pour tout h ∈ H(F ). Sur H(F ), le module δP est trivial et ΞM coïncide avec ΞG0 . D’autre part, on a une majoration |(ε, ρ(h)ε)|  ΞH (h) pour tout h ∈ H(F ). Enfin, l’intégrale Z ΞH (h)ΞG0 (h)dh H(F )

est convergente d’après 4.3(4). Cela prouve (2). Pour g ∈ G(F ), on a donc l’égalité X X (3) J(θρ , f, g) = |W L ||W G |−1 ε∈ B

0 Kg ρ

L∈ L (Mmin )

X

JL, O (ε, f, g).

O∈Π2 (L)f

6.3. Apparition des entrelacements tempérés. — On poursuit le calcul précédent. Fixons L ∈ L (Mmin ) et O ∈ Π2 (L)f . Pour c ∈ N, introduisons le sous-groupe U (F )c de U (F ), cf. 4.3. On a (1) il existe c0 ∈ N tel que, pour tout c ≥ c0 , tout g ∈ M (F )K et tout h ∈ H(F ), on ait l’égalité Z Z f O (g −1 hug)ξ(u) du = f O (g −1 hug)ξ(u) du. U (F )

U (F )c

La preuve est similaire à celle du lemme 3.5. Pour c ≥ c0 notons U (F )c − U (F )c0 le complémentaire de U (F )c0 dans U (F )c . Il existe c0 tel que pour tout c ≥ c0 et tout u ∈ U (F )c − U (F )c0 , l’intégrale Z ξ(aua−1 )da A(F )∩Kf

soit nulle. Choisissons un tel c0 , soit c ≥ c0 . Parce que A commute à M et à H ⊂ M , parce que Kf est distingué dans K et parce que f O est, comme f , biinvariante par Kf , on a l’égalité f O (g −1 hug) = f O (g −1 ha−1 uag) pour tous g ∈ M (F )K, h ∈ H(F ), a ∈ A(F ) ∩ Kf . Alors Z (2) f O (g −1 hug)ξ(u) du = (mes(A(F ) ∩ Kf ))−1 U (F )c −U (F )c0

Z

Z

U (F )c −U (F )c0

f O (g −1 ha−1 uag)ξ(u) da du.

A(F )∩Kf

Cette expression est absolument convergente. On peut effectuer le changement de variable u 7→ aua−1 et l’expression ci-dessus devient Z Z (mes(A(F ) ∩ Kf ))−1 f O (g −1 hug) ξ(aua−1 )da du. U (F )c −U (F )c0

ASTÉRISQUE 346

A(F )∩Kf

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

267

L’intégrale intérieure est nulle donc aussi le membre de gauche de (2). Cela suffit à prouver (1). Comme en 1.6, on fixe τ ∈ O et un élément Q ∈ P (L). Pour simplifier les notations, on pose πλ = IndG Q (τλ ) pour tout λ ∈ i A ∗L,F . On réalise toutes les représentations πλ dans l’espace commun Kf Kf KG du sous-espace ( K G . Pour tout g ∈ Q,τ . Fixons une base orthonormée B O Q,τ )

G(F ), on a l’égalité

−1 f O (g) = [i A ∨O : i A ∨ L,F ]

X Z K

e∈ B O f

iA

∗ L,F

m(τλ )(e, πλ (g −1 )πλ (f )e) dλ.

Soient c0 vérifiant (1), c ≥ c0 , g ∈ M (F )K et ε ∈ Eρ . D’après (1) et la définition 6.2(1), on a l’égalité Z −1 ] (ε, ρ(h)ε) JL, O (ε, f, g) = [i A ∨O : i A ∨ L,F H(F )×U (F )c

X Z

m(τλ )(πλ (g)e, πλ ((hu)−1 g)πλ (f )e)ξ(u) dλ du dh.

i A∗ L,F

K

e∈ B O f

En changeant h et u en leurs inverses, on obtient Z −1 JL, O (ε, f, g) = [i A ∨O : i A ∨ ] L,F

(ρ(h)ε, ε)

H(F )×U (F )c

X Z Kf

e∈ B O

i A∗ L,F

¯ dλ du dh. m(τλ )(πλ (g)e, πλ (hug)πλ (f )e)ξ(u)

Remarquons que, pour g fixé, on a une majoration |(πλ (g)e, πλ (hug)πλ (f )e)|  ΞG (hu) pour tous λ, h, u. Grâce à 4.3(4), on en déduit que l’expression ci-dessus est absolument convergente. On peut donc permuter les intégrales : X Z ∨ ∨ −1 JL, O (ε, f, g) = [i A O : i A L,F ] m(τλ ) K

e∈ B O f

Z

i A∗ L,F

¯ (ρ(h)ε, ε))(πλ (g)e, πλ (hug)πλ (f )e)ξ(u)du dh dλ.

H(F )×U (F )c

On reconnaît l’intégrale intérieure : c’est L πλ ,ρ,c (ε ⊗ πλ (g)e, ε ⊗ πλ (g)πλ (f )e). D’après le lemme 5.1, quitte à accroître c0 (en fait, la preuve de (1) montre que ce n’est pas nécessaire), c’est aussi L πλ ,ρ (ε ⊗ πλ (g)e, ε ⊗ πλ (g)πλ (f )e). On obtient X Z −1 (3) JL, O (ε, f, g) = [i A ∨O : i A ∨ ] m(τλ ) L,F K

e∈ B O f

i A∗ L,F

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

268

J.-L. WALDSPURGER

L πλ ,ρ (ε ⊗ πλ (g)e, ε ⊗ πλ (g)πλ (f )e)dλ pour tout g ∈ M (F )K. On peut écrire L et τ comme en 5.3. C’est-à-dire que ˜ L = GLk1 × · · · × GLks × G et τ = µ1 ⊗ · · · ⊗ µs ⊗ π ˜ . Si m(˜ π , ρ) = 0, on a aussi L π˜ ,ρ = 0 d’après la proposition 5.7 et L πλ ,ρ = 0 pour tout λ d’après le lemme 5.3(ii). Posons m( O, ρ) = m(˜ π , ρ). Alors (4) si m( O, ρ) = 0, JL, O (ε, f, g) = 0 pour tout g ∈ M (F )K et tout ε ∈ Eρ . Supposons désormais m( O, ρ) = 1. On fixe des familles (ε0j )j=1,...,n , (εj )j=1,...,n , 0 (ej )j=1,...,n , (ej )j=1,...,n , (ϕj )j=1,...,n vérifiant le lemme 5.3(iii). Soit N ≥ 1. Pour λ ∈ i A ∗L,F , g ∈ M (F )K et e ∈ K G Q,τ , considérons la somme X Xλ (e, g) = L πλ ,ρ (ε ⊗ πλ (g)e, ε ⊗ πλ (g)πλ (f )e). K0

ε∈ Bρ g

Supposons λ en position générale. Alors πλ est irréductible. Fixons un élément non nul lλ ∈ HomH,ξ (πλ , ρ). Comme on l’a dit en 5.1, il existe un nombre complexe non nul cλ tel que L πλ ,ρ (ε0 ⊗ e0 , ε ⊗ e) = cλ (ε0 , lλ (e))(lλ (e0 ), ε) pour tous ε, ε0 ∈ Eρ , e, e0 ∈ K G Q,τ . La propriété (iii) du lemme 5.3 s’écrit X (5) cλ ϕj (λ)(ε0j , lλ (ej ))(lλ (e0j ), εj ) = 1. j=1,...,n

On a Xλ (e, g) = cλ

X ε∈ B

(ε, lλ (πλ (g)πλ (f )e))(lλ (πλ (g)e), ε).

0 Kg ρ

L’élément lλ (πλ (g)πλ (f )e) est invariant par Kg0 pour tout e ∈ K G Q,τ . Alors X lλ (πλ (g)πλ (f )e) = (ε, lλ (πλ (g)πλ (f )e))ε, K0

ε∈ Bρ g

et Xλ (e, g) = cλ (lλ (πλ (g)e), lλ (πλ (g)πλ (f )e)). On peut multiplier Xλ (e, g) par le membre de gauche de (5) et on obtient X (6) Xλ (e, g) = ϕj (λ)Xλ,j (e, g), j=1,...,n

où on a posé Xλ,j (e, g) = c2λ (lλ (πλ (g)e), lλ (πλ (g)πλ (f )e))(ε0j , lλ (ej ))(lλ (e0j ), εj ). Fixons j. Le produit de l’un des facteurs cλ et des deux premiers produits scalaires est égal à L πλ ,ρ (ε0j ⊗ πλ (g)e, lλ (πλ (g)πλ (f )e) ⊗ ej ).

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

269

Supposons g ∈ M (F )K. En choisissant un entier c0 assez grand, on peut ici remplacer L πλ ,ρ par L πλ ,ρ,c pour tout c ≥ c0 . Remarquons que c0 est indépendant de λ : cela résulte de la preuve du lemme 3.5. On a donc Z ¯ (ρ(h)ε0j , lλ (πλ (g)πλ (f )e))(πλ (g)e, πλ (hu)ej )cλ (lλ (e0j ), εj )ξ(u)du Xλ,j (e, g) = dh H(F )U (F )c

pourvu que c ≥ c0 . Dans l’intégrale, le produit de cλ et des produits scalaires extrêmes est égal à L πλ ,ρ (ρ(h)ε0j ⊗ e0j , εj ⊗ πλ (g)πλ (f )e). On peut encore remplacer L πλ ,ρ par L πλ ,ρ,c pourvu que c ≥ c0 . On obtient Z Z (πλ (g)e, πλ (hu)ej )(ρ(h0 h)ε0j , εj ) (7) Xλ,j (e, g) = H(F )U (F )c

H(F )U (F )c

0 ¯ 0 )ξ(u)du ¯ (e0j , πλ (h0 u0 g)πλ (f )e)ξ(u dh0 du dh.

On a (8) pour g fixé, cette expression est absolument convergente, uniformément en λ. En effet, elle est majorée en valeur absolue par Z Z ΞG (hu)ΞH (h0 h)ΞG (h0 u0 )du0 dh0 du dh H(F )U (F )c

H(F )U (F )c

qui est convergente d’après 4.3(5). On peut maintenant lever l’hypothèse que λ est en position générale. Grâce à (8), la formule (7) définit une fonction C ∞ de λ et l’égalité (6) se prolonge par continuité à tout λ. Pour deux entiers c, c0 ∈ N, et pour g ∈ M (F )K, posons Z Z Xλ,j,c,c0 (e, g) = (ρ(h)ε0j , εj )(πλ (h0 u0 g)e, πλ (hu)ei ) H(F )U (F )c

H(F )U (F )c0

0 ¯ (e0j , πλ (h0 u0 g)πλ (f )e)ξ(u)du dh0 du dh.

Comme (7), cette expression est absolument convergente. On a (9) il existe c0 indépendant de N et λ tel que, si c ≥ c0 et c0 ≥ c0 , alors Xλ,j (e, g) = Xλ,j,c,c0 (e, g) pour tout g ∈ M (F )K. Pour a appartenant à un sous-groupe ouvert compact assez petit de A(F ), le changement de variables u 7→ aua−1 , u0 7→ au0 a−1 dans la définition ci-dessus de −1 ¯ ¯ Xλ,i,c,c0 (e, g) revient à y remplacer ξ(u) par ξ(aua ). Comme dans la preuve de (1), on en déduit que, si c0 est assez grand, Xλ,j,c,c0 (e, g) ne dépend pas de c, pourvu que c ≥ c0 . Pour c, c0 ≥ c0 , on a donc Xλ,j,c,c0 (e, g) = Xλ,j,c0 ,c0 . On peut remplacer c par c0 dans la formule (7). Dans cette formule, effectuons le changement de variables h 7→ h0 −1 h, u 7→ h−1 h0 u0 −1 h0 −1 hu. Alors le membre de droite de (7) devient Xλ,j,c0 ,c0 . Cela prouve (9). En rassemblant l’égalité (2), la définition de Xλ (e, g), l’égalité (6) et la propriété (9), on obtient le résultat suivant. Rappelons que l’on a supposé m( O, ρ) = 1. Il existe c0 tel que, pour tout N , tout g ∈ M (F )K, tous c, c0 tels que c ≥ c0 , c0 ≥ c0 , on a

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

270

J.-L. WALDSPURGER

l’égalité (10)

−1 JL, O (ε, f, g) = [i A ∨O : i A ∨ L,F ]

X K0

X K

e∈ B O f

ε∈ Bρ g

X j=1,...,n

Z i A∗ L,F

m(τλ )ϕj (λ)Xλ,j,c,c0 (e, g) dλ.

On note JL, O (θρ , f, g) le membre de droite de cette égalité, qui est défini pour tout g ∈ M (F )K. 6.4. Une première approximation. — D’après 6.2(3), 6.3(4) et 6.3(10), on a l’égalité X X JL, O (θρ , f, g), J(θρ , f, g) = |W L ||W G |−1 L∈ L (Mmin )

O∈Π2 (L)f ,m( O,ρ)=1

pour tout g ∈ M (F )K. Par définition, JN (θρ , f ) est l’intégrale de J(θρ , f, g)κN (g) sur g ∈ H(F )U (F )\G(F ) ou, ce qui revient au même, l’intégrale de J(θρ , f, mk)κN (mk)δP (m)−1 sur m ∈ H(F )\M (F ) et k ∈ K. Donc Z Z X (1) JN (θρ , f ) = |W L ||W G |−1 H(F )\M (F )

X

K L∈ L (M min )

JL, O (θρ , f, mk)κN (mk)δP (m)−1 dk dm.

O∈Π2 (L)f ,m( O,ρ)=1

Soient L ∈ L (Mmin ) et O ∈ Π2 (L)f tel que m( O, ρ) = 1. On reprend les notations du paragraphe précédent. On fixe c0 vérifiant 6.3(9) et c ≥ c0 . Pour tout entier C ∈ N, posons X X Z ∨ ∨ −1 JL, O,N,C (θρ , f ) = [i A O : i A L,F ] m(τλ )ϕj (λ) K

e∈ B O f

Z

j=1,...,n

¯ 1σ 0 tels que l’on ait la majoration |ΦN (g 0 ) − ΦY (g 0 )|  N −R pour tout N ≥ 2, tout g 0 ∈ G(F ) tel que σ(g 0 ) ≤ Clog(N ) et tout Y ∈ A + Pmin vérifiant les inégalités c1 log(N ) ≤ α(Y ) ≤ c2 N pour tout α ∈ ∆min . Démonstration. — Commençons par préciser le calcul de convergence que l’on a fait avant l’énoncé. On a (2) il existe R1 ≥ 0 tel que |ΦN (g 0 )|  N R1 pour tout N ≥ 2 et tout g 0 ∈ G(F ) tel que σ(g 0 ) ≤ Clog(N ).

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

275

En effet, pour e ∈ K G Q,τ on a plus précisément la majoration |Φ(e, g, g 0 , λ)|  ΞG (g 0 −1 g)ΞG (g) pour tous λ, g, g 0 . Grâce à 3.3(5), il existe R2 > 0 tel que ΞG (g 0 −1 g)  exp(R2 σ(g 0 ))ΞG (g). Pour g 0 tel que σ(g 0 ) ≤ Clog(N ),on obtient |Φ(e, g, g 0 , λ)|  N CR2 ΞG (g)2 . D’après 4.3(2), il existe R3 > 0 tel que l’intégrale (1) soit essentiellement majorée par N R3 . On en déduit (2) avec R1 = CR2 + R3 . Définissons une fonction D sur Mmin (F )+ par D(m) = mes(KmK)mes(K ∩ Mmin (F ))−1 . D’après [16], on a une majoration D(m)  δPmin (m). Pour toute fonction f 0 ∈ Cc∞ (G(F )), on a l’égalité Z Z Z Z f 0 (g)dg = D(m)f 0 (k1 mk2 )dm dk1 dk2 , G(F )

K

K

Mmin (F )+

cf. [6] (1.3). Pour tout Q1 = L1 U1 ∈ F (Mmin ), la famille Y permet de définir des Q1 fonctions ζ 7→ σM (ζ, Y ) et ζ 7→ τQ1 (ζ − YQ1 ) sur A Mmin (cf.[6] ; on a repris les min définitions en [17] 10.3). On vérifie que l’égalité Q1 σM (ζ, Y )τQ1 (ζ − YQ1 ) = 1 min

entraîne (3) β(ζ) > inf {α(Y ); α ∈ ∆} ≥ 0 pour toute racine β de AMmin dans u1 . On a l’égalité X Q1 σM (ζ, Y )τQ1 (ζ − YQ1 ) = 1 min Q1 ∈ F (Mmin )

pour tout ζ. L’inégalité précédente entraîne que, pour ζ ∈ A + Pmin , seuls interviennent de façon non nulle les Q1 contenant Pmin . C’est-à-dire que, pour ζ ∈ A + Pmin , on a l’égalité X Q1 σM (ζ, Y )τQ1 (ζ − YQ1 ) = 1. min Pour toute

Q1 ∈ F (Mmin ),Pmin ⊂Q1 fonction f 0 ∈ Cc∞ (G(F )),

Z (4) G(F )

f 0 (g)dg =

on a donc l’égalité Z Z Z X

Q1 ∈ F (Mmin ),Pmin ⊂Q1

K

K

Mmin (F )+

Q1 f 0 (k1 mk2 )D(m)σM (HMmin (m), Y )τQ1 (HMmin (m) − YQ1 )dm dk1 dk2 . min

Pour Q1 ∈ F (Mmin ) tel que Pmin ⊂ Q1 , posons X Z −1 ΦN,Y,Q1 (g 0 ) = [i A ∨O : i A ∨ ] L,F K

e∈ B O f

i A∗ L,F

Z Z Z m(τλ )ϕ(λ) K

K

Mmin (F )+

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

276

J.-L. WALDSPURGER

Φ(e, k1 mk2 , g 0 , λ)κN (k1 mk2 ) Q1 D(m)σM (HMmin (m), Y )τQ1 (HMmin (m) − YQ1 )dm dk1 dk2 dλ. min

En l’appliquant (4), on a l’égalité ΦN (g 0 ) =

X

ΦN,Y,Q1 (g 0 ).

Q1 ∈ F (Mmin ),Pmin ⊂Q1

Considérons d’abord le sous-groupe parabolique Q1 = G. Dans ce cas, pour g = G k1 mk2 , avec k1 , k2 ∈ K et m ∈ Mmin (F )+ , on a σM (HMmin (m), Y )τG (HMmin (m) − min YG ) = 1 si et seulement si u(g, Y ) = 1. On a donc X Z −1 ΦN,Y,G (g 0 ) = [i A ∨O : i A ∨ ] m(τλ )ϕ(λ) L,F K

e∈ B O f

Z

i A∗ L,F

Φ(e, g, g 0 , λ)κN (g)u(g, Y )dg dλ.

G(F )

Montrons que (5) il existe c2 > 0 tel que, si α(Y ) ≤ c2 N pour tout α ∈ ∆min , on a l’égalité u(g, Y )κN (g) = u(g, Y ) pour tout g ∈ G(F ). Ecrivons g = muk, avec m ∈ M (F ), u ∈ U (F ) et k ∈ K. On a κN (g) = κN (m) et une majoration σ(m)  σ(g). Par définition de la fonction κN , il existe c3 tel que la majoration σ(m) ≤ c3 N entraîne κN (m) = 1. On a d’autre part une majoration σ(g)  sup{α(Y ); α ∈ ∆} pour tout g ∈ G(F ) tel que u(g, Y ) = 1. La combinaison de ces propriétés entraîne (5). On déduit de (5) que, si Y vérifie les conditions de cette assertion, on a l’égalité ΦN,Y,G (g 0 ) = ΦY (g 0 ) pour tout g 0 . Pour démontrer la proposition, il suffit donc de trouver c1 tel que, si Y vérifie les minorations de l’énoncé, on a la majoration (6)

|ΦN,Y,Q1 (g 0 )|  N −R

pour g 0 comme dans l’énoncé et tout Q1 6= G. On fixe désormais Q1 = L1 U1 ∈ F (Mmin ) tel que Pmin ⊂ Q1 et Q1 6= G. Pour simplifier la rédaction, on va considérer un réel c1 et supposer α(Y ) ≥ c1 log(N ) pour tout α ∈ ∆. On montrera que toutes les propriétés dont on a besoin sont vérifiées si c1 est assez grand. Soient g 0 ∈ G(F ) tel que σ(g 0 ) ≤ Clog(N ), k1 , k2 ∈ K et Q1 m ∈ Mmin (F )+ . On pose ζ = HMmin (m) et on suppose σM (ζ, Y )τQ1 (ζ − YQ1 ) = 1. min 0 0 0 0 −1 0 −1 0 0 Ecrivons g k1 = k l u , avec u ∈ U1 (F ), l ∈ L1 (F ), k ∈ K. On a (7) si c1 est assez grand, k2−1 m−1 u0 mk2 appartient à Kf . Posons u0 = exp(X 0 ), avec X 0 ∈ u1 (F ) et fixons une norme |.| sur g(F ). On a log(|X 0 |)  σ(u0 )  σ(g 0 ) ≤ Clog(N ). Il existe donc c4 , c5 > 0 tels que log(|m−1 X 0 m|) ≤ c4 log(N ) − c5 inf {β(ζ); β racine de AMmin dans u1 }. Grâce à (3), m−1 X 0 m est aussi petit que l’on veut pourvu que c1 soit assez grand. On peut en particulier imposer que m−1 u0 m = exp(m−1 X 0 m) appartienne à Kf . Puisque Kf est distingué dans K, (7) en résulte.

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

277

Notons Mmin (F )L1 ,+ l’ensemble des m0 ∈ Mmin (F ) tels que α(HMmin (m0 )) ≥ 0 pour tout α ∈ ∆L1 . Posons K1 = K ∩ L1 (F ). L’élément l0 m de L1 (F ) s’écrit l0 m = k3 m0 k4 , avec k3 , k4 ∈ K1 et m0 ∈ Mmin (F )L1 ,+ . Posons ζ 0 = HMmin (m0 ). On a (8) pour tout c > 0, m0 appartient à Mmin (F )+ et vérifie α(ζ 0 ) > clog(N ) pour tout α ∈ ∆ − ∆L1 , pourvu que c1 soit assez grand. Soit α. Notons uα le sous-espace radiciel de umin associé à α. Il existe c6 , c7 > 0 tels que, pour tout élément non nul X ∈ uα (F ) et tout x ∈ Mmin (F ), on ait les inégalités (9)

α(HMmin (x)) − c6 ≤ log(

|xXx−1 | ) ≤ α(HMmin (x)) + c7 . |X|

Supposons α ∈ ∆ − ∆L1 , soit X un élément non nul de uα (F ). On a m0 Xm0 −1 = k3−1 l0 mk4−1 Xk4 m−1 l0 −1 k3 . Puisque k4 appartient à L1 (F ), k4−1 Xk4 appartient à u1 (F ). En utilisant (9) pour m, on voit qu’il existe des constantes c8 , c9 , c10 > 0 telles que l’on ait l’inégalité log(|m0 Xm0 −1 |) ≥ log(|X|) − c8 σ(l0 ) + c9 inf {β(ζ); β racine de AMmin dans u1 } − c10 . On a σ(l0 )  σ(g 0 ) ≤ Clog(N ). Grâce à (3), on a donc log(|m0 Xm0 −1 |) > log(|X|) + clog(N ) + c7 pourvu que c1 soit assez grand. Alors (9) entraîne la minoration cherchée de α(ζ 0 ). En particulier α(ζ 0 ) > 0. Puisque l’on a aussi α(ζ 0 ) ≥ 0 pour α ∈ ∆L1 par définition de m0 , on a bien m0 ∈ Mmin (F )+ . Cela prouve (8). K Pour tout λ et tout e ∈ B O f , on a l’égalité Φ(e, k1 mk2 , g 0 , λ) = (πλ (l0 u0 mk2 )e, πλ (k 0 )e0 )(πλ (k1−1 )e00 , πλ (mk2 )πλ (f )e). Grâce à (7), on peut supprimer u0 de cette expression pourvu que c1 soit assez grand. On obtient Φ(e, k1 mk2 , g 0 , λ) = (πλ (m0 k4 k2 )e, πλ (k3−1 k 0 )e0 )(πλ (k1−1 )e00 , πλ (mk2 )πλ (f )e). Posons

Φw (e, k1 mk2 , g 0 , λ) = (πλ (m0 k4 k2 )e, πλ (k3−1 k 0 )e0 )Q1 ,λ (πλ (k1−1 )e00 , πλ (mk2 )πλ (f )e)Q1 ,λ . Le lemme 6.5 affirme l’existence de réels R4 ≥ 0 et ε > 0 tels que la valeur absolue de la différence Φ(e, k1 mk2 , g 0 , λ) − Φw (e, k1 mk2 , g 0 , λ) soit bornée par la somme de δQ1 (m0 )−1/2 ΞL1 (m0 )σ(m0 )R4 sup{exp(−εα(ζ 0 ); α ∈ ∆ − ∆L1 }|(πλ (k1−1 )e00 , πλ (mk2 )πλ (f )e)| et de δQ1 (m0 )−1/2 |(πλ (m0 k4 k2 )e, πλ (k3−1 k 0 )e0 )Q1 ,λ |δQ1 (m)−1/2 ΞL1 (m) σ(m)R4 sup{exp(−εα(ζ); α ∈ ∆ − ∆L1 }. On sait qu’il existe R5 ≥ 0 tel que δQ1 (x)−1/2 ΞL1 (x)  ΞG (x)σ(x)R5 pour tout x ∈ Mmin (F )+ . D’autre part, ΞG (m0 ) = ΞG (l0 m). En utilisant (8), on voit qu’il existe

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

278

J.-L. WALDSPURGER

R6 ≥ 0 tel que, pour tout c > 0, les expressions ci-dessus soient essentiellement majorées par N −c ΞG (l0 m)ΞG (m)σ(l0 )R6 σ(m)R6 pourvu que c1 soit assez grand. Définissons des ΦN,Y,Q1 ,w (g 0 ) en remplaçant Φ(e, k1 mk2 , g 0 , λ) par définitions de ΦN,Y,Q1 (e, g 0 , λ) et ΦN,Y,Q1 (g 0 ). Si on majoration ci-dessus, le même calcul qu’en (2) montre

termes ΦN,Y,Q1 ,w (e, g 0 , λ) et Φw (e, k1 mk2 , g 0 , λ) dans les oublie le facteur N −c de la l’existence de R7 ≥ 0 tel que

|ΦN,Y,Q1 (g 0 ) − ΦN,Y,Q1 ,w (g 0 )|  N R7 . En réintroduisant le facteur N −c , on voit que, pour tout c > 0, la différence ci-dessus est essentiellement majorée par N −c pourvu que c1 soit assez grand. Pour prouver (6), il suffit donc de prouver la majoration |ΦN,Y,Q1 ,w (g 0 )|  N −R . On poursuit le calcul précédent, avec les mêmes notations. D’après la définition de 6.5, on peut décomposer Φw (e, k1 mk2 , g 0 , λ) en X Φs1 ,s2 (e, k1 mk2 , g 0 , λ) s1 ,s2 ∈W (L1 |G|L)

où Φs1 ,s2 (e, k1 mk2 , g 0 , λ) = 1

((s1 τ )s1 λ ) ◦ γ(s1 ) ◦ πλ (m0 k4 k2 )e, JQ1,s |s1 Qs−1 ((s1 τ )s1 λ ) ◦ γ(s1 ) ◦ πλ (k3−1 k0 )e0 )L1 |s1 Qs−1 1 1 1

2

((s2 τ )s2 λ ) ◦ γ(s2 ) ◦ πλ (k1−1 )e00 , JQ˜ 1,s |s2 Qs−1 ((s2 τ )s2 λ ) ◦ γ(s2 ) ◦ πλ (mk2 )πλ (f)e)L1 . |s2 Qs−1 2 2 2

(JQ˜ 1,s (JQ1,s

Cette définition n’a bien sûr de sens que « presque partout » en λ, les opérateurs d’entrelacement pouvant avoir des pôles. Au moins formellement, on peut décomposer de même ΦN,Y,Q1 ,w (g 0 ) en somme de termes ΦN,Y,Q1 ,s1 ,s2 (g 0 ). Il y a un problème de convergence à cause des pôles des opérateurs d’entrelacement. L’assertion suivante va résoudre ce problème. Soient s1 , s2 ∈ W (L1 |G|L). Montrons que (10) pour e fixé, la fonction m(τλ )Φs1 ,s2 (e, k1 mk2 , g 0 , λ) est une combinaison linéaire de fonctions qui sont elles-mêmes des produits f1 (m0 , λ)f2 (m, λ)f3 (k1 , k2 , k3 , k4 , k 0 )f4 (λ), où : 1 f1 (m0 , λ) = δQ1 (m0 )−1/2 (IndL L ∩s 1

L1 pour des éléments e1 et e01 de K L ∩s 1

−1 1 Qs1 ,s1 τ

−1 1 Qs1

;

1 f2 (m, λ) = δQ1 (m)−1/2 (e02 , IndL L ∩s 1

L1 pour des éléments e2 et e02 de K L ∩s 1

−1 2 Qs2 ,s2 τ

((s1 τ )s1 λ , m0 )e01 , e1 )

−1 2 Qs2

((s2 τ )s2 λ , m)e2 )

;

f3 est une fonction localement constante des variables k1 , k2 , k3 , k4 et k 0 ; f4 est une fonction C ∞ de λ.

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

279

Fixons un sous-groupe ouvert compact K0 de K, distingué dans K, inclus dans ˜ , Kf et tel que e, e0 et e00 soient invariants par K0 . Fixons des bases B, resp. B1 , B 1 G G ˜ B2 , B2 , des sous-espaces des éléments invariants par K0 dans K Q,τ , resp. K Q1,s1 ,s1 τ , G G KG ˜ 1,s ,s1 τ , K Q1,s ,s2 τ , K Q ˜ 1,s ,s2 τ . Considérons par exemple le terme Q 2 1 2

JQ˜ 1,s

2

((s2 τ )s2 λ ) |s2 Qs−1 2

◦ γ(s2 ) ◦ πλ (mk2 )πλ (f )e

qui intervient dans la définition de Φs1 ,s2 (e, k1 mk2 , g 0 , λ). D’après les propriétés d’entrelacement de nos opérateurs, il est égal à IndG ˜ 1,s ˜ 1,s ((s2 τ )s2 λ , m) ◦ JQ Q 2

−1 2 |s2 Qs2

((s2 τ )s2 λ ) ◦ γ(s2 ) ◦ πλ (k2 )πλ (f )e.

On peut remplacer πλ (k2 )πλ (f )e par son expression dans la base B. Les coefficients sont combinaisons linéaires de produits d’une fonction localement constante en k2 et d’une fonction C ∞ en λ. Pour tout b ∈ B, on peut ensuite remplacer ˜ . Les coefficients JQ˜ 1,s |s2 Qs−1 ((s2 τ )s2 λ ) ◦ γ(s2 )b par son expression dans la base B 2 2 2 sont des fonctions de λ de la forme (˜b2 , JQ˜ 1,s

2

((s2 τ )s2 λ ) |s2 Qs−1 2

◦ γ(s2 )b)

˜ . Notons j ˜ (λ) une telle fonction. En appliquant le même où b ∈ B, ˜b2 ∈ B 2 Q1,s2 calcul aux autres termes, l’expression m(τλ )Φs1 ,s2 (e, k1 mk2 , g 0 , λ) apparaît comme une combinaison linéaire de fonctions qui sont produits de fonctions des deux derniers types de l’assertion et de fonctions des types 0 ˜ L1 ˜ ,b ∈ B ; — une fonction (IndG où ˜b1 ∈ B 1 1 1 ˜ 1,s ((s1 τ )s1 λ , m )b1 , b1 ) Q 1 G L1 ˜ ˜ ˜ — une fonction (b , (Ind ((s τ ) , m)b ) où b ∈ B , b ∈ B ; 2

˜ 1,s Q 2

2

s2 λ

2

2

2

2

2

— une fonction m(τλ )jQ˜ 1,s (λ)jQ1,s2 (λ)jQ˜ 1,s (λ)jQ1,s1 (λ) ; 1

2

Considérons le premier type ci-dessus. Notons e01 , resp. e1 , la restriction de ˜b1 , resp. 1 b1 , à K1 . Ce sont des éléments de K L . En explicitant les définitions, on L ∩s Qs−1 ,s τ 1

1

1

1

voit que, pour tout x ∈ L1 (F ), a fortiori pour x ∈ Mmin (F ), on a l’égalité L1 1 ˜ = δQ1 (x)−1/2 (IndL (IndG ˜ 1,s ((s1 τ )s1 λ , x)b1 , b1 ) Q L ∩s 1

1

−1 1 Ls1

((s1 τ )s1 λ , x)e01 , e1 ).

La fonction du premier type ci-dessus est donc aussi du premier type de l’assertion (10). De même pour les fonctions du deuxième type. Reste celles du troisième type ci-dessus. Or la démonstration du corollaire V.2.3 de [16] s’applique à ces fonctions et montre qu’elles sont des restrictions à i A ∗L,F de fonctions rationnelles sur ( A ∗L ⊗R ∗ C)/i A ∨ L,F , holomorphes au voisinage de i A L,F . Ces fonctions sont donc du quatrième type de l’assertion (10). Cela démontre cette assertion. A l’aide de (10), on voit que l’expression qui définit ΦN,Y,Q1 ,s1 ,s2 (g 0 ) est absolument convergente : puisqu’il n’y a pas de singularité en λ, il suffit de reprendre la preuve déjà faite pour ΦN (g 0 ). On a alors X ΦN,Y,Q1 ,w (g 0 ) = ΦN,Y,Q1 ,s1 ,s2 (g 0 ). s1 ,s2 ∈W (L1 |G|L)

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

280

J.-L. WALDSPURGER

Il suffit de prouver que, pour tous s1 , s2 , on a une majoration |ΦN,Y,Q1 ,s1 ,s2 (g 0 )|  N −R . Fixons s1 , s2 . Posons s = s1 s−1 2 . Supposons d’abord vérifiée l’hypothèse (Hyp) il n’existe pas de sous-groupe parabolique Q2 = L2 U2 ∈ F Mmin tel que Q1 ⊂ Q2 6= G et que s fixe tout point de A L2 . Dans ce cas, introduisons une fonction Z Z Z Z 0 ΨN,Y,Q1 ,s1 ,s2 (g ) = f4 (λ) f1 (m0 , λ)f2 (m, λ)f3 (k1 , k2 , k3 , k4 , k 0 ) i A∗ L,F

K

Mmin (F )+

K

Q1 D(m)κN (k1 mk2 )σM (ζ, Y )τQ1 (ζ − YQ1 )dm dk1 dk2 dλ min

où f1 , f2 , f3 et f4 vérifient les conditions de l’assertion (10). Cette assertion nous dit que ΦN,Y,Q1 ,s1 ,s2 (g 0 ) est combinaison linéaire de fonctions de ce type. On va majorer |ΨN,Y,Q1 ,s1 ,s2 (g 0 )|. Pour tout x ∈ L1 (F ), on choisit des éléments ls1 (x) ∈ s1 L(F )s−1 1 , us1 (x) ∈ L1 (F ) ∩ s1 UQ (F )s−1 et k (x) ∈ K de sorte que x = l (x)u (x)k (x). s1 1 s1 s1 s1 1 On a l’égalité Z 0 f1 (m , λ) = f10 (m0 , x)exp(−(s1 λ)(HL1 ∩s1 Qs−1 (xm0 )))dx, 1

K1

où L1 (ls1 (xm0 ))1/2 . f10 (m0 , x) = δQ1 (m0 )−1/2 ((s1 τ )(ls1 (xm0 ))(e01 (ks1 (xm0 ))), e1 (x))δL 1 ∩s1 Qs1

Le produit scalaire figurant dans cette expression est celui de deux éléments de Es1 τ = Eτ . On écrit f2 (m, λ) de la même façon et on obtient Z Z Z Z Z 0 ΨN,Y,Q1 ,s1 ,s2 (g ) = f3 (k1 , k2 , k3 , k4 , k 0 )D(m)κN (k1 mk2 ) K Q1 σM (ζ, min

K

Mmin (F )+

K1

K1

Y )τQ1 (ζ − YQ1 )f10 (m0 , x)f20 (m, y)f5 (xm0 , ym)dy dx dm dk1 dk2

où

f5 (xm0 , ym) =

Z iA

∗ L,F

f4 (λ)exp(−(s1 λ)(HL1 ∩s1 Qs−1 (xm0 )) + (s2 λ)(HL1 ∩s2 Qs−1 (ym)))dλ. 1

2

Par le changement de variable λ 7→ s−1 1 λ, on a Z 0 f5 (xm0 , ym) = f4 (s−1 1 λ)exp(−λ(ζ(xm , ym)))dλ, i A∗ s

1 Ls1 ,F

où ζ(xm0 , ym) = HL1 ∩s1 Qs−1 (xm0 )) − s(HL1 ∩s2 Qs−1 (ym)). 1

2

Q1 On suppose toujours, comme il est loisible, que m vérifie la relation σM (ζ, Y )τQ1 (ζ− min YQ1 ) = 1. Munissons A Mmin d’une norme euclidienne |.| invariante par l’action de W G . Montrons que (11) on a la minoration |ζ(xm0 , ym)|  log(N )

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

281

pour tous x, y ∈ K1 pourvu que c1 soit assez grand. Posons ζ(xm0 ) = HPmin (xm0 ), ζ(ym) = HPmin (ym). Rappelons que, par définition −1 de W (L1 |G|L), on a L1 ∩ Pmin ⊂ L1 ∩ s1 Qs−1 1 , L1 ∩ Pmin ⊂ L1 ∩ s2 Qs2 . On en déduit que ζ(xm0 , ym) = (ζ(xm0 ) − sζ(ym))s1 Ls−1 , 1

où, comme toujours, ζ 00 7→ ζs00 Ls−1 désigne la projection orthogonale de A Mmin sur 1

1

A s1 Ls−1 . Il suffit de minorer |(ζ(xm0 ) − sζ(ym))L1 |. Parce que x, y ∈ K1 et k3 m0 k4 = 1

l0 m, avec k3 , k4 ∈ K1 , on a les égalités

ζ(xm0 )L1 = ζL0 1 = ζL1 + HL1 (l0 ) = ζ(ym)L1 + HL1 (l0 ). Puisque σ(l0 )  σ(g 0 ) ≤ Clog(N ), on a la majoration |HL1 (l0 )|  log(N ). Il nous suffit de montrer que, pour tout c > 0, on a la minoration |(ζ(ym) − sζ(ym))L1 | > clog(N ) pourvu que c1 soit assez grand. Soit c > 0. Le même calcul qu’en (8) montre que l’on a une majoration β(ζ(ym)) > clog(N ) pour toute racine β de AMmin dans u1 , pourvu que c1 soit assez grand. A fortiori 0 β(ζ(ym)) > 0. Introduisons un élément Pmin ∈ P (Mmin ) tel que ζ(ym) ∈ A + . La 0 Pmin 0 relation précédente entraîne que Pmin ⊂ Q1 . Notons ∆0 la base de racines simples 0 0 associée à Pmin , (∆0 )L1 le sous-ensemble associé à L1 ∩ Pmin , {ˇ α; α ∈ ∆0 } l’ensemble de coracines associé à ∆0 , et {$α ; α ∈ ∆0 } la base de A Mmin duale de ∆0 . Ecrivons X ζ(ym) = α(ζ(ym))$α . α∈∆0

Tous les coefficients sont positifs ou nuls. On a X (ζ(ym) − sζ(ym))L1 = α(ζ(ym))($α − s$α )L1 . α∈∆0

On sait que, pour tout α ∈ ∆0 , $α − s$α est combinaison linéaire à coefficients positifs ou nuls de coracines βˇ pour β ∈ ∆0 . Donc ($α − s$α )L1 appartient au cône fermé engendré par les βˇL1 pour β ∈ ∆0 − (∆0 )L1 . Si α ∈ ∆0 − (∆0 )L1 , l’élément ($α − s$α )L1 n’est pas nul. En effet, s’il l’était, on aurait (s$α )L1 = ($α )L1 = $α . Puisque s$α est de même norme que $α , cela entraînerait s$α = (s$α )L1 = $α . Mais cette égalité est interdite par l’hypothèse (Hyp), d’où la conclusion. Des propriétés ci-dessus résulte une minoration X |(ζ(ym) − sζ(ym))L1 |  α(ζ(ym)). α∈∆0 −(∆0 )L1

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

282

J.-L. WALDSPURGER

Mais on a dit ci-dessus que α(ζ(ym)) > clog(N ) pour tous les α qui interviennent ici. La minoration cherchée en résulte, d’où (11). La fonction f5 est la transformée de Fourier d’une fonction C ∞ évaluée en ζ(xm0 , ym). Elle est donc à décroissance rapide en cette variable. Grâce à (11), pour tout entier D ≥ 0, on a une majoration |f5 (xm0 , ym)|  N −D pourvu que c1 soit assez grand. On a aussi Z Z Z |ΨN,Y,Q1 ,s1 ,s2 (g 0 )|  N −D K

Z

D(m)κN (k1 mk2 )

Mmin (F )+

K

Z

K1

K1

Q1 σM (ζ, Y )τQ1 (ζ − YQ1 )|f10 (m0 , x)f20 (m, y)|dy dx dm dk1 dk2 . min

On a −1

L1 |f20 (m, y)|  δQ1 (m)−1/2 δL ∩s

−1 2 Qs2

1

(ls2 (ym))1/2 Ξs2 Ls2 (ls2 (ym)).

D’après [16] lemmes II.1.6 et II.1.1, on en déduit l’existence d’un réel R8 ≥ 0 tel que Z |f20 (m, y)|dy  δQ1 (m)−1/2 ΞL1 (m)  ΞG (m)σ(m)R8 . K1

On a une majoration analogue pour la fonction f10 (m0 , x) et on obtient une majoration Z Z Z 0 −D |ΨN,Y,Q1 ,s1 ,s2 (g )|  N D(m)κN (k1 mk2 ) K G

0

G

0

K R8

Ξ (l m)Ξ (m)σ(l m)

Mmin (F )+

σ(m)R8 dm dk1 dk2 .

On a déjà majoré une intégrale de ce type : il existe R9 ≥ 0 tel qu’elle soit essentiellement majorée par N R9 . En tenant compte du facteur N −D et en se rappelant que D est quelconque, on obtient |ΨN,Y,Q1 ,s1 ,s2 (g 0 )|  N −R pourvu que c1 soit assez grand. C’est ce que l’on voulait démontrer. Supposons maintenant que l’hypothèse (Hyp) n’est pas vérifiée. Dans ce cas, on va montrer que ΦN,Y,Q1 ,s1 ,s2 (g 0 ) = 0. En se rappelant la définition de ce terme, on voit qu’il suffit de prouver que l’assertion suivante est vérifiée : (12) soient λ ∈ i A ∗L,F , m ∈ Mmin (F )+ et k1 , k2 ∈ K ; alors X Φs1 ,s2 (e, k1 mk2 , g 0 , λ) = 0. K

e∈ B O f

Notons X(λ) la somme ci-dessus. C’est une fonction méromorphe en λ (plus exactement, c’est la restriction à i A ∗L,F d’une fonction méromorphe sur ( A ∗L ⊗R C)/i A ∨ L,F ). Il suffit de montrer qu’elle est nulle pour presque tout λ. On peut donc supposer que tous les opérateurs d’entrelacement qui vont intervenir n’ont pas de pôles en λ et sont inversibles. Revenons à la définition de Φs1 ,s2 (e, k1 mk2 , g 0 , λ). On a une égalité Φs1 ,s2 (e, k1 mk2 , g 0 , λ) =

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

(JQ˜ 1,s

−1 1 |s1 Qs1

(e2 , JQ˜ 1,s

283

((s1 τ )s1 λ ) ◦ γ(s1 ) ◦ πλ (m0 k4 k2 )e, e1 )L1

−1 2 |s2 Qs2

((s2 τ )s2 λ ) ◦ γ(s2 ) ◦ πλ (mk2 )πλ (f )e)L1 ,

G 0 où e1 ∈ K G Q1,s1 ,s1 τ et e2 ∈ K Q1,s2 ,s2 τ . On a πλ (k2 )πλ (f )e = πλ (f )πλ (k2 )e, où

f0 =

k2

de ( K

Kf

f . Posons B\

G Kf Q,τ )

K

= {πλ (k2 )e; e ∈ B O f }. C’est encore une base orthonormée

et on a X X(λ) = (JQ˜ 1,s

1

((s1 τ )s1 λ ) |s1 Qs−1 1

◦ γ(s1 ) ◦ πλ (m0 k4 )e, e1 )L1

Kf

e∈ B\

(e2 , JQ˜ 1,s

2

((s2 τ )s2 λ ) |s2 Qs−1 2

◦ γ(s2 ) ◦ πλ (m)πλ (f 0 )e)L1 .

Il existe une fonction j1 (λ) qui est méromorphe, au même sens que ci-dessus, telle que JQ˜ 1,s

1

((s1 τ)s1 λ ) ◦ γ(s1 ) |s1 Qs−1 1

= j1 (λ)JQ˜ 1,s

˜

1 |sQ1,s2 s

−1

((s1 τ)s1 λ ) ◦ γ(s) ◦ JQ˜ 1,s

2

((s2 τ)s2 λ ) ◦ γ(s2 ). |s2 Qs−1 2

L’ensemble {JQ˜ 1,s

2

((s2 τ )s2 λ ) |s2 Qs−1 2

K

◦ γ(s2 )e; e ∈ B\ f }

Kf est une base de ( K G . Les propriétés d’adjonction et de composition des ˜ 1,s ,s2 τ ) Q 2 opérateurs d’entrelacement entraîne qu’elle est orthogonale et que tous ses éléments ont la même norme. Notons j2 (λ) cette norme et divisons tout élément de cette base p K Kf par j2 (λ). On obtient une base orthonormée de ( K G que l’on note B] f . ˜ 1,s ,s2 τ ) Q 2 On a l’égalité X 0 L1 X(λ) = j1 (λ)j2 (λ) (JQ˜ 1,s |sQ˜ 1,s s−1 ((s1 τ )s1 λ ) ◦ γ(s) ◦ IndG ˜ 1,s ((s2 τ )s2 λ , m k4 )e, e1 ) Q 1

2

2

Kf

e∈ B]

G 0 L1 (e2 , IndG ˜ 1,s ((s2 τ )s2 λ , f )e) . ˜ 1,s ((s2 τ )s2 λ , m)IndQ Q 2

2

A ce point, le sous-groupe parabolique Q = LUQ n’intervient plus (sauf via les fonc˜ 1,s . tions j1 et j2 ). Pour simplifier les notations, on peut supposer s2 = 1 et Q = Q 2 Auquel cas s1 = s et l’expression précedente se simplifie en X X(λ) = j1 (λ)j2 (λ) (JQ˜ 1,s |sQs−1 ((sτ )sλ ) ◦ γ(s) ◦ πλ (m0 k4 )e, e1 )L1 Kf

e∈ B]

(e2 , πλ (m)πλ (f 0 )e)L1 . Puisque l’hypothèse (Hyp) n’est pas vérifiée, on peut fixer un sous-groupe parabolique Q0 = L0 U 0 ∈ F (Mmin ) tel que Q1 ⊂ Q0 6= G et s fixe tout élément de A L0 . Cela 0 ¯1 ⊂ Q ¯ 0 , donc aussi Q ˜ 1,s ⊂ Q ¯ 0 . Introduisons les espaces entraîne s ∈ W L . On a Q ⊂ Q 0 0 K LL0 ∩Q,τ et K LL0 ∩Q˜ 1,s ,sτ . On dispose de l’opérateur 0

0

0

L JLL0 ∩Q˜ 1,s |L0 ∩sQs−1 ((sτ )sλ ) ◦ γ(s) : K L ˜ 1,s ,sτ . L0 ∩Q,τ → K L0 ∩Q

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

284

J.-L. WALDSPURGER

0

0

L Posons π 0 = IndL L0 ∩Q (τλ ) et réalisons cette représentation dans K L0 ∩Q,τ . Pour e ∈ KG ¯ 0 ,π 0 , posons Q Z 0 b1 (e) = δQ0 (m0 k4 )−1/2 ((JLL0 ∩Q˜ 1,s |L0 ∩sQs−1 ((sτ )sλ ) ◦ γ(s) ◦ π 0 (m0 k4 )(e(1)))(x), e1 (x))dx,

K1

b2 (e) = δQ0 (m)−1/2

Z

(e2 (x), (π 0 (m)(e(1)))(x))dx.

K1

Expliquons par exemple la signification de 0

(JLL0 ∩Q˜ 1,s |L0 ∩sQs−1 ((sτ )sλ ) ◦ γ(s) ◦ π 0 (m0 k4 )(e(1)))(x). 0

On évalue e au point 1. On obtient un élément e(1) de K L L0 ∩Q,τ . On applique successivement à cet élément les opérateurs π 0 (m0 k4 ) (notons que m0 k4 ∈ L1 (F ) ⊂ L0 (F )) 0 0 puis JLL0 ∩Q˜ |L0 ∩sQs−1 ((sτ )sλ ) ◦ γ(s). On obtient un élément de K L ˜ 1,s ,sτ , que l’on L0 ∩Q 1,s évalue au point x ∈ K1 ⊂ K ∩ L0 (F ). Définissons une forme sesquilinéaire B sur KG ¯ 0 ,π 0 par Q B(e0 , e) = b1 (e0 )b2 (e). G Identifions K G ¯ 0 ,π 0 . Modulo cette identification, on a les égalités Q,τ à K Q

(e2 , πλ (m)e)L1 = b2 (e), (JQ˜ 1,s |sQs−1 ((sτ )sλ ) ◦ γ(s) ◦ πλ (m0 k4 )e, e1 )L1 = b1 (e), pour tout e ∈ K G ¯ 0 ,π 0 . Alors Q 0 0 X(λ) = j1 (λ)j2 (λ)traceB (IndG ¯ 0 (π , f )). Q

Comme f , la fonction f 0 est très cuspidale. Puisque b1 (e) et b2 (e) ne dépendent que de e(1), la forme B vérifie la condition (3) de 2.1. Le lemme de ce paragraphe entraîne X(λ) = 0. Cela prouve (12) et achève la preuve de la proposition.

6.7. Utilisation des calculs spectraux d’Arthur. — Les données sont les mêmes que dans le paragraphe précédent. Pour tout ε > 0, on note D(ε) l’ensemble des éléments Y ∈ A + Pmin tels que inf{α(Y ); α ∈ ∆} > ε sup{α(Y ); α ∈ ∆}. 0

Pour L0 ∈ L (L) et t ∈ W L (L)reg , notons Λ O (t) l’ensemble des λ ∈ i A ∗L tels que ∗ t(τλ ) ' τλ . Cet ensemble est stable par translation par i A ∨ L,F + i A L0 . L’ensemble des orbites est fini. Soit λ un élément de cet ensemble. Arthur définit en [6] p.87 un signe, qu’il note εσ¯ (t) et que nous notons ετλ (t). Sa définition ne nous importe pas. Il ne dépend que de l’orbite de λ. On dispose de l’opérateur RQ (t, τλ ) de K G Q,τ . Soit Q0 = L0 U 0 ∈ P (L0 ). Posons Q(Q0 ) = (L0 ∩ Q)U 0 . C’est un élément de P (L).

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

285

Définissons sur i A ∗L0 une fonction ν 7→ jQ0 (t, λ, ν) et, pour g 0 ∈ G(F ), une fonction ν 7→ dQ0 (t, λ, g 0 , ν) par X jQ0 (t, λ, ν) = (e, JQ(Q¯ 0 )|Q (τλ )−1 JQ(Q¯ 0 )|Q (τλ+ν )RQ (t, τλ )πλ (f )e), K

e∈ B O f

dQ0 (t, λ, g 0 , ν) = (JQ(Q0 )|Q (τλ )−1 JQ(Q0 )|Q (τλ+ν )RQ (t, τλ )e00 , πλ (g 0 )e0 ) (e0 et e00 sont fixés comme dans le paragraphe précédent). Les opérateurs d’entrelacement peuvent avoir des pôles, mais, au moins pour λ dans un ouvert dense de Λ O (t), les fonctions ci-dessus sont bien définies pour ν proche de 0. Nous utiliserons une troisième fonction ν 7→ cQ0 (ν) : c’est celle qui est notée cQ¯ 0 (ν) en [6] (12.12). Il nous est inutile de rappeler sa définition. Les trois familles de fonctions (jQ0 (t, λ))Q0 ∈ P (L0 ) , (dQ0 (t, λ, g 0 ))Q0 ∈ P (L0 ) et (cQ0 )Q0 ∈ P (L0 ) sont des (G, L0 )-familles. De plus cQ0 (0) = 1 pour tout Q0 . Posons (jdc)Q0 (t, λ, g 0 ) = jQ0 (t, λ)dQ0 (t, λ, g 0 )cQ0 . La famille ((jdc)Q0 (t, λ, g 0 ))Q0 ∈ P (L0 ) est encore une (G, L0 )-famille, à laquelle on associe un nombre (jdc)L0 (t, λ, g 0 ). Admettons un instant que ce nombre soit fonction C ∞ de 0 λ. Rappelons par ailleurs que, puisque t ∈ W L (L)reg , l’opérateur t − 1 sur A L / A L0 est inversible et a donc un déterminant d´et(t − 1) A L / A L0 non nul. Posons X X X −1 ετλ (t)|d´et(t − 1) A L / A L0 |−1 Φ(g 0 ) = [i A ∨O ; i A ∨ L,F ] ∗ L0 ∈ L (L) t∈W L0 (L)reg λ∈Λ O (t)/(i A ∨ L,F +i A L0 )

Z iA

∗ L0 ,F

(jdc)L0 (t, λ + µ, g 0 )ϕ(λ + µ)dµ.

Proposition. — (i) Pour tout L0 ∈ L (L), la fonction λ 7→ (jdc)L0 (t, λ, g 0 ) est C ∞ . (ii) Soit ε > 0 et R ≥ 1 un entier. On a une majoration |ΦY (g 0 ) − Φ(g 0 )|  σ(g 0 )R ΞG (g 0 )|Y |−R pour tout g 0 ∈ G(F ) et tout Y ∈ D(ε) ∩ A Mmin ,F . Démonstration. — Fixons ε et R. Oublions d’abord g 0 , c’est-à-dire supposons g 0 = 1. Supposons aussi que la fonction ϕ est constante de valeur 1. Dans [6] p.69, Arthur étudie une expression qu’il note K T (f ). C’est une somme sur M ∈ L (Mmin ), σ ∈ {Π2 (M (F ))} d’expressions qui sont des intégrales sur i A ∗M,F × G(F ). Dans le cas où le T d’Arthur est notre Y et où le couple (M, σ) d’Arthur est égal à notre couple (L, τ ), cette expression est presque notre terme ΦY (1). Plus précisément, notre terme dépend d’éléments fixés e0 , e00 et le terme d’Arthur est égal à la somme de nos termes ΦY (1) associés à un nombre fini de couples (e0 , e00 ). Dans [6], proposition 11.3, p.88, ˜ ). C’est une somme sur les couples (M, σ) comme ci-dessus figure une expression J(f d’expressions qui, pour le couple (M, σ) = (L, τ ), sont presqu’égales à notre terme Φ(1). Plus exactement, ce terme est la somme de termes Φ(1) associés aux mêmes couples (e0 , e00 ) que ci-dessus. Arthur démontre en [6] corollaire 10.4 que les fonctions λ 7→ (jcd)L0 (t, λ, 1) sont C ∞ . Entre les pages 69 et 88 de [6], il démontre le résultat suivant. Il existe une fonction T 7→ J T (f ) qui vérifie les trois conditions

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

286

J.-L. WALDSPURGER

1 A Mmin ,F )/ A Mmin ,F , un polynôme (1) il existe un entier D ≥ 1 et, pour tout ξ ∈ ( D P T 7→ qξ (T ), de sorte que J T (f ) = ξ∈( D1 A M ,F )/ A M ,F exp(ξ(T ))qξ (T ) pour tout min min T ∈ A Mmin ,F ; ˜ ) = q0 (0) ; (2) J(f (3) on a une majoration |K T (f ) − J T (f )|  |T |−R pour tout T ∈ D(ε) ∩ A Mmin ,F . Cf. la preuve du lemme 11.1 de [6]. En inspectant la preuve, on voit d’une part que la somme sur les couples (M, σ) ne sert à rien dans ce passage : la preuve se fait terme par terme. D’autre part sommer sur un ensemble fini de couples (e0 , e00 ) ne sert à rien non plus, la même preuve s’applique pour chaque couple. En revenant à nos notations, cette preuve montre donc qu’il existe une fonction Y 7→ ΦY sur A Mmin ,F , de la forme X (4) ΦY = exp(ξ(Y ))qξ (Y ), 1 ξ∈( D

A Mmin ,F )/ A Mmin ,F

telle que Φ(1) = q0 (0) et telle que l’on ait une majoration (5)

|ΦY (1) − ΦY |  |Y |−R

pour tout Y ∈ D(ε) ∩ A Mmin ,F . Montrons qu’en fait (6) q0 est constant et qξ = 0 si ξ 6= 0. Remarquons que, pour Y ∈ D(ε), on a des majorations |Y |  sup{α(Y ); α ∈ ∆}  |Y |. On peut donc remplacer sup{α(Y ); α ∈ ∆} par |Y | dans l’énoncé de la proposition 6.6. Fixons c1 et c2 vérifiant cet énoncé modifié. Soit Y ∈ D(ε) ∩ A Mmin ,F . Notons NY la partie entière de 2c−1 2 |Y | + 1. Si |Y | est assez grand, le couple (NY , Y ) vérifie les conditions de la proposition 6.6. Plus généralement, il en est de même du couple (NY , Y 0 ) pour tout Y 0 ∈ D(ε) ∩ A Mmin tel que |Y − Y 0 | ≤ |Y |/2. Pour un tel Y 0 , on a donc |ΦY 0 (1) − ΦY (1)|  |ΦY 0 (1) − ΦNY (1)| + |ΦNY (1) − ΦY (1)|  NY−R  |Y |−R . En appliquant (5), on a aussi 0

|ΦY − ΦY |  |Y |−R . Si Y 7→ ΦY ne vérifie pas (6), c’est-à-dire n’est pas constante, on peut fixer Y0 ∈ A Mmin ,F tel que la fonction Y 7→ ΦY +Y0 − ΦY soit non nulle. Cette fonction est encore de la forme (4). Pour Y assez grand, le point Y 0 = Y + Y0 vérifie |Y − Y 0 | ≤ |Y |/2. La fonction est donc essentiellement majorée par |Y |−R . Mais une fonction de la forme (4) ne peut vérifier cette majoration que si elle est nulle. Cela contredit le choix de Y0 , d’où (6). Grâce à (6), on a Φ(1) = ΦY et la majoration (5) est celle que l’on voulait prouver. Si ϕ n’est plus la fonction constante de valeur 1, on inspecte la preuve d’Arthur et on voit que l’on peut glisser la fonction ϕ tout le long de cette preuve. Le résultat est celui annoncé. Bien sûr, la constante qui se trouve implicitement dans la majoration de l’énoncé dépend de ϕ. Cela ne nous gêne pas pourvu que ϕ soit fixée mais, pour

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

287

la suite du raisonnement, précisons tout-de-même cette dépendance. Pour tout entier k ≥ 0, fixons une base X k de l’espace des opérateurs différentiels sur i A ∗L , à coefficients constants et de degré ≤ k. Posons |ϕ|k = sup{|(Xϕ)(λ); λ ∈ i A ∗L,F , X ∈ X k }. Dans la preuve d’Arthur, les approximations interviennent à deux endroits. D’abord dans l’utilisation du théorème 8.1. Cette approximation porte uniquement sur l’intégrale intérieure sur G(F ). Quand on intègre ensuite sur i A ∗L,F , la constante implicite est simplement multipliée par |ϕ|0 . Il y a ensuite les approximations de la page 80 qui se réfèrent elles-mêmes à [3], p. 1306,1307. D’après cette dernière référence, la constante implicite est de la forme sup{(Ψϕ)ˆ(Z)|Z|R ; Z ∈ A L,F }, où Ψ est une certaine fonction C ∞ sur i A ∗L,F indépendante de ϕ et (Ψϕ)ˆest la transformée de Fourier de Ψϕ. Comme on le sait bien, le terme ci-dessus est essentiellement borné par |ϕ|R . Finalement, la majoration de l’énoncé se précise en (7)

|ΦY (1) − Φ(1)| ≤ c|ϕ|R |Y |−R

où c est indépendant de ϕ. Passons au cas général où g 0 est quelconque. Fixons un sous-groupe ouvert compact K0 de K tel que e00 soit invariant par K0 . On utilise le fait que la fonction g 7→ u(g, Y ) est invariante à gauche par K (ce que n’était pas la fonction g 7→ κN (g) des paragraphes précédents). Dans la définition de ΦY (g 0 ) donnée au début du paragraphe 6.6, on remplace la variable g par kg et on intègre sur k ∈ K0 , en divisant le tout par mes(K0 ). On obtient une expression analogue à ΦY (g 0 ), où le terme πλ (g 0 )e0 est remplacé par Z mes(K0 )−1 πλ (kg 0 )e0 dk. K0 K0

K0 Fixons une base orthonormée B O de ( K G . On peut encore remplacer le terme Q,τ ) ci-dessus par X (e0 , πλ (g 0 )e0 )e0 . K

e0 ∈ B O 0

Alors ΦY (g 0 ) est une somme de termes analogues où le triplet (g 0 , e0 , ϕ) est remplacé par (1, e0 , ϕ0 ), où ϕ0 (λ) = ϕ(λ)(e0 , πλ (g 0 )e0 ). Une décomposition analogue vaut pour le terme Φ(g 0 ). En utilisant la majoration (7), on voit que, pour obtenir la majoration de l’énoncé, il nous reste à prouver que l’on a une majoration |ϕ0 |R  σ(g 0 )R ΞG (g 0 )

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

288

J.-L. WALDSPURGER

pour toute fonction ϕ0 comme ci-dessus. Il suffit de prouver une majoration analogue pour la fonction ϕ00 (λ) = (e0 , πλ (g 0 )e0 ). On a Z ϕ00 (λ) = (e0 (k), τ (lQ (kg 0 ))e0 (kQ (kg 0 ))δQ (lQ (kg 0 ))1/2 exp(λ(HQ (kg 0 )))dk. K

Appliquer un opérateur différentiel X ∈ X R revient à glisser dans l’intégrale un terme P (HQ (kg 0 )), où P est un polynôme de degré ≤ R. On a une majoration |HQ (kg 0 )|  σ(kg 0 ) = σ(g 0 ). D’où les majorations |ϕ00 |  σ(g 0 )R

Z

|(e0 (k), τ (lQ (kg 0 ))e0 (kQ (kg 0 ))|δQ (lQ (kg 0 ))1/2 dk

K 0 R

Z

 σ(g )

ΞL (lQ (kg 0 ))δQ (lQ (kg 0 ))1/2 dk

K

 σ(g 0 )R ΞG (g 0 ) d’après [16] lemme II.1.6. C’est ce que l’on voulait démontrer. 6.8. Simplification de Φ(g 0 ). — Pour L0 ∈ L (L) et λ ∈ i A ∗L , on définit les groupes 0 0 0 W L (τλ ), (W L )0 (τλ ) et RL (τλ ) : ce sont les analogues de W (τλ ), W 0 (τλ ) et R(τλ ) 0 0 quand on remplace G par L0 . Notons ΛLO,ell l’ensemble des λ ∈ i A ∗L tels que RL (τλ ) ∩ 0

∗ W L (L)reg 6= ∅. Cet ensemble est stable par translations par i A ∨ L,F + i A L0 . Soit λ 0 un élément de cet ensemble. On a (partiellement) décrit le groupe RL (τλ ) en 4.1. 0 0 Notons RL (τλ )∨ son groupe dual. La représentation IndL L0 ∩Q (τλ ) se décompose en 0

0

L ∨ somme de sous-représentations irréductibles IndL L0 ∩Q (τλ , ζ) pour ζ ∈ R (τλ ) . On a conformément la décomposition en somme orthogonale 0

0

K LL0 ∩Q,τ = ⊕ζ∈RL0 (τλ )∨ K LL0 ∩Q,τλ ,ζ . Fixons S 0 = L0 U 0 ∈ P (L0 ) et, comme dans le paragraphe précédent, notons Q(S 0 ) = G (L0 ∩ Q)U 0 ∈ P (L). L’opérateur RQ(S 0 )|Q (τλ ) est une isométrie de K G Q,τ sur K Q(S 0 ),τ . 0

D’autre part, ce dernier espace s’identifie à un espace de fonctions de K dans K L L0 ∩Q,τ . La décomposition ci-dessus de cet espace induit une décomposition orthogonale (1)

G KG Q(S 0 ),τ = ⊕ζ∈RL0 (L) K Q(S 0 ),τλ ,ζ .

G Notons projλ,ζ la projection de K G Q(S 0 ),τ sur K Q(S 0 ),τλ ,ζ . Remarquons que ces sousespaces et ces projections ne dépendent que de l’orbite de λ. Rappelons d’autre part que l’on note aL0 la dimension de A L0 .

Lemme. — Pour tout g 0 ∈ G(F ), on a l’égalité X X −1 Φ(g 0 ) = [i A ∨O : i A ∨ (−1)aL0 L,F ] L0 ∈ L (L)

ASTÉRISQUE 346

0

∗ λ∈ΛLO,ell /(i A ∨ L,F +i A L0 )

0

|RL (τλ )|2aL0 −aL

X ζ∈RL0 (τλ )∨

289

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

Z iA

∗ L0 ,F

0 0 (projλ,ζ ◦ RQ(S 0 )|Q (τλ+µ )e00 , projλ,ζ ◦ RQ(S 0 )|Q (τλ+µ ) ◦ IndG Q (τλ+µ , g )e ) 0

L JLG0 (IndL 0 ∩Q (τλ+µ , ζ), f )ϕ(λ + µ)dµ. 0

Démonstration. — Fixons L0 ∈ L (L), t ∈ W L (L)reg et λ ∈ Λ O (t). Considérons le terme (jdc)L0 (t, λ, g 0 ). On commence par remplacer, dans les définitions des (G, L0 )-familles (jQ0 (t, λ))Q0 ∈ P (L0 ) et (dQ0 (t, λ, g 0 ))Q0 ∈ P (L0 ) , les opérateurs d’entrelacement par des opérateurs normalisés. Cela multiplie ces familles par des (G, L0 )-familles à valeurs scalaires. Quitte à multiplier la (G, L0 )-famille (cQ0 )Q0 ∈ P (L0 ) par ces familles, on retrouve une expression similaire à celle de départ (la famille qui remplace (cQ0 )Q0 ∈ P (L0 ) dépend de λ). On peut donc considérer que l’on a X L0 jQ0 (t, λ, ν) = (e, RQ(Q¯ 0 )|Q (τλ )−1 RQ(Q¯ 0 )|Q (τλ+ν )RQ (t, τλ )πλ (f )e). K

e∈ B O f K

L’ensemble {RQ(S 0 )|Q (τλ )(e); e ∈ B O f } est une base orthonormée de K G Q(S 0 ),τ . Kf Notons-la B O,Q(S 0 ) . Posons X G 0 L0 jQ (e, RQ(Q¯ 0 )|Q(S 0 ) (τλ )−1 RQ(Q¯ 0 )|Q(S 0 ) (τλ+ν )RQ(S 0 (t, λ, ν) = 0 ) (t, τλ )IndQ(S 0 ) (τλ , f)e). K

f e∈ B O,Q(S 0)

On a l’égalité 0 jQ 0 (t, λ, ν) =

X

0

L (e, RQ(Q¯ 0 )|Q (τλ )−1 RQ(Q¯ 0 )|Q (τλ+ν )RQ (t, τλ )πλ (f )r(λ, ν)e),

K

e∈ B O f

où r(λ, ν) = RQ(S 0 )|Q (τλ+ν )−1 RQ(S 0 )|Q (τλ ). Le point est que cet opérateur ne dépend pas de Q0 et que r(λ, 0) est l’identité. Une propriété familière des (G, L0 )-familles entraîne que l’on a l’égalité 00

00

Q (j 0 )Q L0 (t, λ) = jL0 (t, λ)

pour tout Q00 ∈ F (L0 ). D’après les formules de descente, on peut donc remplacer la 0 famille (jQ0 (t, λ))Q0 ∈ P (L0 ) par (jQ 0 (t, λ))Q0 ∈ P (L0 ) . De la même façon, on peut remplacer la famille (dQ0 (t, λ, g 0 ))Q0 ∈ P (L0 ) par (d0Q0 (t, λ, g 0 ))Q0 ∈ P (L0 ) , où d0Q0 (t, λ, g 0 , ν) = (RQ(Q0 )|Q(S 0 ) (τλ )−1 RQ(Q0 )|Q(S 0 ) (τλ+ν )RQ(S 0 ) (t, τλ )RQ(S 0 )|Q (τλ )e00 , 0 0 RQ(S 0 )|Q (τλ )IndG Q (τλ , g )e ). 0 Autrement dit, quitte à remplacer les éléments IndG et e00 par Q (τλ )e G 0 00 RQ(S 0 )|Q (τλ )IndQ (τλ )e et RQ(S 0 )|Q (τλ )e , on peut remplacer le parabolique Q par Q(S 0 ). Les considérations qui précèdent l’énoncé sont valables indépendamment 0 0 de l’hypothèse RL (τλ ) ∩ W L (L)reg 6= ∅. On a la décomposition (1). On peut supKf poser que la base B O,S 0 est réunion de bases des différents sous-espaces. L’opérateur 0

L RQ(S 0 ) (t, τλ ) agit par ζ(t) sur le sous-espace K G Q(S 0 ),τ,ζ (où le caractère ζ de R (τλ )

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

290

J.-L. WALDSPURGER

0

0

est étendu en un caractère de W L (τλ ) trivial sur (W L )0 (τλ )). On en déduit une égalité X 0 0 ζ(t)jQ jQ 0 (t, λ, ν) = 0 (λ, ζ, ν), ζ∈RL0 (τλ )∨

avec une définition plus ou moins évidente de ce dernier terme. On a alors X 00 0 ¯ 00 aL0 −aL00 ζ(t)JLQ0 (IndL (j 0 )Q L0 ∩Q (τλ , ζ), f ) L0 (t, λ) = (−1) ζ∈RL0 (τλ )∨

¯ 00 en exposant pour tout Q00 = L00 U 00 ∈ F (L0 ). Le signe (−1)aL0 −aL00 ainsi que le Q 0 ¯ viennent de ce que c’est le parabolique Q(Q ) qui intervient dans la définition de 0 0 0 L0 jQ (τλ ) ∩ W L (L)reg = ∅, c’est-à-dire λ 6∈ ΛLO,ell . 0 (t, λ, ν), cf. [6] p.92. Supposons R 0

Dans ce cas, toutes les représentations IndL L0 ∩Q (τλ , ζ) sont combinaisons linéaires d’induites propres et les termes ci-dessus sont nuls (lemme 2.2(ii)). Grâce à la formule de descente, on obtient 0 (2) (jdc)(t, λ, g 0 ) = 0 si λ 6∈ ΛLO,ell . 0

0

0

Supposons maintenant RL (τλ ) ∩ W L (L)reg 6= ∅, c’est-à-dire λ ∈ ΛLO,ell . Le lemme 2.2(i) et la formule de descente entraîne l’égalité 0 0 (jdc)(t, λ, g 0 ) = (j 0 )G L0 (t, λ)dQ0 (t, λ, g , 0)cQ0 (λ, 0)

où Q0 est un élément quelconque de P (L0 ). On a cQ0 (λ, 0) = 1, 0 d0Q0 (t, λ, g 0 , 0) = (RQ(S 0 ) (t, τλ )RQ(S 0 )|Q (τλ )e00 , RQ(S 0 )|Q (τλ )IndG Q (τλ )e )

et X

aL0 (j 0 )G L0 (t, λ) = (−1)

0

ζ(t)JLG0 (IndL L0 ∩Q (τλ , ζ), f ).

ζ∈RL0 (τλ )∨ 0

0

0

Rappelons que l’hypothèse RL (τλ ) ∩ W L (L)reg 6= ∅ entraîne que (W L )0 (τλ ) = {1}, 0 0 0 0 donc RL (τλ ) = W L (τλ ). De plus, RL (τλ ) ∩ W L (L)reg possède un unique élément. 0 Puisque t appartient à cette intersection, cet unique élément est t. Soit x ∈ RL (τλ ), x 6= t. Considérons la représentation virtuelle X 0 ζ(x)IndL L0 ∩Q (τλ , ζ). ζ∈RL0 (τλ )∨

D’après [7] proposition 2.1(b), c’est une somme, à coefficients dans Z, de représentations induites. D’après le lemme 2.2(ii), on a donc X 0 ζ(x)JLG0 (IndL L0 ∩Q (τλ , ζ), f ) = 0. ζ∈RL0 (τλ )∨ 0

Il en résulte que ζ(t)JLG0 (IndL L0 ∩Q (τλ , ζ), f ) est indépendant de ζ. On en déduit l’égalité 0

0

aL0 (j 0 )G |RL (τλ )|ζ(t)JLG0 (IndL L0 (t, λ) = (−1) L0 ∩Q (τλ , ζ), f )

ASTÉRISQUE 346

UNE FORMULE INTÉGRALE RELIÉE À LA CONJ. LOCALE DE GROSS-PRASAD

291

0

pour tout ζ ∈ RL (τλ )∨ . D’autre part, on peut décomposer les éléments 0 00 RQ(S 0 )|Q (τλ )IndG Q (τλ )e et RQ(S 0 )|Q (τλ )e selon la décomposition (1). Il en résulte l’égalité X 0 0 d0Q0 (t, λ, g 0 , 0) = ζ(t)(projλ,ζ ◦ RQ(S 0 )|Q (τλ )e00 , projλ,ζ ◦ RQ(S 0 )|Q (τλ ) ◦ IndG Q (τλ , g )e ). ζ∈RL0 (τλ )∨

Des deux égalités précédentes résulte la relation 0 (3) si λ ∈ ΛLO,ell , 0

(jdc)L0 (t, λ, g 0 ) = (−1)aL0 |RL (τλ )|

X ζ∈RL0 (τλ )∨ 0

L 0 0 G (projλ,ζ ◦ RQ(S 0 )|Q (τλ )e00 , projλ,ζ ◦ RQ(S 0 )|Q (τλ ) ◦ IndG Q (τλ , g )e )JL0 (IndL0 ∩Q (τλ , ζ), f ). 0

D’autre part, le signe ετλ (t) vaut 1 parce que (W L )0 (τλ ) = {1}, cf. [6] p.95. Enfin, on a décrit t en 4.2 et on voit que |d´et(t − 1) A L / A L0 |−1 = 2aL0 −aL . Il suffit de reporter ces égalités et celles des relations (2) et (3) dans la définition de Φ(g 0 ) pour obtenir l’égalité de l’énoncé. 6.9. Evaluation d’une limite Lemme. — On a l’égalité −1 limN →∞ JL, O,N,C (θρ , f ) = [i A ∨O : i A ∨ L,F ]

X L0 ∈ L (L)

0

Z

X

|RL (τλ )|2aL0 −aL

0

ζ∈RL0 (τλ )∨ ;m(IndL (τ ,ζ),ρ)=1 L0 ∩Q λ

X

(−1)aL0 0

∗ λ∈ΛL /(i A ∨ L,F +i A L0 ) O,ell 0

i A∗ L0 ,F

JLG0 (IndL L0 ∩Q (τλ+µ , ζ), f )dµ.

0

L Remarque. — Le nombre m(IndL 0 ∩Q (τλ , ζ), ρ) a été défini en 6.1.

Démonstration. — Considérons la définition de JL, O,N,C (θρ , f ) donnée avant le lemme 6.4. Il y intervient des objets εj , ε0j , ej , e0j et ϕj pour j = 1, . . . , n. Dans les paragraphes précédents, on a introduit des fonctions ΦN (g 0 ), ΦY (g 0 ) et Φ(g 0 ) qui dépendaient de choix d’éléments e0 , e00 et d’une fonction ϕ. On note ΦN,j (g 0 ), ΦY,j (g 0 ), Φj (g 0 ) ces fonctions relatives à e0 = ej , e00 = e0j , ϕ = ϕj . On a alors X Z ¯ JL, O,N,C (θρ , f ) = 1σ 0 que nous préciserons par la suite. Introduisons des constantes c1 , c2 qui vérifient les conditions de la proposition 6.6 pour chaque couple de fonctions (ΦN,j (g 0 ), ΦY,j (g 0 )). Il y a une constante c3 > 0 telle que, pour tout N , il existe Y ∈ A Mmin ,F tel que c3 N < α(Y ) < c2 N pour tout α ∈ ∆. Fixons un tel c3 et, pour tout N , un élément YN vérifiant ces inégalités. Si N est assez grand, YN vérifie les hypothèses de la proposition 6.6 et celles

SOCIÉTÉ MATHÉMATIQUE DE FRANCE 2012

292

J.-L. WALDSPURGER

de la proposition 6.7 (pour chacune de nos fonctions ΦN,j (g 0 ) etc.). Ces propositions entraînent que l’on a une majoration |ΦN,j (g 0 ) − Φj (g 0 )|  (1 + σ(g 0 )R ΞG (g 0 ))N −R pour tout j, tout N assez grand et tout g 0 ∈ G(F ) tel que 1σ