UNITARY SHIMURA CORRESPONDENCES FOR SPLIT REAL GROUPS

J. ADAMS, D. BARBASCH, A. PAUL, P. TRAPA, AND D. A. VOGAN, JR.

Abstract. We find a relationship between certain complementary series representations for nonlinear coverings of split simple groups, and spherical complementary series for (different) linear groups. The main technique is Barbasch’s method of calculating some intertwining operators purely in terms of the Weyl group.

1. Introduction

Let G be the split real form of a connected reductive algebraic group 𝔾. Let θ be a Cartan involution of G and K = G^{θ} the corresponding maximal compact subgroup. Let B be a Borel subgroup of G with unipotent radical N. Then

A = B ∩ θB

is a θ-stable split Cartan subgroup of G. Notice the departure from standard notation for real reductive groups, in which A often refers to the identity component of a maximal split torus. We will write this identity component as A^{0}. In our notation, the Iwasawa decomposition is G = KA^{0}N. Write

M = B ∩ K = A ∩ K ≃ ℤ_{2}^{n};

then

B = AN = MA^{0}N.

Let 𝔞 = Lie(A). For ν ∈ 𝔞_{ℂ}^{∗} = Hom_{ℝ}(𝔞, ℂ), the character e^{ν} of A^{0} extends naturally to a character of B trivial on MN. Let

I_{G}(ν) = Ind_{B}^{G}(e^{ν})

(normalized induction) be the corresponding induced representation, a (possibly non-unitary) spherical principal series representation. Let J_{G}(ν) be the irreducible spherical composition factor of I_{G}(ν), i.e., the unique irreducible composition factor containing the trivial representation K. Define the spherical complementary series of G:

CS(G) = {ν ∈ 𝔞_{ℂ}^{∗} | J_{G}(ν) is unitary} ⊂ 𝔞_{ℂ}^{∗}.

1991 Mathematics Subject Classification. Primary 22E46.

Key words and phrases. Complementary series.

The first author was supported in part by NSF grant 0532393.

The second author was supported in part by NSF grants 0070561 and 0300172.

The fourth author was supported in part by NSF grant 0300106.

The fifth author was supported in part by NSF grants 9721441 and 0532088.

This work began during a visit in 2002 to the Institute for Mathematical Sciences, National University of Singapore. The visit was supported by the Institute and the National University of Singapore. We are grateful to our colleagues at NUS for their generous hospitality.

(More precisely, we require that J_{G}(ν) be infinitesimally equivalent to a unitary representation.) This is a closed, Weyl group-invariant subset of the complex vector space 𝔞_{ℂ}^{∗}.

Apart from the exceptional groups of Type E, CS(G) has been explicitly computed ([V2], [V3], [Ba1], [Ci]).

The goal of this paper is to consider a natural analogue of the spherical complementary series for certain nonlinear groups, and to relate them to spherical complementary series for linear groups. We begin by introducing the nonlinear groups in question.

Definition 1.1. Assume that 𝔾 is simple and simply connected. In this case the split real form G is a connected Lie group. It turns out that G admits a unique nontrivial two–fold cover G̃. That is, we have a short exact sequence of Lie groups

1 → {±1} → G̃ → G → 1

with G̃ connected. (Such coverings have been studied by many people for a long time. There are general results about coverings of algebraic groups in [St]. A detailed description of coverings of real algebraic groups may be found in [A].) The group G̃ is the nonlinear group we wish to study.

Write H̃ for the inverse image in G̃ of a subgroup H of G. A representation π of H̃ is called genuine if π(−1) = −I. If π is irreducible, then π is genuine if and only if π does not factor to H.

The subgroups A^{0} and N of B are connected and simply connected and hence lift to subgroups of G̃. We therefore have a semidirect product decomposition

B̃ = ÃN = M̃A^{0}N

with N normal. Fix an irreducible genuine representation δ̃ of M̃. We also assume that δ̃ is pseudospherical (Definition 4.9); this additional assumption is automatic except for Sp(2n, ℝ) and Spin(2n + 1, 2n). For any ν ∈ 𝔞^{∗}, δ̃ ⊗ e^{ν} defines a representation of B̃. The pseudospherical principal series representation is

IG̃(δ̃, ν) = IndG̃ ; B̃(δ̃ ⊗ e^{ν}).

Just as in the spherical case, this principal series representation has a distinguished irreducible composition factor JG̃(δ̃, ν) containing a certain K̃–type (Proposition 5.2). Define the pseudospherical complementary series of G̃:

CS(δ̃, G̃) = {ν ∈ 𝔞_{ℂ}^{∗} | JG̃(δ̃, ν) is unitary}.

Again this is a closed Weyl group-invariant subset of the vector space 𝔞_{ℂ}^{∗}. It is independent of the choice of pseudospherical representation δ̃ (cf. Lemma 5.4).

We are interested in comparing CS(δ̃, G̃) to the spherical complementary series of a linear group G^{ℓ}. For example if G = SL(2, ℝ) then (if we identify 𝔞_{ℂ}^{∗} with ℂ in such a way that the roots are ±2)

CS(G) = iℝ ∪ [−1, 1].

Here the imaginary axis corresponds to unitarily induced spherical principal series representations, the open interval (−1, 1) to Bargmann’s complementary series representations, and the endpoints ±1 to the trivial representation. Similarly,

CS(δ̃, G̃) = iℝ ∪ [−1/2, 1/2].

In this case the endpoints correspond to a summand of a metaplectic representation of S̃L(2, ℝ). The two sets CS(G) and CS(δ̃, G̃) are identical except for a scale factor of 1/2.

In order to formulate such a comparison in general, we must first describe the linear group G^{ℓ}. What we seek to achieve is that representation theory on the nonlinear group G̃ should be formally similar to representation theory on the linear group G^{ℓ}. (The most important point will be to relate the reducibility hyperplanes for the pseudospherical principal series IG̃(δ̃, ν) to the reducibility hyperplanes of the spherical principal series I_{G^{ℓ}}(ν^{ℓ}).) We therefore refer to G^{ℓ} as a linear reincarnation of G̃. It will be a split real form of a simple algebraic group 𝔾^{ℓ}, with a split Borel subgroup and maximal torus 𝔹^{ℓ} ⊃ 𝔸^{ℓ}.

Write B^{ℓ}, A^{ℓ}, M^{ℓ}, and δ^{ℓ} (an arbitrary character of M^{ℓ}) for the objects on G^{ℓ} corresponding to B, A, M, and δ on G. We say a principal series representation I_{G^{ℓ}}(δ^{ℓ}, ν^{ℓ}) of G^{ℓ} is pseudospherical if its restriction to the identity component (G^{ℓ})^{0} of G^{ℓ} is spherical. The corresponding complementary series CS(δ^{ℓ}, G^{ℓ}) is independent of δ^{ℓ}, and is equal to the spherical complementary series CS((G^{ℓ})^{0}).

The construction of 𝔾^{ℓ} will provide a natural covering homomorphism

(1.2)  Ψ: 𝔸^{ℓ} → 𝔸.

The differential dΨ is an isomorphism which carries each coroot of 𝔾^{ℓ} to a multiple (usually two) of a coroot of 𝔾. In particular, dΨ identifies the Weyl groups W and W^{ℓ}.

The group G^{ℓ} and the homomorphism Ψ are analogues of the construction of [Sa] and [Sa2] in the p-adic case. See Remark 7.3.

Here is our main conjecture.

Conjecture 1.3. Suppose G̃ is a double cover of a split real form of a simply connected complex algebraic group (Definition 1.1). Let G^{ℓ} be the split linear group described in Definitions 1.6 and 1.7. Fix a pseudospherical representation δ^{ℓ} of M^{ℓ}, and a genuine pseudospherical representation δ̃ of M̃. Then dΨ^{t} (cf. (1.2)) induces a bijection

CS(δ̃, G̃) → CS(δ^{ℓ}, G^{ℓ}).

In the case of one root length or G_{2}, G^{ℓ} is locally isomorphic to G and dΨ is multiplication by 2. If we replace G^{ℓ} with G (which does not affect Conjecture 1.3), then the correspondence takes IG̃(δ̃, ν) to I_{G}(2ν). In this case the conjecture says

JG̃(δ̃, ν) is unitary ⇔ J_{G}(2ν) is unitary (G simply laced or G_{2}).

This is known a posteriori for SL(n) [Hu1], [V2] and G_{2} [V3]; the case of SL(2) follows from the classical description of complementary series given above. In the case of two root lengths differing by a factor of two, G^{ℓ} is locally isomorphic to the dual group of G.

Theorem 1.4. If G is a classical group then dΨ^{t} induces an injection

CS(δ̃, G̃) ↪ CS(δ^{ℓ}, G^{ℓ})

The case of G = Sp(2n, ℝ) is particularly interesting. In this case G^{ℓ} = SO(n + 1, n); Ψ : A^{ℓ} ≃ A is an isomorphism, given by the identity map in the usual coordinates. (One can find in Chapitre 6 of [Bo] an embedding of each simple root system in a Euclidean space ℝ^{n}. It is these embeddings that we call “usual coordinates.”) The correspondence J(δ̃, ν) → J(ν^{ℓ}) agrees with the Howe dual pair correspondence. In this case we can prove Conjecture 1.3, which takes the following form. Recall that the maximal compact subgroup K of Sp(2n, ℝ) is U(n). Its preimage K̃ in S̃p(2n, ℝ) has a genuine character det^{1/2} whose square is the determinant character of U(n). This character of K̃ is the representation mentioned in Definition 1.1 as characterizing the distinguished composition factor J(δ̃, ν) of I(δ̃, ν). That is, the irreducible representations of S̃p(2n, ℝ) containing det^{1/2} are precisely the various J(δ̃, ν).

Theorem 1.5. Suppose π is an irreducible representation of S̃p(2n, ℝ) containing the K̃-type det^{1/2}. Let θ(π) denote the representation of SO(n+1, n) corresponding to π via the Howe correspondence. Then π is unitary if and only if θ(π) is unitary.

This paper uses many of the ideas of [Ba1], and uses the main result of that paper in the proof of Theorem 1.4. The main technique is to reduce the question of unitarity of a representation π to the computation of the signature of its Hermitian form restricted to the K-isotypic subspaces for certain “petite” representations of K (Definition 4.9). On these K-types the Hermitian form is given by a formal computation involving representations of Weyl groups. The resulting Weyl group calculations for G̃ and G^{ℓ} are closely related, and this is enough to give the injection of Theorem 1.4. This is all discussed in Section 13. For S̃p(2n) we use the dual pair correspondence to obtain the reverse inclusions (Section 14).

We conclude this introduction with a little more information about the definition of G^{ℓ}. Here is a precise description in case G is simply laced: that is, in case the roots of G all have the same length. The description applies equally to groups of type G_{2}.

Definition 1.6. Suppose 𝔾 is simple and simply connected, with maximal torus 𝔸 having Lie algebra 𝔞. The cocharacter lattice of 𝔸 is

X_{∗}(𝔸) = R^{∨} ⊂ 𝔞,

the lattice of coroots. The character lattice is

X^{∗}(𝔸) = P ⊂ 𝔞^{∗},

the lattice of integral weights. The root lattice R is a sublattice in P of finite index, and the integral coweight lattice P^{∨} contains R^{∨} with finite index.

An algebraic group covered by the simply connected group 𝔾 is specified precisely by its cocharacter lattice L. This can be any lattice between R^{∨} and P^{∨}:

R^{∨} ⊂ L ⊂ P^{∨}

When 𝔾 is simply laced or of type G_{2}, we define 𝔾^{ℓ} to be the group covered by 𝔾 and having cocharacter lattice

L = \frac{1}{2}R^{∨} ∩ P^{∨}.

Clearly L contains R^{∨} (since R^{∨} is contained in both \frac{1}{2}R^{∨} and P^{∨}), and L is contained in P^{∨}. Write 𝔸^{ℓ} for the torus of 𝔾^{ℓ} with cocharacter lattice L. Multiplication by 2 defines a W -equivariant inclusion

Ψ: L → R^{∨},

and therefore by functoriality an algebraic map of tori

Ψ: 𝔸^{ℓ} → 𝔸.

This is the map of (1.2). Write G^{ℓ} for a split real form of 𝔾^{ℓ}, containing the split real form A^{ℓ} of 𝔸^{ℓ}. The mapping Ψ defines

Ψ: A^{ℓ} → A,

which carries M^{ℓ} into M and (A^{ℓ})^{0} isomorphically onto A^{0}, in a W -equivariant way. Composition with the differntial dΨ defines a map on characters in the other direction:

dΨ^{t}: 𝔞^{∗} → (𝔞^{ℓ})^{∗}.

If we identify 𝔞 with 𝔞^{ℓ} (as we may, since the corresponding groups are locally isomorphic) then dΨ^{t} is just multiplication by 2.

In the same way, we have a map on characters

Ψ^{t}: {characters of M} → {characters of M^{ℓ}}.

We will see in Lemma 6.4 below that Ψ^{t} induces a natural simply transitive action of the group of pseudospherical characters of M^{ℓ} on the set of pseudospherical genuine representations of M̃. If we fix a single pseudospherical genuine representation of M̃, this means that we get a bijection

{genuine pseudosph. chars. of M̃} ↔ {pseudosph. chars. of M^{ℓ}}.

This bijection can be incorporated into Conjecture 1.3; it adds nothing to the difficulty of proving the conjecture, but makes the statement more attractive. (One may hope that the bijection will also be relevant to global statements relating automorphic forms on linear and nonlinear groups.)

The description of G^{ℓ} in the remaining cases (Definition 6.1 below) is more complicated; here are some of the main points.

Definition 1.7. Suppose 𝔾 is a simply connected simple group not of type G_{2}; that is, that any two root lengths differ by a factor of one or two. We will define in Section 6 a connected simple algebraic group 𝔾^{ℓ} with root system dual to that of 𝔾. We let G^{ℓ} be a split real form. Because G^{ℓ} has the dual root system to that of G, the Weyl groups W and W^{ℓ} may be identified. The construction of 𝔾^{ℓ} will provide a natural Weyl group-equivariant covering map

Ψ: A^{ℓ} → A.

Just as in the simply laced case, Ψ^{t} induces a natural simply transitive action (explained in Lemma 6.4) of the group of pseudospherical representations of M^{ℓ} on the set of pseudospherical genuine representations of M̃.

2. Representations of Covers of Abelian Groups

In this section H is an abelian Lie group with finite component group H/H^{0}. We consider the representation theory of a (possibly nonabelian) two-fold cover H̃ of G. This means that there is a short exact sequence

(2.1)  1 → {±1} → H̃ →^{p} H → 1.

A representation π of H̃ is called genuine if π(−1) = −I. Write Π_{g}(H̃) and Π_{g}(Z(H̃)) for the set of equivalence classes of irreducible genuine representations of H̃ and Z(H̃) respectively.

Proposition 2.2. For every χ ∈ Π_{g}(Z(H̃)) there is a unique representation π ∈ Π_{g}(H̃) for which π|_{Z(H̃)} is a multiple of χ. Denote this representation π(χ).

The map χ → π(χ) defines a bijection Π_{g}(Z(H̃)) → Π_{g}(H̃). The dimension of π(χ) is n, and IndH̃ ; Z(H̃)(χ) = nπ, where n = |H̃/Z(H̃)|^{\frac{1}{2}} .

Proof. The key point is that if π is a genuine representation of H̃ then the character θ_{π} of π is supported on Z(H̃), as we now show. Suppose g ∈ H̃ − Z(H̃). Choose h ∈ H̃ so that hgh^{−1} ≠ g. Since H is abelian p(hgh^{−1}) = p(h)p(g)p(h)^{−1} = p(g), so hgh^{−1} = −g. Computing θ_{π} of both sides we have θ_{π}(hgh^{−1}) = θ_{π}(g) = χ(−1)θ_{π}(g). Since χ is genuine χ(−1) = −1, so θ_{π}(g) = 0.

Therefore every irreducible genuine representation of H̃ is determined by its restriction to Z(H̃), and any two irreducible representations with the same central character are isomorphic. Fix χ ∈ Π_{g}(Z(H̃)). Let I(χ) = IndH̃ ; Z(H̃)(χ). This has central character χ, so there exists an irreducible representation π(χ), and I(χ) = cπ(χ) for some integer c. The dimension of I(χ) is |H̃/Z(H̃)|, so

I(χ)|_{Z(H̃)} = |H̃/Z(H̃)|χ, dim π(χ) = |H̃/Z(H̃)|/c.

By Frobenius reciprocity, it follows that

dim HomH̃(I(χ), I(χ)) = dim Hom_{Z(H̃)}(I(χ)|_{Z(H̃)}, χ) = |H̃/Z(H̃)|.

On the other hand, I(χ) is a direct sum of c copies of one irreducible representation (the complete reducibility following from finiteness of H/H^{0}). Schur’s lemma therefore implies that

dim HomH̃(I(χ), I(χ)) = c^{2}.

Comparing these two formulas gives c = n, and the remaining claims are immediate. □

For x, y ∈ H let x̃, ỹ be inverse images of x, y in H̃, and let {x, y} = x̃ỹx̃^{−1}ỹ^{−1}. This is independent of the choices of x̃, ỹ, and is contained in ker(p) = {±1}. The center of H̃ is the inverse image of the radical of this form: if we define

Z_{0}(H) = p(Z(H̃))

then it follows that

(2.3)  Z_{0}(H) = {x ∈ H | {x, y} = 1 for all y ∈ H}.

3. The center of Ã

We return now to the setting of Definition 1.1: 𝔾 is a connected, simple, simply connected, algebraic group, with split real form G and nontrivial two-fold cover G̃. The split torus Ã = M̃A^{0} of G̃ is a two-fold cover of A. To study principal series of G̃ we need to understand the irreducible representations of Ã. By Proposition 2.2, we need to compute the center of Ã. Because the identity component A^{0} of Ã is automatically central, it suffices to compute the center of M̃. By (2.3) we need to compute the radical of the commutator pairing (defined in Section 2).

Let R^{∨} be the coroot lattice of 𝔸, and

P = Hom_{ℤ}(R^{∨}, ℤ)

the dual lattice of integral weights. Write ⟨, ⟩ for the canonical pairing P ×R^{∨} → ℤ. Since 𝔾 is simply connected, the character and cocharacter lattices of 𝔸 are

P = X^{∗}(𝔸), R^{∨} = X_{∗}(𝔸).

Consequently

(3.1)  𝔸 = R^{∨} ⊗_{ℤ} ℂ^{×},  A = R^{∨} ⊗_{ℤ} ℝ^{×}

via the homomorphism λ^{∨} ⊗ x ↦ λ^{∨}(x). In particular, the subgroup M of A (consisting of all elements of order 2) may be identified as

(3.2)  M = R^{∨}/2R^{∨}

via the homomorphism λ^{∨} ↦ λ^{∨}(−1).

We recall the definition of the Hilbert symbol over ℝ: for x, y ∈ ℝ^{×} we have

(x, y)_{ℝ} = \begin{cases}−1 & x < 0, y < 0 \\ 1 & else. \end{cases}

The rational vector space R^{∨}⊗_{ℤ}ℚ carries a Weyl group-invariant positive definite quadratic form (·, ·), unique up to a positive scalar multiple (which we will fix in a moment). By duality this form induces a positive quadratic form (still written (·, ·)) on P ⊗_{ℤ} ℚ, which contains the roots. In this dual form, the roots have at most two (squared) lengths. Roots of the greater (or only) root length are called long, and the other roots (if any) are called short. We normalize the quadratic form so that the greater (or only) root length is 2:

(α, α) = 2 if α is a long root.

Because the natural pairing of a root with its coroot is equal to 2, this requirement is equivalent to

(α^{∨}, α^{∨}) = 2 if α is a long root.

The other root length is equal to 1 in types B_{n}, C_{n}, and F_{4}, and equal to 2/3 in type G_{2}. Short coroots have (squared) length 2; the other coroot length is 6 for G_{2} and 4 in types B_{n}, C_{n}, and F_{4}.

We emphasize that in the simply laced case, all roots are called long.

With this normalization, the quadratic form takes integer values on the coroot lattice R^{∨}. By [Ma] the commutator for Ã is given on A by

(3.3)  {γ^{∨} ⊗ x, λ^{∨} ⊗ y} = (x, y)_{ℝ}^{(γ^{∨},λ^{∨})} (γ^{∨}, λ^{∨} ∈ R^{∨}; x, y ∈ ℝ^{×}).

The quadratic form (,) is positive definite, and in particular non-degenerate. It therefore induces an isomorphism (from left to right)

R^{∨} ⊗ ℚ ≃ Hom_{ℚ}(R^{∨} ⊗ ℚ, ℚ).

Since P ≃ Hom_{ℤ}(R^{∨}, ℤ) we obtain an isomorphism (in the opposite direction)

(3.4)(a)  Γ: P ⊗ ℚ ≃ R^{∨} ⊗ ℚ.

Tracing through the definitions we see that

(3.4)(b)  ⟨λ, γ^{∨}⟩ = (Γ(λ), γ^{∨}) (λ ∈ P, γ^{∨} ∈ R^{∨}).

In particular we conclude

(3.4)(c)  Γ(P) = {γ^{∨} ∈ R^{∨} ⊗ ℚ | (γ^{∨}, α^{∨}) ∈ ℤ for all α^{∨} ∈ R^{∨}}.

Except in the case of C_{n} or G_{2}, Γ is the identity in the usual coordinates ([Bo], Chapitre 6). For C_{n} (respectively G_{2}) we have Γ = 1/2 (respectively 1/3).

Let

c = max_{α,β} \frac{(α, α)}{(β, β)} ∈ \{1, 2, 3\}.

Let t(α) = 1 if α is long, and c if α is short.

Lemma 3.5. For all roots α

\begin{aligned}Γ(α) & = \frac{1}{t(α)}α^{∨} \\ & = \frac{2α^{∨}}{(α^{∨}, α^{∨})}. \end{aligned}

Proof. Fix a long root β, so ||β^{∨}||^{2} = 2. That Γ commutes with the simple reflection s_{α} is equivalent to Γ(α) = τ(α)α for some τ(α) ∈ ℚ. By (3.4)(b) we have

2 = ⟨α, α^{∨}⟩ = (τ(α)α^{∨}, α^{∨})
= τ(α)(||α^{∨}||^{2}/||β^{∨}||^{2})||β^{∨}||^{2}
= 2τ(α)||β||^{2}/||α||^{2}

so τ(α) = ||α||^{2}/||β||^{2}. For the final equality note that

||α||^{2}/||β||^{2} = ||β^{∨}||^{2}/||α^{∨}||^{2} = 2/||α^{∨}||^{2}.
□

From (3.4)(b) and Lemma 3.5 we have the following useful formulas (for α any root).

(3.6)(a)  t(α)⟨α, Γ(λ)⟩ = ⟨λ, α^{∨}⟩ (λ ∈ P ⊗_{ℤ} ℚ)
(3.6)(b)  t(α)⟨α, γ^{∨}⟩ = (α^{∨}, γ^{∨}) (γ^{∨} ∈ R^{∨}).

Note that (3.6)(a) implies that for λ ∈ P ⊗_{ℤ} Q, ⟨α, Γ(λ)⟩ ∈ ℤ if and only if \frac{1}{t(α)}⟨λ, α^{∨}⟩ ∈ ℤ. In particular if 𝔾 is simply laced we conclude

(3.7)  Γ(R) = R^{∨},  Γ(P) = P^{∨}.

Another convenient formula for the commutator is Lemma 4.8 below.

The following lemma is similar to [Sa, Section 2.2] and [Sa2, page 117].

Lemma 3.8. In the identification (3.2) of M with R^{∨}/2R^{∨}, the preimage of the center of M̃ is

(3.9)  Z_{0}(M) ≃ [2Γ(P) ∩ R^{∨}]/2R^{∨}.

Proof. By (3.3) the commutator on M is given by

{γ^{∨}(−1), λ^{∨}(−1)} = (−1)^{(γ^{∨},λ^{∨})} (γ^{∨}, λ^{∨} ∈ R^{∨}).

By (3.4)(c) we conclude γ^{∨}(−1) ∈ Z_{0}(M) ⇔ γ^{∨} ∈ 2Γ(P) which gives (3.9).

□

It is sometimes convenient to use the following commutator formula.

Lemma 3.10. Suppose g ∈ A, α^{∨} is a coroot, and x ∈ ℝ^{×}. Then

(3.11)  {g, α^{∨}(x)} = (α(g), x)_{ℝ}^{t(α)}

where t(α) = 1 if α is long, and c if it is short.

Proof. It is enough to check this for g = β^{∨}(y) for β^{∨} ∈ R^{∨}. The left hand side is then (y, x)_{ℝ}^{(β^{∨},α^{∨})}. The right hand side is

(α(β^{∨}(y)), x)_{ℝ}^{t(α)} = (y^{⟨α,β^{∨}⟩}, x)_{ℝ}^{t(α)} = (x, y)_{ℝ}^{t(α)⟨α,β^{∨}⟩}

The result follows from (3.6)(b). □

In the simply laced case the result is particularly simple.

Lemma 3.12. If G is simply laced or of type G_{2}, then Z(M̃) = Z(G̃) and Z(Ã) = Z(G̃)A^{0}.

Proof. The hypothesis means that t(α) is always odd, and therefore it can be dropped from (3.11) (since the values of the Hilbert symbol are ±1). It follows immediately that Z_{0}(M) = {m ∈ M | α(m) = 1 for all α} = Z(G). Similarly Z_{0}(A) = {g ∈ A | α(g) > 0 for all α}. It is easy to see this is precisely Z(G)A^{0}. □

Using Lemmas 3.8 and 3.12 it is easy to compute Z_{0}(M) in each case. Its cardinality (equal to the number of irreducible genuine representations of M̃) is given in Table 1 in Section 6.

Example 3.13. Let G = Spin(n, n) or Spin(n + 1, n). In the usual coordinates P = ℤ^{n}∪(ℤ+ \frac{1}{2})^{n} and R^{∨} = ℤ_{e}^{n} (elements of ℤ^{n} for which the sum of the coordinates is even). We conclude

2Γ(P) ∩ R^{∨} = \begin{cases}2ℤ^{n} ∪ (2ℤ + 1)^{n} & n even \\ 2ℤ^{n} & n odd \end{cases}

and |Z_{0}(M)| = 4 or 2 respectively.

Let λ^{∨} = (2, 0, …, 0) in R^{∨}, and γ_{±}^{∨} = (1, 1, …, ±1) (which is in R^{∨} if n is even). Then Z_{0}(M) contains λ^{∨}(−1), and also γ_{±}^{∨}(−1) if n is even.

Example 3.14. Let G be the split group of type F_{4}. Then in the usual coordinates, representatives for 2Γ(P) ∩ R^{∨}/2R^{∨} are 0, (2, 0, 0, 0), (±1, 1, 1, 1).

4. Pseudospherical Principal Series

In this section we recall some additional structure for split real groups. In order to treat at the same time the nonlinear group G̃ of Definition 1.1 and the linear group G^{ℓ} (whose definition will be completed in Section 6), it is convenient for this section to weaken our assumptions on G and to work in the following setting.

Definition 4.1. Assume that G is a split real form of a connected reductive algebraic group, with Cartan involution θ, Borel subgroup B, and split maximal torus

A = B ∩ θB ⊂ B.

Assume that G̃ is a (possibly trivial) two-fold cover of G; that is, that we have a short exact sequence of Lie groups

1 → {±1} → G̃ → G → 1.

In general we write H̃ for the preimage in G̃ of any subgroup H of G. A representation π of H̃ is called genuine if π(−1) = −I. Write N for the unipotent radical of B and A^{0} for the identity component of A. Because these groups are simply connected, their coverings must split. We identify

N = identity component of Ñ,

and similarly for A^{0}.

If we choose G̃ to be the trivial cover G × {±1}, then genuine representations of G̃ may be identified with representations of G. Therefore a discussion of “genuine representations of G̃” in the setting of Definition 4.1 covers both split linear groups and the nonlinear groups of Definition 1.1.

Let N(Ã) be the normalizer of Ã in G̃, and let W = N(Ã)/Ã (the Weyl group of Ã in G̃). This group is independent of the covering, and may be identified (through its action on the Lie algebra 𝔞) with the Weyl group of the root system of G. Because Ã may be nonabelian, the action of N(Ã) on Ã may not factor to an action of W; but we do obtain an action of W on Z(Ã). In any case N(Ã) acts on Π_{g}(Ã) (the set of genuine irreducible representations of Ã) by

(4.2)  (g · π)(a) = π(g^{−1}ag) (π ∈ Π_{g}(Ã), g ∈ N(Ã), a ∈ Ã).

This factors to an action of W on Π_{g}(Ã) in all cases: if g ∈ Ã then π ≃ g · π, with intertwining operator π(g^{−1}). For w ∈ W and χ a genuine character of Z(Ã) we have (in the notation of Proposition 2.2)

π(χ)^{w} ≃ π(χ^{w}).

Corresponding statements hold with M̃ in place of Ã.

We denote (real) Lie algebras by the corresponding Gothic letter 𝔤, 𝔨, 𝔞, etc. Let α be a root of 𝔞 in 𝔤. Choose a Lie algebra homomorphism

ϕ_{α}: 𝔰𝔩(2, ℝ) ⟶ 𝔤

and define

(4.3)  Z_{α} = ϕ_{α} (\begin{matrix}0 & 1 \\ −1 & 0 \end{matrix}) ∈ 𝔨

as in [V4, 4.3.6]. (This means that ϕ_{α} carries the diagonal torus in 𝔰𝔩(2, ℝ) into 𝔞, the upper triangular matrices into the root space 𝔤_{α}, and 𝔰𝔬(2) into 𝔨.) The homomorphism ϕ_{α} is unique up to conjugation of 𝔰𝔩(2, ℝ) by the diagonal matrix with entries i and −i. In particular, this means that the Lie algebra element Z_{α} is defined only up to sign.

Exponentiating we obtain homomorphisms

Φ̃_{α}: S̃L(2, ℝ) → G̃, Φ_{α}: SL(2, ℝ) → G.

Here S̃L(2, ℝ) is the unique nontrivial two-fold cover of SL(2, ℝ). If G̃ is linear then Φ̃_{α} factors through SL(2, ℝ); otherwise this may or may not be the case.

Definition 4.4. A root α is said to be metaplectic if Φ̃_{α} does not factor to SL(2, ℝ).

Lemma 4.5 (for example, [A, Theorem 1.6]). Suppose G̃ is the nontrivial double cover of the group of real points of a simply connected simple split group. If G is not of type G_{2}, then α is metaplectic if and only if α is long. If G is of type G_{2}, then all roots are metaplectic.

Let

(4.6)  m̃_{α} = expG̃(πZ_{α}) ∈ M̃.

Changing the choice of ϕ_{α} replaces Z_{α} by −Z_{α}, and so replaces m̃_{α} by m_{α}^{−1}. The image of m̃_{α} in G is

m_{α} = exp_{G}(πZ_{α}) ∈ M.

If α is not metaplectic—for example if G̃ is linear—then m̃_{α} has order 2 and is independent of the choice of Φ_{α}, as in [V4, 4.3.6]. Almost as the definition of the coroot α^{∨}, we have

Φ_{α} (\begin{matrix}t & 0 \\ 0 & t^{−1} \end{matrix}) = α^{∨}(t).

In particular,

(4.7)  m_{α} = α^{∨}(−1) ∈ A.

If α is metaplectic, then m̃_{α} does depend on the choice of Φ_{α}; changing the choice replaces m̃_{α} by m_{α}^{−1}. More explicitly, recall that {±1} is the kernel of the covering map from G̃ to the linear group G. Then m̃_{α} has order 4, m_{α}^{2} = −1 and m_{α}^{−1} = −m̃_{α}. Let

h_{1} = exp (\begin{matrix}0 & π \\ −π & 0 \end{matrix}) , h_{2} = exp (\begin{matrix}0 & −π \\ π & 0 \end{matrix})

be the two elements of S̃L(2, ℝ) lying over −I ∈ SL(2, ℝ). Then

m̃_{α} = Φ̃_{α}(h_{1}).

We have h_{i}^{4} = 1 and h_{2} = h_{1}^{−1}.

The following formula is an immediate consequence of (3.3), Lemma 3.5, and (4.7).

Lemma 4.8. Suppose G̃ is the nontrivial double cover of the group of real points of a simply connected simple split group. For all roots α, β,

\{m_{α}, m_{β}\} = \begin{cases}(−1)^{⟨α,β^{∨}⟩} & α, β metaplectic \\ 1 & else. \end{cases}

Definition 4.9. Suppose G̃ is a split group as in Definition 4.1.

(1) An irreducible representation δ̃ of M̃ is pseudospherical if −1 is not an eigenvalue of δ̃(m̃_{α}) for any α.

(2) An irreducible representation μ of K̃ is

\begin{cases}pseudospherical \\ fine \\ petite \end{cases} if |γ| ≤ \{\frac{1}{2} 1 2

respectively, for every root α and every eigenvalue γ of dμ(iZ_{α}). More generally, we say a K-type μ is level k if |γ| ≤ k for every α and every eigenvalue γ of dμ(iZ_{α}).

(3) A representation of G̃ is pseudospherical if it contains a pseudospherical K̃-type.

Because Z_{α} is defined using a generator for the Lie algebra of SO(2) or its double cover, every eigenvalue γ of dμ(iZ_{α}) belongs to \frac{1}{2}ℤ (if α is metaplectic) or to ℤ (if not).

Remark 4.10. Suppose δ̃ is genuine. If α is metaplectic then condition (1) in Definition 4.9 is automatic: m_{α}^{2} = −1 and the only possible eigenvalues of δ̃(m̃_{α}) are ±i, independent of the choice of m̃_{α}. In particular if G̃ is simply laced or of type G_{2} then every irreducible genuine representation of M̃ is pseudospherical.

Suppose α is not metaplectic. Because m_{α}^{2} = 1, δ̃(m̃_{α}) has eigenvalues ±1. Note that the inverse image in G̃ of m_{α} ∈ G consists of ±m̃_{α}. If δ̃ is genuine then δ̃(−m̃_{α}) = −δ̃(m̃_{α}). Thus the definition of pseudospherical depends on the fact that we have specified m̃_{α} to be the unique inverse image of m_{α} that is in the image of Φ_{α}.

Lemma 4.11. Suppose that G̃ is as in Definition 4.1, and δ̃ is an irreducible genuine representation of M̃.

(1) Suppose β is a root of A in G that is not metaplectic (Definition 4.4), so that Φ̃_{β}: SL(2, ℝ) → G̃. Then the element m̃_{β} of (4.6) has order at most 2, and belongs to Z(M̃).

(2) The representation δ̃ is pseudospherical if and only if δ̃(m̃_{β}) = 1 for all nonmetaplectic roots β. This is a condition on the central character δ̃|_{Z(M̃)}.

(3) The action of the Weyl group W on irreducible genuine representations of M̃ fixes each pseudospherical representation.

(4) Define Z_{0}(M) to be the preimage in M of Z(M̃), and \overline{Z}_{0}(M) its quotient by the subgroup generated by elements m_{β} for β not metaplectic. Then there is natural simply transitive action of the group of characters of \overline{Z}_{0}(M) on genuine pseudospherical representations of M̃. The character \overline{χ} acts on δ̃ by tensoring the central character of δ̃ with the pullback of \overline{χ} to Z(M̃). Equivalently, one can choose any extension of \overline{χ} to a character χ of M, and tensor δ̃ with the pullback of χ to M̃.

Proof. In (1), that m̃_{β} has order at most 2 is immediate from the definition of metaplectic (see (4.6)). If m̃ ∈ M̃, then

(4.12)  Ad(m̃)(Z_{α}) = α(m̃)Z_{α} = ±Z_{α}.

Multiplying by π and exponentiating gives

(4.13)  m̃m̃_{α}m^{−1} = m_{α}^{α(m̃)}.

If m̃_{β} has order at most 2, then this reads

m̃m̃_{β}m^{−1} = m_{β}^{±1} = m̃_{β},

proving that m̃_{β} is central in M̃.

For (2), Remark 4.10 shows that the condition of being pseudospherical depends only on the non-metaplectic roots β; the rest of the statement follows from (1).

For (3), suppose that χ ∈ Π_{g}(Z(M̃)) is the central character of a pseudospherical representation of M̃. By Proposition 2.2, it is enough to show that s_{α}χ = χ for every simple reflection s_{α} ∈ W. To see this, define

(4.14)  σ̃_{α} = expG̃(πZ_{α}/2).

The image of σ̃_{α} in G is

σ_{α} = exp_{G}(πZ_{α}/2) = Φ_{α} (\begin{matrix}0 & 1 \\ −1 & 0 \end{matrix}) ,

a representative for the reflection s_{α} in the Weyl group. We wish to show that

(4.15)  χ(σ_{α}^{−1}m̃σ̃_{α}) = χ(m̃)

for all m̃ ∈ Z(M̃).

Multiplying (4.12) by π/2 and exponentiating gives

m̃σ̃_{α}m^{−1} = σ_{α}^{α(m̃)} = \begin{cases}σ̃_{α} & if α(m̃) = 1 \\ σ̃_{α}m_{α}^{−1} & if α(m̃) = −1. \end{cases}

We multiply this equation by σ_{α}^{−1} on the left and by m̃ on the right, obtaining

σ_{α}^{−1}m̃σ̃_{α} = \begin{cases}m̃ & if α(m̃) = 1 \\ m_{α}^{−1}m̃ & if α(m̃) = −1. \end{cases}

The left side of (4.15) is therefore

(4.16)  χ(σ_{α}^{−1}m̃σ̃_{α}) = \begin{cases}χ(m̃) & if α(m̃) = 1 \\ χ(m_{α}^{−1}m̃) & if α(m̃) = −1. \end{cases}

We now consider two possibilities. If α is not metaplectic, then χ(m̃_{α}) = 1 by (2), so both cases of (4.16) are equal to χ(m̃), as we wished to show.

If α is metaplectic, then m_{α}^{−1} = −m̃_{α} (see the discussion after (4.7)). Now (4.13) and the assumption that m̃ is central in M̃ imply that α(m̃) = 1. We are therefore in the first case of (4.16), so we have χ(m̃), as we wished to show.

Part (4) follows from the description of genuine representations of M̃ in Proposition 2.2. □

Remark 4.17. For a discussion of fine K-types for linear groups see [V4, 4.3]. Petite K-types are discussed in Section 12.

Here is a characterization of petite representations of K̃ in terms of their highest weights. Because the group K̃ may be disconnected, a little care is required in talking about highest weights. If G̃ is nonlinear, then K̃ is connected, and the group T_{K} in the following lemma is a maximal torus. If G̃ is linear, then its Cartan subgroups are abelian; so T_{K} is an abelian subgroup of K containing a maximal torus. We write X^{∗}(T_{K}) for the abelian group of characters of T_{K}; this maps by restriction onto the lattice X^{∗}(T_{K}^{0}) of characters of the maximal torus. Each imaginary root β gives rise to a three-dimensional root subalgebra of 𝔤, which is isomorphic either to 𝔰𝔩(2, ℝ) (if β is noncompact) or to 𝔰𝔲(2) (if β is compact). These isomorphisms ψ_{β} may be chosen to satisfy

ψ_{β}(𝔰𝔬(2)) ⊂ 𝔱_{K}

in both cases, and we may calculate the coroot as

β^{∨} = ψ_{β} (\begin{matrix}0 & i \\ −i & 0 \end{matrix}) ∈ i𝔱_{K}.

If β is noncompact, the map ψ_{β} may exponentiate only to the double cover S̃L(2, ℝ), and so the coroot β^{∨} may take half integer values on the differential of a character of T_{K}. This explains how half integers can arise in the lemma.

Lemma 4.18. Suppose G̃ is a split group as in Definition 4.1, and suppose μ is an irreducible representation of K̃. Choose a fundamental θ-stable Cartan subgroup T_{f} of G̃. Thus T_{K} = T_{f} ∩ K̃ contains a maximal torus of K̃. Let λ ∈ X^{∗}(T_{K}) be a highest weight of μ (with respect to some choice of positive roots). If G is of type B_{2n+1} or G_{2}, let

C = {noncompact imaginary roots β of T_{f} in G}.

Otherwise let

C = {imaginary roots β of T_{f} in G | β is noncompact long, or compact short}.

If β ∈ C then ⟨λ, β^{∨}⟩ ∈ \frac{1}{2}ℤ is defined. The representation μ is level k (Definition 4.9) if and only if

⟨λ, β^{∨}⟩ ≤ k for all β ∈ C.

Proof. If α is a root of 𝔞, then Z_{α} ∈ 𝔱_{K} for some fundamental Cartan subalgebra 𝔱_{f}. The definition of level is in terms of the eigenvalues of the elements iZ_{α}; so we need to identify these elements in terms of the coroots of imaginary roots. This can be done using the theory of Cayley transforms ([Sc], Proposition 2.16; [Kn], pages 417–420; [V4, 4.3.6]).

A Cayley transform c from 𝔞 to 𝔱_{f} may be defined by a maximal set S of strongly orthogonal roots of 𝔞. For each root α in this set, the definition of c identifies iZ_{α} with a noncompact imaginary coroot (of the same length). If α is a long root of 𝔞 then we may choose S containing α, and it follows that iZ_{α} is conjugate to the coroot for a long noncompact imaginary root. If α is short the same argument applies in type B_{2n+1} or G_{2}.

The remaining cases are groups of type B_{2n}, C_{n}, or F_{4}, in which a maximal strongly orthogonal set of roots must consist entirely of long roots. In this case one can also construct 𝔱_{f} from 𝔞 using a different maximal set S′ of orthogonal roots, including exactly two short roots α and α′. In this case, a calculation in type B_{2} shows that iZ_{α} is conjugate to the coroot corresponding to a short compact imaginary root.

So we may suppose that iZ_{α} is equal to an imaginary coroot β^{∨}. The eigenvalue of iZ_{α} on the γ-weight space of μ is ⟨γ, β^{∨}⟩. If in addition β is K-dominant (as we may as well assume, since S is invariant under the compact Weyl group), then ⟨γ, β^{∨}⟩ ≤ ⟨λ, β^{∨}⟩. The result follows. □

Lemma 4.19. Suppose G is a split linear group as in Definition 4.1. Let δ be a character of M, and let μ be an irreducible representation of K.

(1) The character δ is pseudospherical if and only if δ(m_{α}) = 1 for every root α (cf. (4.6)); equivalently, if and only if δ restricted to M ∩ G^{0} is trivial.

(2) The representation μ is pseudospherical if and only if its restriction to K ∩ G^{0} = K^{0} is trivial.

(3) The natural surjections

M ↠ G/G^{0}, K ↠ G/G^{0}

identify the characters of the component group of G with the pseudospherical representations of M and K.

Proof. The first condition in (1) is clear from the definition of pseudospherical; or it is a special case of Lemma 4.11. Since M ∩ G^{0} is generated by the elements {m_{α}}, the second condition in (1) follows. Computing in SO(2) (mapped into K by the homomorphism Φ_{α} of (4.3)), we see that dμ(iZ_{α}) can have only integer eigenvalues. Therefore μ pseudospherical implies dμ(iZ_{α}) = 0. The root space decomposition of 𝔤 shows that the various {Z_{α}} span the Lie algebra of K, and (2) follows. Statement (3) is an elementary consequence of (1) and (2).

□

From the point of view of unitarity, studying pseudospherical representations of a linear group is equivalent to studying spherical representations, as the next lemma shows. We have included pseudospherical (and not just spherical) representations of linear groups primarily because of the bijection of Lemma 6.4.

Lemma 4.20. (1) Suppose G is a Type I group, and H ⊂ G is a subgroup of finite index. Then an irreducible representation π of G is unitary if and only if any, equivalently every, summand of π restricted to H is unitary.

(2) Suppose G is a split linear group. Then for all pseudospherical characters δ of M and ν ∈ 𝔞^{∗}, I_{G}(δ, ν)|_{G^{0}} ≃ I_{G^{0}}(ν), and J_{G}(δ, ν)|_{G^{0}} ≃ J_{G^{0}}(ν). In particular CS(δ, G) = CS(G^{0}) (notation as in Definition 1.1).

(3) Suppose G, G′ are split linear groups with a finite number of connected components, which are locally isomorphic, and δ, δ′ are pseudospherical. Then (if we identify 𝔞 and 𝔞′ by the local isomorphism) CS(δ, G) = CS(δ′, G′).

Proof. Statement (1) is elementary, and (2) follows from the fact that G = G^{0}A. A spherical representation of G factors to G/Z for any finite subgroup Z of the center of G, since Z is contained in K, and (3) follows easily. □

A description of the pseudospherical representations for certain disconnected groups is found in Table 3. The pseudospherical representations of nonlinear groups will be discussed in Section 5.

5. Pseudospherical representations of nonlinear groups

In this section 𝔾 is simple, simply connected, with split real form G, and G̃ is the nontrivial two-fold cover of G. Our goal is to describe precisely the pseudospherical representations of M̃ and of K̃. More general results (applicable to arbitrary coverings of arbitrary real reductive groups) may be found in [W, 11.A.2]. In our special setting it is possible to get some additional explicit information about the representations, which we will use in the proof of Theorem 1.4. For this reason we give proofs of the results.

We first compute the pseudospherical representations of M̃. Recall (Proposition 2.2) that the irreducible genuine representations of M̃ are parametrized by genuine characters of Z(M̃).

Lemma 5.1. (1) Suppose G ≄ Spin(2n + 1, 2n) or Sp(4n, ℝ). Restriction to Z(G̃) defines a canonical bijection between the genuine pseudospherical representations of M̃ and the genuine characters of Z(G̃).

(2) If G = Spin(2n + 1, 2n) or Sp(4n, ℝ) then M̃ has exactly two genuine pseudospherical representations. These have the same restriction to Z(G̃).

The pseudospherical representations of M̃ are listed in Table 1.

Proof. Suppose G is simply laced or of type G_{2}. Then every root is is metaplectic and therefore (Remark 4.10) every irreducible genuine representation of M̃ is pseudospherical. On the other hand Z(G̃) = Z(M̃) in this case (Lemma 3.12), and the result follows.

The case of Sp(2n, ℝ) is well-known; see Section 12.2. There are two pseudospherical one-dimensional representations det^{±1/2} of K̃, whose restrictions δ̃_{±} are the pseudospherical representations of M̃. These satisfy δ̃_{+}^{∗} ≃ δ̃_{−}. If n is odd then Z(G̃) ≃ ℤ/4ℤ and δ̃_{±} have distinct characters when restricted to Z(G̃). If n is even then Z(G̃) ≃ ℤ/2ℤ × ℤ/2ℤ and δ̃_{±} have the same central character.

Let G = Spin(n + 1, n), n ≥ 2. See Example 3.13. For all short roots α, m_{α} is the unique nontrivial central element of G. Because α is not metaplectic, it follows that

Z(G̃) = {1, −1, m̃_{α}, −m̃_{α}} ≃ (ℤ/2Z)^{2}.

Therefore every genuine pseudospherical representation of M̃ has the same character when restricted to Z(G̃) (trivial on m̃_{α}, and −1 on −1); in fact a genuine representation of M̃ is pseudospherical if and only if it has this restriction. Let χ be a character of Z_{0}(M) such that χ(m_{α}) = −1; regard χ as a (non-genuine) character of Z(M̃). Then for any genuine character ξ̃ of Z(M̃), precisely one of ξ̃ and χ ⊗ ξ̃ is pseudospherical. From Table 1 we conclude G̃ has two genuine pseudospherical representations if n is even, and one if n is odd. The result follows in this case.

Suppose G is of type F_{4}. In this case Z(G) is trivial and Z_{0}(M) is generated by {m_{α} | α short} (cf. Example 3.14). It follows that Z(M̃) is an elementary twogroup, and has exactly one genuine pseudospherical representation. □

Proposition 5.2. Let (δ̃, Fδ̃) be a pseudospherical representation of M̃. There is a unique pseudospherical representation (μ(δ̃), Fδ̃) of K̃ such that μ(δ̃)|M̃ = δ̃. The map δ̃ → μ(δ̃) is a bijection between the pseudospherical representations of M̃ and K̃.

Proof. Essentially this may be found in Section 6 of [V0], but we repeat the argument for completeness. If δ̃ is not genuine, this is Lemma 4.19; so assume δ̃ is genuine.

Fix a root α. Since m_{α}^{4} = 1, the possible eigenvalues of δ̃(m̃_{α}) are {±1, ±i}. Since δ̃ is pseudospherical, −1 does not occur. Therefore Fδ̃ is the sum of the three eigenspaces Fδ̃(α, 1) and Fδ̃(α, ±i). Let log be the principal branch of the logarithm; recall that

log(1) = 0, log(i) = iπ/2, log(−i) = −iπ/2.

Recall (4.6) m̃_{α} = exp(πZ_{α}), so (if μ exists) dμ(Z_{α}) must preserve the γ eigenspace Fδ̃(α, γ), and must act there with eigenvalues λ ∈ \frac{1}{π} log(γ) + 2iℤ. The requirement that μ be pseudospherical means that |iλ| ≤ 1/2. This means that we are forced to define

(5.3)  dμ(Z_{α})v = \frac{log(γ)}{π}v  (v ∈ Fδ̃(α, γ)).

A little more informally,

dμ(Z_{α}) = \frac{log(δ̃(m̃_{α}))}{π}.

This defines dμ(Z_{α}) ∈ Aut(Fδ̃). Note that the eigenvalues of dμ(iZ_{α}) are equal to 0 or ±1/2, so μ is pseudospherical if it is a representation.

The root space decomposition of 𝔤 shows that the Z_{α} are a basis of 𝔨, so dμ is a well-defined map 𝔨 → Aut(Fδ̃).

Uniqueness of μ is now clear: if μ is pseudospherical this is the only possible choice of dμ. For existence, we must show first that the linear map dμ is a representation of 𝔨. That is, for any two roots α and β, we must show that

dμ([Z_{α}, Z_{β}]) = [dμ(Z_{α}), dμ(Z_{β})].

This reduces to a rank two calculation, as follows. Let H be the subgroup of G generated by the root subgroups for ±α and ±β, and H̃ its inverse image in G̃. The pseudospherical representation δ̃ of M̃ is the sum of irreducible genuine pseudospherical representations δ̃_{H} of M̃_{H} = M̃ ∩ H̃. If we can prove our claim for H̃—that is, show that each δ̃_{H} defines a representation dμ_{H} of 𝔥 ∩ 𝔨 by the formula (5.3)—then the commutation relation we want for Z_{α} and Z_{β} follows. Evidently H is a connected split semisimple group of type A_{1} × A_{1}, A_{2}, B_{2}, or G_{2}. In the first case the claim is obvious. In the remaining cases, the assumption that G is simply connected implies that H must also be simply connected; so H is SL(3, ℝ), Sp(4, ℝ), or the split real form of G_{2}.

If the covering H̃ is trivial, then none of the roots of H can be metaplectic, so δ̃_{H} is trivial on all m̃_{γ}. It follows that dμ_{H} is trivial on 𝔥 ∩ 𝔨, and in particular dμ_{H} is a representation.

We may therefore assume that the covering H̃ is not trivial; so it is the covering of Definition 1.1. We need to consider each genuine pseudospherical representations δ̃_{H} of M̃_{H}, and show that dμ_{H} is a Lie algebra representation. If μ′ is any irreducible genuine pseudospherical representation of K̃_{H} = H̃ ∩ K̃, then the arguments above show that μ′ restricts to an irreducible genuine pseudospherical representation δ′ of M̃_{H}, and that dμ′ is the linear map defined by δ′. This establishes our claim for all such δ′. It therefore suffices to show that the number of irreducible genuine pseudospherical representations of K̃_{H} is equal to the number of irreducible genuine pseudospherical representations of M̃_{H}. These numbers (one, two, and one in the cases A_{2}, B_{2}, and G_{2}) may be found in Tables 1 and 2. (They are computed using Lemma 4.18 and Lemma 6.3 below.)

This proves that dμ is indeed a representation of the Lie algebra 𝔨. To check that dμ exponentiates to the group K̃, the simplest approach is to use the fact that K̃ is simply connected except in type C_{n}. We will give a different argument, which can be generalized to show (for any reductive group G) that “(𝔨, M)-modules” satisfying some simple compatibility conditions must always exponentiate to K.

It is enough to restrict attention to a maximal torus in K̃. We can construct such a torus using a maximal strongly orthogonal set of roots α_{1}, …, α_{n}. These roots define by (4.3) a group homomorphism

Φ̃ = Φ̃_{α_{1}} × ⋯ × Φ̃_{α_{n}}: S̃O(2)^{n} → K̃.

The group

T̃ = Φ̃(S̃O(2)^{n})

is a maximal torus in K̃. By construction (because the eigenvalues involved were half integers) it is clear that dμ exponentiates to a representation μS̃O of S̃O(2)^{n}. Write x_{i} for an element of order 4 in S̃O(2)_{i}; that is,

Φ̃(x_{i}) = m̃_{α_{i}}.

Because α_{i} is a well-defined character of T̃, corresponding to the character 2 of SO(2)_{i}, it follows that

ker Φ̃ = Q ⊂ ⟨x_{i} | 1 ≤ i ≤ n⟩ = (ℤ/4ℤ)^{n}.

In order to show that μS̃O descends to T̃, we must show that it is trivial on Q. But we also have a short exact sequence

1 → Q → ⟨x_{i}⟩ → ⟨m̃_{α_{i}}⟩ → 1.

Since μS̃O(x_{i}) = δ̃(m̃_{α_{i}}), it follows that μS̃O descends to δ̃ on ⟨m̃_{α_{i}}⟩, and therefore must be trivial on Q. Therefore μ exponentiates to T̃ and consequently to K̃.

□

Suppose g ∈ 𝔾(ℂ) normalizes G. Because the covering G̃ is canonical, the action of g on G lifts uniquely to an action on G̃.

Lemma 5.4. Norm_{𝔾(ℂ)}(G) acts transitively on the irreducible genuine pseudospherical representations of M̃ and K̃.

Proof. The simplest proof is to check in each case that Norm_{𝔾(ℂ)}(G) acts transitively on the irreducible genuine pseudospherical K̃-types (cf. Table 2). For example if G = SL(2n) then the action of element diag(−ζ, ζ, …, ζ), where ζ^{2n} = −1, interchanges Spin_{+} and Spin_{−}. (It is easier to realize this automorphism of SL(2n) as conjugation by diag(−1, 1, …, 1) ∈ GL(2n)). It is helpful to keep in mind that the map μ → μ^{∗} (sending a representation of K̃ to its dual) may be realized as the action of an element of 𝔾(ℂ). □

The significance of this result is that it implies CS(δ̃, G̃), for δ̃ pseudospherical, is independent of δ̃. Compare Lemma 4.20.

Finally we can complete the definition of the representations J(δ̃, ν) appearing in the introduction.

Definition 5.5. Suppose δ̃ is an irreducible genuine pseudospherical representation of M̃. Let μ(δ̃) be the irreducible representation of K̃ given by Lemma 4.19 and Proposition 5.2. Notice that Frobenius reciprocity implies that μ(δ̃) occurs with multiplicity one in I(δ̃, ν). We let J(δ̃, ν) be the unique irreducible constituent of I(δ̃, ν) containing μ(δ̃).

It can be shown (compare [V4, 6.6.15]) that J(δ̃, ν) coincides with the (unique) Langlands subquotient of I(δ̃, ν).

As a consequence of Proposition 5.2, Lemma 4.11, and Harish-Chandra’s subquotient theorem, the representation J(δ̃, ν) is the unique irreducible representation of G̃ containing the K-type μ(δ̃) and having infinitesimal character ν. We will use this observation to calculate theta correspondences in Section 14.

The following proposition summarizes the conclusions of this section.

Proposition 5.6. There are canonical bijections between the following sets: (1) irreducible genuine pseudospherical representations of M̃;

(2) irreducible genuine pseudospherical representations of K̃; and

(3) genuine pseudospherical principal series representations of G̃ with fixed infinitesimal character.

If G ≄ Sp(4n, ℝ) or Spin(2n + 1, 2n) (n ≥ 1) these sets are also in canonical bijection with

(4) genuine characters of Z(G̃).

6. The group G^{ℓ}

We continue with the notation of the preceding section.

Definition 6.1. In order to describe G^{ℓ}, we begin with its root system Δ^{ℓ}:

Δ^{ℓ} = \begin{cases}Δ & G simply laced or type G_{2} \\ Δ^{∨} & G not of type G_{2} \end{cases}

Of course the simply laced groups fall into both cases; but then Δ is naturally isomorphic to Δ^{∨} by the map Γ (cf. (3.7)). Notice that there is a natural bijection α ↔ α^{ℓ} between Δ and Δ^{ℓ}, defined by

α^{ℓ} = \begin{cases}α & G simply laced or type G_{2} \\ α^{∨} & G not of type G_{2} \end{cases}

This bijection provides an identification of Weyl groups

W (Δ) ≃ W (Δ^{ℓ}), s_{α} ↔ s_{α^{ℓ}}.

Write

R^{ℓ} ⊂ P^{ℓ} ⊂ (𝔞^{ℓ})^{∗}

for the root lattice, integral weight lattice, and ambient vector space of Δ^{ℓ}; similarly we write

(R^{ℓ})^{∨} ⊂ (P^{ℓ})^{∨} ⊂ 𝔞^{ℓ}

for the coroot and integral coweight lattices.

To complete the definition of G^{ℓ}, we must describe its coweight lattice L,

(R^{ℓ})^{∨} ⊂ L ⊂ (P^{ℓ})^{∨}.

The definition is

L = \begin{cases}\{γ ∈ P^{∨} | 2γ ∈ R^{∨}\} & G simply laced or type G_{2} \\ \{γ ∈ P | 2Γ(γ) ∈ R^{∨}\} & G not of type G_{2} \end{cases}

Notice first of all that (P^{ℓ})^{∨} is equal to P^{∨} in the first case and to P in the second case, so L is actually a sublattice of (P^{ℓ})^{∨}. Next, the two definitions agree in the simply laced case, since we are using Γ to identify P with P^{∨} then. Finally, the fact that L ⊃ (R^{ℓ})^{∨} is obvious in the first case, and is a consequence of Lemma 3.5 in the second.

We can now define a linear reincarnation of G̃ as a complex simple algebraic group 𝔾^{ℓ} with root system Δ^{ℓ} and coweight lattice L. There is in particular a maximal torus

𝔸^{ℓ} = ℂ^{×} ⊗_{ℤ} L.

We fix a split real form G^{ℓ} of 𝔾^{ℓ} with split maximal torus

A^{ℓ} = ℝ^{×} ⊗_{Z} L.

Define a W -equivariant lattice homomorphism

Ψ: L → R^{∨}

by

Ψ(γ) = \begin{cases}2γ & G simply laced or type G_{2} \\ 2Γ(γ) & G not of type G_{2} \end{cases}

On the level of coroots, we can calculate from Lemma 3.5 that

(6.2)  Ψ((α^{ℓ})^{∨}) = \begin{cases}2α^{∨} & if α is metaplectic \\ α^{∨} & if α is not metaplectic. \end{cases}

This map induces by functoriality a W -equivariant group homomorphism

Ψ: A^{ℓ} → A.

Lemma 6.3. The image of the group homomorphism

Ψ: A^{ℓ} → A

(Definition 6.1) is equal to Z_{0}(A). The map Ψ restricts to a surjection

Ψ: M^{ℓ} ↠ Z_{0}(M).

Proof. Exactly as in (3.2), we have M^{ℓ} = L/2L. It follows that the image Ψ(M^{ℓ}) is

2L/2R^{∨} ⊂ R^{∨}/2R^{∨} = M.

The second assertion is now just a reformulation of Lemma 3.8 if G is not of type G_{2}. When G is of type G_{2}, L = R^{∨}, Ψ is squaring, and the image Ψ(M^{ℓ}) is trivial. Since Z_{0}(M) is also trivial in this case (Lemma 3.12), the second assertion is proved in general. The first is an immediate consequence. □

Choose a maximal compact subgroup K^{ℓ} of G^{ℓ}, so that A^{ℓ} = M^{ℓ}(A^{ℓ})^{0} with M^{ℓ} = A^{ℓ} ∩ K^{ℓ} as usual. We count the pseudospherical representations of M^{ℓ} (and therefore of K^{ℓ} and of the component group of G^{ℓ}, by Lemma 4.19) in Table 3.

Lemma 6.4. (1) Suppose that G is simply laced or of type G_{2}. Then the kernel of the group homomorphism

Ψ: M^{ℓ} ↠ Z_{0}(M)

(Lemma 6.3) is the subgroup generated by the various m_{α^{ℓ}} (defined in (4.6)); equivalently, the intersection M^{ℓ} ∩ (G^{ℓ})^{0}. Consequently the transpose map

Ψ^{t}: {characters of Z_{0}(M)} ↪ {characters of M^{ℓ}}

has as image precisely the pseudospherical characters of M^{ℓ} (Definition 4.9).

(2) Suppose that G is not of type G_{2}. Then the kernel of the group homomor-

phism

Ψ: M^{ℓ} ↠ Z_{0}(M)

is the subgroup generated by the various m_{α^{ℓ}} with α long. If β is short, then

Ψ(m_{β^{ℓ}}) = m_{β}.

Consequently the transpose map

Ψ^{t}: {characters of Z_{0}(M)} ↪ {characters of M^{ℓ}}

has as image precisely the characters of M^{ℓ} trivial on m_{α^{ℓ}} with α long. In particular,

Ψ^{t}: {characters of Z_{0}(M) trivial on m_{β} for β short} ↪ {characters of M^{ℓ}}

has as image precisely the pseudospherical characters of M^{ℓ}.

(3) For any G, define \overline{Z}_{0}(M) to be the quotient of Z_{0}(M) by the subgroup generated by the elements m_{β} with β not metaplectic (Lemma 4.11). Then the transpose map

Ψ^{t}: \{characters of \overline{Z}_{0}(M)\} ↪ \{characters of M^{ℓ}\}

has as image precisely the pseudospherical characters of M^{ℓ}.

(4) There is a natural simply transitive action of the group of pseudospherical characters of M^{ℓ} on the set of genuine pseudospherical representations of M̃, as follows. Given a pseudospherical character δ^{ℓ} of M^{ℓ}, choose a character δ of M so that Ψ^{t}(δ) = δ^{ℓ}. Regard δ as a (non-genuine) character of M̃. Then if δ̃ is any genuine pseudospherical representation of M̃, we define

δ^{ℓ} · δ̃ = δ ⊗ δ̃.

Proof. Parts (1) and (2) are a straightforward consequence of the calculation of Γ in Lemma 3.5 and of Definition 6.1; we leave the details to the reader. Parts (3) and (4) then follow immediately from Lemma 4.11. □

In Table 1, the column |Π_{g}(M̃)| counts irreducible genuine representations of M̃; |Π_{gs}(M̃)| counts irreducible genuine pseudospherical representations of M̃; and Dim is the dimension of an irreducible genuine representation of M̃. In Table 3, the column |Π_{s}(M^{ℓ})| counts the irreducible pseudospherical representations of M^{ℓ}.

7. A conjectural shimura correspondence

Retain the notation of Definition 1.1 and Section 4. Fix an irreducible genuine pseudospherical representation δ̃ of M̃ and let μ(δ̃) denote its pseudospherical extension to K̃ (Proposition 5.2). Recall from Definition 5.5 that J(δ̃, ν) is defined to be the composition factor of I(δ̃, ν) containing μ(δ̃). Let

CS(δ̃, G̃) = {ν ∈ 𝔞_{ℂ}^{∗} | J(δ̃, ν) is unitary},

the pseudospherical complementary series of G̃. Lemma 4.11(3) implies that

J(δ̃, ν) ≃ J(δ̃, ν′)

Table 1. Genuine representations of M̃

Type G  |Π_{g}(M̃)| |Π_{gs}(M̃)| Dim
A_{2m−1} SL(2m)  2  2  2^{m−1}
A_{2m}  SL(2m + 1)  1  1  2^{m}
B_{2m}  Spin(2m + 1, 2m)  4  2  2^{m−1}
B_{2m−1} Spin(2m, 2m − 1)  2  1  2^{m−1}
C_{n}  Sp(2n)  2^{n}  2  1
D_{2m}  Spin(2m, 2m)  4  4  2^{m−1}
D_{2m+1} Spin(2m + 1, 2m + 1) 2  2  2^{m}
E_{6}  E_{6}(C_{4})  1  1  8
E_{7}  E_{7}(A_{7})  2  2  8
E_{8}  E_{7}(D_{8})  1  1  16
F_{4}  F_{4}(A_{1} × C_{3})  4  1  2
G_{2}  G_{2}(A_{1} × A_{1})  1  1  2

Table 2. Pseudospherical representations of K̃

G  K̃  μ̃  Dim(μ̃)
SL(2m)  Spin(2m)  Spin_{±}  2^{m−1}
SL(2m + 1)  Spin(2m + 1)  Spin  2^{m}
Spin(2m + 1, 2m)  Spin(2m + 1) × Spin(2m)  1 ⊗ Spin_{±} 2^{m−1}
Spin(2m, 2m − 1)  Spin(2m) × Spin(2m − 1)  1 ⊗ Spin 2^{m−1}
Sp(2n)  { \atop U(n)}  det^{± \frac{1}{2}}  1
Spin(2m, 2m)  Spin(2m) × Spin(2m)  1 ⊗ Spin_{±},
Spin_{±} ⊗ 1 2^{m−1}
Spin(2m + 1, 2m + 1) Spin(2m + 1) × Spin(2m + 1) 1 ⊗ Spin,
Spin ⊗ 1 2^{m}
E_{6}  Sp(8)  ℂ^{8}  8
E_{7}  SU(8)  ℂ^{8}, (ℂ^{8})^{∗} 8
E_{8}  Spin(16)  ℂ^{16}  16
F_{4}  Sp(6) × Spin(3)  1 ⊗ Spin 2
G_{2}  Spin(3) × Spin(3)  Spin ⊗ 1 2

if and only if there exists w ∈ W such that ν′ = w · ν. Consequently CS(G̃, δ̃) is a closed W -invariant set.

Conjecture 7.1. Let G^{ℓ} be given as in Section 6. Let Ψ: A^{ℓ} → A be given by Lemma 6.3, with adjoint dΨ^{t}: 𝔞_{ℂ}^{∗} → (𝔞^{ℓ})_{ℂ}^{∗}. Fix a pseudospherical representation δ^{ℓ} of M^{ℓ}. Then dΨ^{t} defines a bijection:

CS(δ̃, G̃) → CS(δ^{ℓ}, G^{ℓ})

That is, JG̃(δ̃, ν) is unitary if and only if J_{G^{ℓ}}(δ^{ℓ}, dΨ^{t}(ν)) is unitary.

By Lemma 4.20, CS(δ^{ℓ}, G^{ℓ}) is independent of δ^{ℓ}, and is equal to the spherical complementary series of the identity component CS((G′)^{0}) of any linear group G′ locally isomorphic to G^{ℓ}.

Table 3. The linear reincarnation G^{ℓ}

G  G^{ℓ}  |Π_{s}(M^{ℓ})|
SL(n)  PGL(n)  2  n even
SL(n)  1  n odd
Spin(n + 1, n) PSp(2n)  2  n even
Sp(2n)  1  n odd
Sp(2n)  SO(n + 1, n) 2
Spin(n, n)  PSO(n, n)  4  n even
SO(n, n)  2  n odd
E_{6}^{sc}  E_{6}^{sc}  1
E_{7}^{sc}  E_{7}^{ad}  2
E_{8}  E_{8}  1
F_{4}  F_{4}  1
G_{2}  G_{2}  1

If G is simply laced or G_{2} then Conjecture 7.1 is equivalent to: JG̃(δ̃, ν) is unitary if and only if the spherical representation J_{G}(2ν) is unitary, i.e.,

2CS(δ̃, G̃) = CS(G).

Example 7.2. Let G = Sp(2n, ℝ) and G^{ℓ} = SO(n + 1, n). As discussed in the introduction Ψ: A^{ℓ} ≃ A is an isomorphism in this case, given by the identity map in the usual coordinates. See Section 12.2.

Note that ρ (one-half the sum of the positive roots) for SO(n + 1, n), i.e., (n − \frac{1}{2}, …, \frac{3}{2}, \frac{1}{2}), is the image under dΨ^{t} of the Harish-Chandra parameter of the infinitesimal character of the oscillator representation of S̃p(2n, ℝ). The trivial representation, which is an isolated point in the unitary dual of SO(n+1, n) (for n ≥ 2) corresponds to one of the components of the oscillator representation.

For example if n = 2, the complementary series CS(δ̃, S̃p(4, ℝ)) = CS(SO(3, 2)) look like this:

Figure 1. Spherical complementary series for SO(3, 2)

If G is a classical group we will prove the inclusion

dΨ^{t} (CS(δ̃, G̃)) ⊂ CS(δ^{ℓ}, G^{∨})

of Conjecture 7.1 (Theorem 13.24), and equality for G̃ = S̃p(2n) (Theorem 14.2).

Remark 7.3. We conclude this section with a discussion of the p-adic analog of Conjecture 7.1. Let F be a p-adic field with ring of integers R, and let G be a split simple algebraic group defined over R. Fix an integer n relatively prime to p such that μ_{n}(F), the nth roots of unity in F, has order n. It follows from [Ma] that there is a “metaplectic” extension G̃ of G by μ_{n}(F) (see, for example, [Sa, 2.1]). Let I denote an Iwahori subgroup of G, and Ĩ its preimage in G̃. The extension splits canonically over I, and we use this splitting to regard I as a subgroup of G̃. The category of smooth representations of G generated by their I-fixed vectors is equivalent to the category of representation of the Hecke algebra H(G) of I bi-invariant compactly supported functions on G. Likewise, the category of smooth (suitably genuine) representation of G̃ generated by their I-fixed vectors is equivalent to the category of representations of H(G̃), the (suitably genuine) I-invariant compactly supported functions on G̃. Savin has predicted and investigated the existence of an isomorphism between H(G^{ℓ}) and H(G̃). This is very subtle; the arguments offered in [Sa] may not be complete. When G is simply laced, the situation is simpler and Savin has proved in [Sa2, Theorem 7.8] that H(G^{ℓ}) ≃ H(G̃). The conclusion in these cases is that there is an equivalence between the category of (suitably genuine) smooth representations of G̃ generated by their I-fixed vectors and the category of smooth representations of G^{ℓ} generated by their I^{ℓ}-fixed vectors. The p-adic analog of Conjecture 7.1 is that this equivalence should preserve unitarity (cf. [Hu2]).

8. Reduction to real infinitesimal character

We continue in the setting of Conjecture 7.1. Recall from [Kn, Section 14.4] the notion of representations with real infinitesimal character. In our setting this is extremely straightforward: J(δ̃, ν) has real infinitesimal character if and only if ν ∈ 𝔞^{∗} ⊂ 𝔞_{ℂ}^{∗}; equivalently, if ⟨ν, α^{∨}⟩ ∈ ℝ for all roots α. In this case we say ν is real. Similarly, we say ν is purely imaginary if ν ∈ i𝔞^{∗}; that is, if iν is real. Many questions about the classification of unitary representations can be reduced to the case of real infinitesimal character. Our purpose in this section is to do that for Conjecture 7.1. Here is the statement.

Proposition 8.4. Suppose G is a simply connected split real group as in Definition 1.1. For any connected subset E of the Dynkin diagram of G, let G_{E} be the subgroup of G corresponding to the roots spanned by E; this is a simply connected split real group, with split torus A_{E} ⊂ A. Assume that Conjecture 7.1 holds for each group G_{E} and real ν_{E} ∈ 𝔞^{∗}. Then Conjecture 7.1 holds for G.

Proof. That each G_{E} is split is clear. That G_{E} is simply connected follows from the fact that each fundamental weight for G_{E} (corresponding to a vertex of its Dynkin diagram) is the restriction to A_{E} of the corresponding fundamental weight for G. We write (G^{ℓ})_{E} for the corresponding algebraic subgroup of G^{ℓ}, and G̃_{E} for the preimage of G_{E} in G̃. The group G̃_{E} is a two-fold cover of G_{E} as in Definition 4.1, so it is either trivial or equal to the distinguished non-trivial cover of Definition 1.1. Similarly, the group (G^{ℓ})_{E} has the same Weyl group as G_{E}, but it may or may not be the group (G_{E})^{ℓ} constructed in Definition 6.1. We will say more about these questions at the end of the proof.

We first describe how to relate the unitarity of pseudospherical representations of G̃ to unitarity of pseudospherical representations of the groups G̃_{E} with real infinitesimal character.

Fix ν ∈ 𝔞_{ℂ}^{∗} and write ν = ν_{ℝ} +iν_{I} with ν_{ℝ}, ν_{I} real. After conjugating ν by W, we may assume that ν_{I} is dominant. Let L be the Levi subgroup of G corresponding to the roots {α | ⟨ν_{I}^{ℓ}, α^{∨}⟩ = 0}, and let P be the corresponding standard parabolic subgroup. Then the spherical representations IL̃(δ̃, ν) and JL̃(δ̃, ν) are defined, and

IG̃(δ̃, ν) = IndG̃ ; P̃(IL̃(δ̃, ν)).

Furthermore

JG̃(δ̃, ν) = IndG̃ ; P̃(JL̃(δ̃, ν))

and

(8.5)(a)  JG̃(δ̃, ν) is unitary if and only if JL̃(δ̃, ν) is unitary.

This is contained [Kn, Theorem 16.10]; the assumption there that the group is linear is not important.

Now let E be the subset of the Dynkin diagram corresponding to L (consisting of the simple roots vanishing on ν_{I}). Write this set as the disjoint union of its connected components:

E = E_{1} ∪ ⋯ ∪ E_{r}.

Let Z ⊂ A be the center of L. Then the identity component of L̃ is finitely covered by the identity component of the product group Z × ∏_{i=1}^{r} G̃_{E_{i}}. We have

𝔞 ≃ 𝔷 ⊕ ∑_{i=1}^{r} 𝔞_{E_{i}}, ν = ν_{Z} + ∑_{i=1}^{r} ν_{i},

with ν_{Z} ∈ 𝔷_{ℂ}^{∗} and ν_{i} ∈ (𝔞_{E_{i}})_{ℂ}^{∗}. Fix some irreducible constituent δ̃_{i} of the restriction of δ̃ to M̃ ∩ G̃_{E_{i}}. Standard facts about unitary representations of product groups show that

(8.5)(b)  JL̃(δ̃, ν) unitary ⇔ ν_{Z} imaginary and each JG̃_{E_{i}} (δ̃_{i}, ν_{i}) unitary.

The construction of L shows also that

each ν_{i} is real.

Equations (8.5) provide a reduction of the classification of unitary pseudospherical representations of G̃ to parallel problems for simple Levi subgroups and real infinitesimal character. We turn next to a similar reduction for G^{ℓ}.

As in Conjecture 7.1, we fix a pseudospherical character δ^{ℓ} of M^{ℓ}, and define ν^{ℓ} = dΨ^{t}(ν) ∈ (𝔞^{ℓ})_{ℂ}^{∗}. Note that dΨ^{t}(ν_{R}) = ν_{R}^{ℓ} and dΨ^{t}(ν_{I}) = ν_{I}^{ℓ}. The fundamental fact is that dΨ carries each coroot (α^{ℓ})^{∨} for G^{ℓ} to a positive multiple of the corresponding coroot α^{∨} for G. This means first of all that ν_{I}^{ℓ} is dominant, and that the Levi subgroup L^{ℓ} defined by ν_{I}^{ℓ} corresponds to the subset {α^{ℓ} | α ∈ E} of the Dynkin diagram for G^{ℓ}. The pseudospherical representations I_{L^{ℓ}}(δ^{ℓ}, ν^{ℓ}) and J_{L^{ℓ}}(δ^{ℓ}, ν^{ℓ}) are defined, and

I_{G^{ℓ}}(δ^{ℓ}, ν^{ℓ}) = Ind_{P^{ℓ}}^{G^{ℓ}}(I_{L^{ℓ}}(δ^{ℓ}, ν^{ℓ})), J_{G^{ℓ}}(δ^{ℓ}, ν^{ℓ}) = Ind_{P^{ℓ}}^{G^{ℓ}}(J_{L^{ℓ}}(δ^{ℓ}, ν^{ℓ})).

We have

(8.6)(a)  J_{G^{ℓ}}(ν^{ℓ}) is unitary if and only if J_{L^{ℓ}}(ν^{ℓ}) is unitary.

Again this is [Kn, Theorem 16.10]; this time what is unimportant is the assumption there that G is connected.

Because the Dynkin diagrams of G and G^{ℓ} are the same except for the orientation of arrows, the connected components of E in the Dynkin diagram of G^{ℓ} are precisely the various E_{i}. Let Z^{ℓ} be the center of L^{ℓ}. Then L^{ℓ} is finitely covered by Z^{ℓ} × ∏_{i=1}^{r}(G^{ℓ})_{E_{i}}, and we have

𝔞^{ℓ} ≃ 𝔷^{ℓ} ⊕ ∑_{i=1}^{r} 𝔞_{E_{i}}^{ℓ}.

The map Ψ: A^{ℓ} → A of Definition 6.1 restricts to

Ψ_{Z}: Z^{ℓ} → Z, Ψ_{i}: A_{E_{i}}^{ℓ} → A_{E_{i}}.

Accordingly we can write

ν^{ℓ} = ν_{Z}^{ℓ} + ∑_{i=1}^{r} ν_{i}^{ℓ}, ν_{Z}^{ℓ} = dΨ_{Z}^{t}(ν_{Z}) ∈ 𝔷_{ℂ}^{∗}, ν_{i}^{ℓ} = dΨ_{i}^{t}(ν_{i}) ∈ (𝔞_{E_{i}}^{ℓ})_{ℂ}^{∗}.

Just as in (8.5)(b),

(8.6)(b) J_{L^{ℓ}}(δ^{ℓ}, ν^{ℓ}) unitary ⇔ ν_{Z}^{ℓ} imaginary and each J_{(G^{ℓ})_{E_{i}}} (δ_{i}^{ℓ}, ν_{i}^{ℓ}) unitary;

here δ_{i}^{ℓ} is the restriction of the character δ^{ℓ} to M^{ℓ} ∩ (G^{ℓ})_{E_{i}}.

Equations (8.6) reduce the classification of unitary pseudospherical representations of G^{ℓ} to parallel problems for simple Levi subgroups and real infinitesimal character.

In order to complete the proof of Proposition 8.4, we must show that the unitarity criteria in (8.5) and (8.6) agree. Because dΨ preserves real and imaginary parts, we have immediately that

(8.7)(a)  ν_{Z} is imaginary ⇔ ν_{Z}^{ℓ} is imaginary.

Next, fix one of the E_{i}. There are two possibilities. If no root in E_{i} is metaplectic, then the covering G̃_{E_{i}} is trivial, so J_{(G̃)_{E_{i}}} (δ̃_{i}, ν_{i}) is unitary if and only if the spherical representation J_{G_{E_{i}}} (ν_{i}) is unitary. According to (6.2), the map Ψ_{i} carries the coroots of G_{E_{i}}^{ℓ} onto those of G_{E_{i}}. Consequently G_{E_{i}} finitely covers the identity component of G_{E_{i}}^{ℓ}, and this covering identifies ν_{i} with ν_{i}^{ℓ}. It follows immediately that

(8.7)(b)  JG̃_{E_{i}} (δ̃_{i}, ν_{i}) is unitary ⇔ J_{(G^{ℓ})_{E_{i}}} (δ_{i}^{ℓ}, ν_{i}^{ℓ}) is unitary.

Finally, assume that some root of E_{i} is metaplectic. It follows that the covering G̃_{E_{i}} is nontrivial, so it must be the covering of Definition 1.1. The map Ψ_{i} inherits the critical properties (6.2) from Ψ, and it follows easily that (G^{ℓ})_{E_{i}} is locally isomorphic to the linear group (G_{E_{i}})^{ℓ} constructed in Definition 6.1. The assertion

(8.7)(c)  JG̃_{E_{i}} (δ̃_{i}, ν_{i}) is unitary ⇔ J_{(G^{ℓ})_{E_{i}}} (δ_{i}^{ℓ}, ν_{i}^{ℓ}) is unitary.

in this case is equivalent to Conjecture 7.1 for the group G_{E_{i}} and the real parameter ν_{i}. The hypothesis of the proposition is that this statement holds.

Equations (8.7) show that the unitarity criteria for ν in G̃ and ν^{ℓ} in G^{ℓ} match perfectly. The proposition follows. □

9. Petite Representations of the Weyl Group

Suppose R is a root system, with Weyl group W. Suppose α, β are adjacent roots. The subgroup ⟨s_{α}, s_{β}⟩ of W generated by s_{α}, s_{β} is isomorphic to the Weyl group of type A_{2}, B_{2} or G_{2}, and so has a one-dimensional sign representation.

Definition 9.1. Let σ be a representation of W. We say σ is petite if

(1) for all pairs α, β of adjacent simple roots, the restriction of σ to ⟨s_{α}, s_{β}⟩ does not contain the sgn representation; and

(2) if α and β are adjacent long roots inside a subsystem of type G_{2}, then the restriction of σ to ⟨s_{α}, s_{β}⟩ does not contain the sgn representation.

Condition (2) excludes the one-dimensional character of W (G_{2}) acting on the product of a set of long positive roots. The one-dimensional character of W (G_{2}) on the product of a set of short positive roots is allowed; it would have been excluded if we omitted the word “simple” in (1). For root systems not containing factors of type G_{2}, the word “simple” in (1) has no effect, since any pair of adjacent roots is (up to sign) conjugate by W to a pair of simple roots.

The motivation for this terminology will be given in Section 10 (cf. Lemma 10.9).

9.1. Petite Representations of Classical Weyl Groups. We recall some notation from [Ba1]. The classical Weyl groups are all realized as linear transformations of 𝔞 ≃ ℝ^{n}. The hyperoctahedral group W (B_{n}) = W (C_{n}) includes all 2^{n} · n! such transformations. The symmetric group W (A_{n−1}) ≃ S_{n} changes no signs, and W (D_{n}) is the subgroup of W (B_{n}) changing an even number of signs.

For (a, b) a partition of n we let (a, b) denote the corresponding irreducible representation of W (A_{n−1}) ≃ S_{n}, of dimension \binom{n}{a}(n − 2a + 1)/(n − a + 1). (Representations attached to partititions into more than two parts contain the sign character of S_{3}, and so are not petite.) The representation (n) is the trivial representation of S_{n}, and (n − 1, 1) is the (n − 1)-dimensional reflection representation. We use the same notation to denote (a, b) pulled back to W (B_{n}) = W (C_{n}) or W (D_{n}) via the natural surjections to S_{n}.

Let χ_{n} denote the character of W (B_{n}) satisfying χ(s_{α}) = 1 (resp. −1) if α is long (resp. short). The Weyl group W (D_{n}) is the kernel of χ_{n}.

Fix non-negative integers a, b satisfying a + b = n. We embed W (B_{a}) × W (B_{b}) in W (B_{n}) as usual. Let (a)×(b) = Ind_{W (B_{a})×W (B_{b})}^{W (B_{n})}(1⊗χ_{b}). These representations are irreducible and distinct, and (a) × (b) ≃ χ_{n} ⊗ ((b) × (a)). The dimension of (a) × (b) is \binom{n}{a}.

We also write (a)×(b) for the restriction of (a)×(b) to W (D_{n}). As representations of W (D_{n}), (a) × (b) ≃ (b) × (a), and this representation is irreducible unless a = b. If n is even the representation (n/2) × (n/2) of W (B_{n}) restricts to the direct sum of two irreducible representations (n/2) × (n/2)_{±} of W (D_{n}), each of dimension \binom{n}{n/2}/2.

Proposition 9.2. Every irreducible petite representation of a classical Weyl group is isomorphic to (a, b), (a) × (b) or (n/2) × (n/2)_{±}.

We omit a proof of this proposition. Our only application of petite Weyl group representations is in the proof of Theorem 13.24 (cf. Proposition 12.11). For these purposes it is enough to use the condition of Proposition 9.2 as the definition of petite.

10. Action of the Weyl group

We work in the setting of Definition 4.1, so G̃ is a linear or nonlinear group. Fix a genuine pseudospherical representation (δ̃, Fδ̃) of M̃ and let (μ_{0}, Fδ̃) be its unique pseudospherical extension to K̃ (Lemma 4.19 and Proposition 5.2).

Let M̃′ be the normalizer of Ã in K̃. We have an exact sequence

(10.1)  1 → M̃ → M̃′ → W → 1

The following Proposition is a basic result in Clifford theory; for example see [CR, Theorem 51.7].

Lemma 10.2. There is a bijection between the irreducible representations of W and the irreducible representations of M̃′ whose restriction to M̃ contains δ̃. The bijection may be constructed as follows.

If (σ, E_{σ}) is an irreducible representation of W, let E_{τ} = E_{σ} ⊗ Fδ̃, and define

τ(g)(v ⊗ w) = σ(g)(v) ⊗ μ_{0}(g)(w) (v ∈ Fδ̃, w ∈ E_{σ}, g ∈ M̃′).

Conversely, suppose (τ, E_{τ}) is an irreducible representation of M̃′ containing δ̃. Define a representation σ of M̃′ on HomM̃(Fδ̃, E_{τ}) by

σ(n)(ϕ) = τ(n) ∘ ϕ ∘ μ_{0}(n^{−1}) (n ∈ M̃′, ϕ ∈ HomM̃(Fδ̃, E_{τ})).

Then σ factors to a representation of W, and

HomM̃(Fδ̃, E_{τ}) ⊗ Fδ̃ ≃ E_{τ}

as representations of M̃′, via the map ϕ ⊗ v → ϕ(v).

Remark 10.3. If we begin only with the exact sequence (10.1) and an irreducible representation δ̃ of M̃, then the bijection of the lemma depends on the existence and choice of a representation of M̃′ extending δ̃. In our setting we can use the canonically defined extension μ_{0} of δ̃ to K̃. This is what makes the bijection of the theorem canonical.

Now consider the principal series representation I(δ̃, ν). The μ-isotypic component of I(δ̃, ν) is given by

(10.4)(a)  HomK̃(V_{μ}, I(δ̃, ν)) ⊗ V_{μ}

which by Frobenius reciprocity is isomorphic to

(10.4)(b)  HomM̃(V_{μ}, Fδ̃) ⊗ V_{μ}.

Definition 10.5. Suppose μ is a representation of K̃. Write

V_{μ}[δ̃] = HomM̃(V_{μ}, Fδ̃) ≃ HomK̃(V_{μ}, I(δ̃, ν)).

The Weyl group representation σ(μ) attached to μ and δ̃ is defined on this space as in Lemma 10.2:

(10.6)  σ(n)(ϕ) = μ_{0}(n) ∘ ϕ ∘ μ(n^{−1})  (n ∈ M̃′, ϕ ∈ V_{μ}[δ̃]).

The representation σ is trivial on M̃, and therefore factors to a representation of W on V_{μ}[δ̃].

The Weyl group representation σ(μ) need not be irreducible. According to Lemma 10.2, it is irreducible if and only if the representation of M̃′ on the δ̃- isotypic subspace of μ is irreducible.

We are interested in the representations of W attached to petite representations of K̃.

Definition 10.7. Let ΣK̃(G̃, δ̃) be the set of irreducible petite representations of K̃ containing the pseudospherical M̃ representation δ̃. Let

(10.8)  Σ_{W} (G̃, δ̃) = {σ ∈ Ŵ | σ ⊂ V_{μ}[δ̃] for some μ ∈ ΣK̃(G̃, δ̃)}.

Lemma 10.9 ([Ba1]). Suppose G is a linear classical group, and δ a pseudospherical character of M. Then Σ_{W} (G, δ) is the set of petite representations of W.

(In fact if μ is an irreducible petite representation of K containing the pseudospherical M representation δ, then V_{μ}[δ] is an irreducible petite representation of W.)

Remark 10.10. The same result holds for exceptional groups (with a slight modification in type G_{2}), but the calculations necessary to prove this are too long to include here.

We discuss nonlinear groups in the next two sections.

11. Construction of petite K̃-types: simply laced case

We continue with the setting of the previous section, where now G̃ is a nonlinear group. We find a relationship between genuine petite representations of K̃, pseudospherical representations of M̃, and petite representations of W.

Proposition 11.1. Assume G is simply laced, and that G̃ is the nonlinear cover of Definition 1.1. Let S = {(δ̃, σ)} where δ̃ is a genuine pseudospherical representation of M̃ and σ is a petite representation of W. There is a bijection between S and the genuine irreducible petite representations of K̃.

Explicitly, suppose μ is such a representation of K̃. Necessarily the restriction of μ to M̃ is a multiple of a genuine pseudospherical representation δ̃ of M̃. Let σ(μ) be the Weyl group representation attached to μ and δ̃ (Definition 10.5). Then σ(μ) is irreducible, and μ corresponds to the pair (δ̃, σ(μ)).

Conversely, given a pair (δ̃, σ) ∈ S, construct an irreducible representation τ of M̃′ by Lemma 10.2. Then τ extends uniquely to a genuine irreducible petite representation μ of K̃.

Remark 11.2. A weaker version of Proposition 11.1 holds in the case of two root lengths, although the construction of μ is more complicated. (To begin with, μ need not restrict to a multiple of an irreducible representation of M̃). We give arguments for types B_{n} and C_{n} in Section 12 below.

Example 11.3. If σ is trivial then we recover the bijection between pseudospherical representations of M̃ and K̃ (Proposition 5.2).

Example 11.4. Let G = SL(3), G̃ = S̃L(3), so K ≃ SO(3) and K̃ ≃ SU(2). The groups M̃, M̃′ and W in (10.1) are the quaternion group of order 8 (cf. Proposition 5.2), the binary octahedral group of order 48, and the symmetric group S_{3}, respectively. The binary octahedral group has three genuine irreducible representations τ_{+}, τ_{−}, and τ′, of dimensions 2, 2, and 4 respectively. Here τ_{+} is the restriction to M̃′ of the two-dimensional representation of K̃. The corresponding representation of W is trivial (Lemma 10.2). The irreducible four-dimensional representation of K̃ restricts to τ′, and corresponds to the reflection representation of W. The representation τ_{−} of M̃′ corresponds to the sgn representation of W. Note that τ_{−} is not the restriction of an irreducible representation of K̃.

Proof. First we show that the two maps are inverses of each other on the level of representations of M̃′. For one direction, suppose (μ, V_{μ}) is a genuine representation of K̃. By Lemma 5.1, μ restricts to a multiple of a genuine pseudospherical representation δ̃ of M̃, and

(11.5)  V_{μ} ≃ HomM̃(Fδ̃, V_{μ}) ⊗ Fδ̃ ≃ V_{μ}[δ̃]^{∗} ⊗ Fδ̃

as representations of M̃′.

On the other hand, given (δ̃, σ) and defining V_{μ} = Fδ̃ ⊗ E_{σ}, we have

V_{μ}[δ̃]^{∗} ≃ HomM̃(Fδ̃, Fδ̃ ⊗ E_{σ}) ≃ E_{σ}.

We are left with two things to show. First, given (δ̃, σ) ∈ S, let τ be the irreducible representation of M̃′ from Lemma 10.2. We must show that τ extends uniquely to an irreducible petite representation μ of K̃. Second, given an irreducible genuine petite representation μ of K, we must show that the representation σ(μ) of W is petite.

For the first assertion, recall from (4.14) the element σ̃_{α} = exp(πZ_{α}/2) ∈ M̃′. We have σ_{α}^{2} = m̃_{α}, and σ̃_{α} is a representative in M̃′ for the reflection s_{α}. Because α is metaplectic and δ̃ is genuine, the possible eigenvalues of τ(m̃_{α}) are ±i. Hence the possible eigenvalues of τ(σ_{α}) are exp(imπ/2), with m = ±1/2 or ±3/2. As in the proof of Proposition 5.2, if an extension μ of τ to K̃ exists, then dμ(Z_{α}) must act on the exp(imπ/4) eigenspace of τ with eigenvalues i(m + 4k), with k integral. If in addition μ is petite, then (Definition 4.9) necessarily |m + 4k| ≤ 2. This last requirement forces the integer k to be zero. That is, dμ(Z_{α}) must act on the exp(imπ/2) eigenspace of τ(σ̃_{α}) by the scalar im. Informally,

(11.6)  dμ(Z_{α}) = 2 log(τ(σ̃_{α}))/π.

This proves uniqueness. It remains to show that dμ is actually a representation of 𝔨 that exponentiates to K̃. As in the proof of Proposition 5.2 this reduces to a rank two calculation. If ⟨α, β^{∨}⟩ = 0 then the corresponding three-dimensional subgroups commute with each other, so [Z_{α}, Z_{β}] = 0, σ̃_{α} and σ̃_{β} commute, and the result is immediate.

Assume therefore that ⟨α, β^{∨}⟩ = ±1. In this case the group H defined in the proof of Proposition 5.2 is SL(3, ℝ). This case was treated in Example 11.4. The petite hypothesis on σ means that the restriction of σ to the Weyl group of H does not contain the sign representation. Therefore the restriction of τ to the binary octahedral group M̃∩ H̃ does not contain σ_{−}. The example therefore shows that τ extends uniquely to a petite representation of K̃ ∩ H̃ = SU(2), a sum of two- and four- dimensional representations.

This completes the proof that dμ is a representation of 𝔨. That it exponentiates to K can be proved as in Proposition 5.2.

For the second assertion, suppose μ is a genuine petite K̃-type. Then μ restricted to any rank 2 subgroup H̃ ∩ K̃ as above is a direct sum of the irreducible 2and 4-dimensional representations of SU(2). These correspond to the trivial and reflection representations of S_{3} by Example 11.4); so the sign representation of S_{3} does not appear. Therefore σ(μ) is a petite representation of W. This completes the proof. □

12. Construction of petite K̃-types: Classical Groups

In this section we enumerate petite representations for classical groups. A tilde over a simply connected split classical group (like S̃pin(n+1, n)) means the nonlinear double cover of Definition 1.1.

12.1. Petite K̃-types for SL(n) and S̃L(n). Let G = S̃L(n), so K̃ = Spin(n). We parametrize the irreducible representations of Spin(n) by highest weights as usual. A typical highest weight is μ = (a_{1}, …, a_{r}), with a_{1} ≥ ⋯ ≥ a_{r−1} ≥ |a_{r}|, with r = [n/2], and a_{i} ∈ ℤ for all i, or a_{i} ∈ ℤ + 1/2 for all i. If n is odd then a_{r} ≥ 0.

For a_{r} ≥ 0 and ϵ = ±, write (a_{1}, …, a_{r})_{ϵ} for the highest weight (a_{1}, …, a_{r−1}, ϵa_{r}) with the understanding that ϵ = + if n is odd.

Let Spin_{ϵ} be the irreducible representation with highest weight (1/2, …, 1/2)_{ϵ}. If n is odd only ϵ = + is allowed, and we write Spin = Spin_{+}.

Lemma 12.1 ([Ba1]). Let G = SL(n, ℝ). For 1 ≤ k ≤ n/2, the K-type μ_{k,ϵ} with highest weight

(\underbrace{2, …, 2}_{k}, 0, …)_{ϵ}

is petite. (Here ϵ = − is allowed only if n is even, and μ_{k,+} = μ_{k,−} unless k = n/2.) The corresponding W -representation (in the notation of Section 9.1 and Definition 10.5) is given by

V_{μ_{k,ϵ}}[1] = (n − k, k).

Moreoever, the K-types {μ_{k,ϵ} | 1 ≤ k ≤ [k/2]} detect spherical unitarity in the sense of Definition 13.11 below.

Now we consider S̃L(n).

Lemma 12.2. Let G = S̃L(n, ℝ).

(1) Assume that n is odd. Write δ̃ for the unique genuine pseudospherical representation of M̃ arising as the restriction of the the spin representations of K̃; see Table 1. For k ≤ (n − 1)/2, the K̃-types with highest weights

μ_{k} = (\underbrace{3/2, …, 3/2}_{k}, 1/2, …, 1/2)

exhaust the set ΣK̃(G̃, δ̃) of δ̃-petite K̃-types in the sense of Definition 10.7. In the notation of Definition 10.5,

V_{μ_{k}}[δ̃] = (n − k, k).

(2) Assue that n is even. According to Table 1, there are two genuine pseudospherical representations δ̃_{±}; they are obtained (respectively) by restricting the two representations Spin_{±} to M̃. For k ≤ n/2, the K̃-types with highest weights

μ_{k,ϵ} = (\underbrace{3/2, …, 3/2}_{k}, 1/2, …, 1/2)_{(−1)^{k}ϵ}

exhaust the set ΣK̃(G̃, δ̃_{ϵ}) of δ̃_{ϵ}-petite K̃-types in the sense of Definition 10.7. In the notation of Definition 10.5,

V_{μ_{k,ϵ}}[δ̃_{ϵ}] = (n − k, k).

Proof. The bijection between genuine petite K̃-types and petite Weyl group representations is given by Proposition 11.1. The explicit calculation of this bijection follows from the methods of [Ba1]. Except in a few cases the correspondence is determined by the dimensions. □

Remark 12.3. Because we will encounter increasingly detailed statements of the form of Lemmas 12.1 and 12.2, it is useful to introduce some notation to make their statements more compact. We summarize the statement of Lemma 12.1 by saying that the following K-types are petite and detect spherical unitarity for SL(n, ℝ).

\begin{matrix}μ & V_{μ}[1] \\ (\underbrace{2, …, 2}_{k}, 0, …)_{ϵ} & (n − k, k) & k ≤ n/2 \end{matrix}

Meanwhile, we summarize the statement of Lemma 12.2 by saying that the petite K̃-types for S̃L(n, ℝ) and corresponding Weyl group representations are as follows.

μ  δ̃  V_{μ}[δ̃]
(\underbrace{3/2, …}_{k}, 1/2, …)_{ϵ} δ̃_{(−1)^{k}ϵ} (n − k, k) k ≤ n/2  n even
(\underbrace{3/2, …}_{k}, 1/2, …)_{+}  δ̃_{+} (n − k, k) k ≤ (n − 1)/2 n odd

12.2. Petite K̃-types for Sp(2n, ℝ) and S̃p(2n, ℝ). Assuming the results of [Ba1] it is straightforward to prove the analogue of Proposition 11.1 for S̃p(2n).

Let K be a maximal compact subgroup of Sp(2n, ℝ), with inverse image K̃ in S̃p(2n, ℝ). We identify the irreducible representations of K̃ and K with their highest weights. A highest weight may be written (a_{1}, …, a_{n}) with a_{1} ≥ ⋯ ≥ a_{n}. Exponentiating to a weight for the torus in K̃ means that either a_{i} ∈ ℤ for all i, or a_{i} ∈ ℤ + \frac{1}{2} for all i. The weight (1, …, 1) corresponds to the one-dimensional determinant character det of K.

Let μ_{±} be the genuine character of K̃ with highest weight ±(1/2, …, 1/2), or equivalently satisfying μ_{±}^{2} = det^{±}. Let δ̃_{±} be the restriction of μ_{±} to M̃.

By Lemma 4.18, a K̃-type μ = (a_{1}, …, a_{n}) is petite if and only if a_{1} ≤ 2, a_{n} ≥ −2, and a_{1} − a_{n} ≤ 2.

Lemma 12.4 ([Ba1]). Retain the notation oulined in Remark 12.3. The following K-types for Sp(2n, ℝ) are petite and detect spherical unitarity.

μ  V_{μ}[1]
(\underbrace{2, …, 2}_{k}, 0, …, 0) (n − k) × (k) k ≤ n
(0, …, 0, \underbrace{−2, …, −2}_{k}) (n − k) × (k) k ≤ n
(\underbrace{1, …, 1}_{k}, 0, …, 0, \underbrace{−1, …, −1}_{k}) (n − k, k)  k ≤ n/2

Lemma 12.5. Retain the notation oulined in Remark 12.3. Let G̃ = S̃p(2n, ℝ). The genuine petite K̃-types and the associated Weyl group representations are as follows:

\begin{matrix}μ & δ̃ & V_{μ}[δ̃] \\ (\underbrace{1/2, …, 1/2}_{k}, −3/2, …, −3/2) & δ̃_{+} & (n − k) × (k) \\ (3/2, …, 3/2, \underbrace{−1/2, …, −1/2}_{k}) & δ̃_{−} & (n − k) × (k) \\ (\underbrace{3/2, …, 3/2}_{k}, 1/2, …, 1/2, \underbrace{−1/2, …, −1/2}_{k}) & δ̃_{+} & (n − k, k) \\ (\underbrace{1/2, …, 1/2}_{k}, −1/2, …, −1/2, \underbrace{−3/2, …, −3/2}_{k}) & δ̃_{−} & (n − k, k) \end{matrix}

Proof. Fix a genuine irreducible representation μ of K̃. Then μ ⊗ δ̃_{±} factors to an irreducible representation of K, and

HomM̃(μ, δ̃_{±}) ≃ Hom_{M}(μ ⊗ δ̃_{∓}, δ̃_{±} ⊗ δ̃_{∓}) ≃ V_{μ⊗δ̃∓}[1]

The result follows immediately from the preceding lemma. □

Remark 12.6. In spite of the simplicity of this proof, note that the pseudospherical unitary dual of S̃p(2n, ℝ) is related to the spherical unitary dual of Spin(n + 1, n), and not to that of Sp(2n, ℝ). See Section 14.

12.3. Petite K̃-types for Spin and S̃pin. Let G̃ = S̃pin(n, n), so K̃ ≃ Spin(n) × Spin(n). See Section 12.1 for a discussion of highest weights for Spin(n).

A K̃-type is given by a highest weight (a_{1}, …, a_{r}) ⊗ (b_{1}, …, b_{r}), with r = [n/2]. This is genuine if a_{i} − b_{j} ∈ ℤ + \frac{1}{2} for all i, j.

Let μ(R, ±) = 1 ⊗ Spin_{±}, and μ(L, ±) = Spin_{±} ⊗ 1; if n is odd write μ(R, ±) = 1 ⊗ Spin. (The letters R and L stand for “right” and “left.”) Let δ̃(R, ±) and δ̃(L, ±) be the restriction of μ(R, ±) and μ(L, ±) to M̃. These are the genuine pseudospherical representations of K̃ and M̃.

Lemma 12.7 ([Ba1]). Retain the notation oulined in Remark 12.3. The K-types for Spin(n, n) listed in Table 4 are petite and detect spherical unitarity.

Lemma 12.8. Retain the notation oulined in Remark 12.3. Let G̃ = S̃pin(n, n). The genuine petite K̃-types and the associated Weyl group representations are as listed in Table 5.

Proof. See the proof of Lemma 12.2. □

Table 4. Petite K-types for Spin(n, n)

μ  V_{μ}[1]
(\underbrace{1, …}_{k}, 0, …)_{ϵ_{1}} ⊗ (\underbrace{1, …}_{k}, 0, …)_{ϵ_{2}} (n − k) × (k)_{ϵ_{1}ϵ_{2}} k ≤ n/2
1 ⊗ (\underbrace{2, …}_{k}, 0, …)_{ϵ} (k, n − k)  k ≤ n/2
(\underbrace{2, …}_{k}, 0, …)_{ϵ} ⊗ 1 (k, n − k)  k ≤ n/2

Table 5. Genuine petite K̃-types for S̃pin(n, n)

μ  δ̃  V_{μ}[δ̃]
(\underbrace{1, …}_{k}, 0, …)_{σ} ⊗ Spin_{ϵ} δ̃(R, (−1)^{k}ϵ) (k) × (n − k)_{ϵσ} k ≤ n/2
Spin_{ϵ} ⊗ (\underbrace{1, …}_{k}, 0, …)_{σ} δ̃(L, (−1)^{k}ϵ) (k) × (n − k)_{ϵσ} k ≤ n/2
1 ⊗ (\underbrace{3/2, …}_{k}, 1/2, …)_{ϵ} δ̃(R, (−1)^{k}ϵ) (n − k, k)  k ≤ n/2
(\underbrace{3/2, …}_{k}, 1/2, …)_{ϵ} ⊗ 1 δ̃(L, (−1)^{k}ϵ) (n − k, k)  k ≤ n/2

Lemma 12.9 ([Ba1]). Retain the notation oulined in Remark 12.3. The following K-types for Spin(n + 1, n) are petite and detect spherical unitarity.

μ  V_{μ}[1]
(\underbrace{1, …, 1}_{k}, 0, …, 0) ⊗ (\underbrace{1, …, 1}_{k}, 0, …, 0)_{ϵ} (n − k) × (k) k ≤ n/2
(\underbrace{1, …, 1}_{n+1−k}, 0, …, 0)_{ϵ} ⊗ (\underbrace{1, …, 1}_{n−k}, 0, …, 0) (n − k) × (k) (n + 1)/2 ≤ k ≤ n
1 ⊗ (\underbrace{2, …, 2}_{k}, 0, …, 0)_{ϵ} (n − k, k)  k ≤ n/2

Now let G̃ = S̃pin(n + 1, n), so K̃ = Spin(n + 1) × Spin(n). If n is even let μ_{±} = 1⊗Spin_{±} and let δ̃_{±} be the restriction of μ_{±} to M̃. If n is odd let μ = 1⊗Spin and let δ̃ be the restriction of μ to M̃. These are the genuine pseudospherical representations of K̃ and M̃ (Table 2).

Lemma 12.10. Retain the notation oulined in Remark 12.3. Let G̃ = S̃pin(n + 1, n). The genuine petite K̃-types and the associated Weyl group representations are as listed in Table 6.

Proof. If n is odd the restriction of any genuine K̃-type to M̃ is a multiple of δ̃ by Proposition 5.6. If n is even δ̃_{±} have the same central character, so μ is not necessarily isotypic for M̃.

Table 6. Genuine petite K̃-types for S̃pin(n + 1, n)

μ  δ̃  V_{μ}[δ̃]
1 ⊗ (\underbrace{3/2, …,}_{k}1/2, …)_{ϵ}  δ̃_{(−1)^{k}ϵ} (n − k, k)
(\underbrace{1, …,}_{k}0, …) ⊗ Spin_{ϵ}  δ̃_{(−1)^{k}ϵ} (n − k) × (k)
n even, k ≤ n/2
(\underbrace{1, …,}_{k}0, …) ⊗ Spin_{ϵ} δ̃_{(−1)^{k+1}ϵ} (k − 1) × (n − k + 1)
n even, k ≤ n/2
(\underbrace{1, …,}_{k}0, …) ⊗ Spin  δ̃  (n − k) × (k) ⊕ (k − 1) × (n − k + 1)
n odd, k ≤ (n − 1)/2
(1, …, ±1) ⊗ Spin  δ̃  ((n − 1)/2) × ((n + 1)/2) n odd

(1) μ = 1 ⊗ (\underbrace{3/2, …,}_{k}1/2, …)_{ϵ}, n even. In this case μ restricts irreducibly to

Spin(n)×Spin(n). By Lemma 12.8, V_{μ}[δ̃] is an irreducible representation of W (B_{n}) whose restriction to W (D_{n}) is (n−k, k). The fact that δ̃ is as stated can be deduced by tensoring μ with the pseudospherical representation μ_{±} and comparing the result with the list of petite representations of Spin(n + 1, n) in Lemma 12.9; the details are as in case (4) below.

(2) μ = (\underbrace{1, …,}_{k}0, …) ⊗ Spin, n odd, k ≤ (n − 1)/2. In order to compute V_{μ}[δ̃] as a

W representation, write μ_{L} = (1, …, 0, …) (with k ones), and use the isomorphism

HomM̃(δ̃, μ_{L} ⊗ Spin) ≃ Hom_{M} (1, μ_{L} ⊗ (Spin ⊗ Spin^{∗})) .

It is well known and easy to check that

Spin ⊗ Spin = ∑_{0≤m<n/2} ⋀^{m}(ℂ^{n}).

The highest weights of these representations are

⋀^{m}(ℂ^{n}) ↔ (\underbrace{1, …,}_{m}0, …) (m < n/2).

The conclusion from these calculations is that

V_{μ}[δ̃] ≃ ∑_{m<n/2}(\underbrace{1, …,}_{k}0, …) ⊗ (\underbrace{1, …,}_{m}0, …)[1].

The summand on the right side has been computed in Lemma 12.9 in case m = k or m = k − 1; these give the Weyl group representations (n − k) × (k) and (k − 1) × (n − k + 1). The dimensions of these two representations add up to \binom{n+1}{k}, which is easily computed to be the dimension of V_{μ}[δ̃]; so the other summands must be zero.

(3) (1, …, ±1) ⊗ Spin, n odd. This is almost identical to case (2).

(4) (\underbrace{1, …,}_{k}0, …) ⊗ Spin_{ϵ}, n even. We begin as in (2), computing

HomM̃(δ̃_{γ}, μ_{L} ⊗ Spin_{ϵ}) ≃ Hom_{M}(1, μ_{L} ⊗ Spin_{ϵ} ⊗ Spin_{γ}^{∗})

It is straightforward to see

Spin_{ϵ} ⊗ Spin_{γ}^{∗} ≃ E ⊕ ∑ ⋀^{k}(ℂ^{n})

where the sum is taken over 0 ≤ m < n/2, k ≡ ϵγ (mod 2). The “error term” E appears if and only if n/2 ≡ ϵγ (mod 2); in that case it is one of the two summands of ⋀^{n/2} ℂ^{n}, and has highest weight of the form (1, …, 1)_{ϕ}. The argument from this point is as in (2), using Lemma 12.9.

This completes the proof. □

From Proposition 11.1 and the tables we conclude:

Proposition 12.11. Suppose G is a classical group or is simply laced, and δ̃ is an irreducible pseudospherical representation of M̃. Then Σ_{W} (G̃, δ̃) (Definition 10.7) is the set of all petite representations of W.

13. Intertwining operator methods

We begin with a formal construction. Recall that Δ is the root system of G, R the root lattice, and Δ^{+} the set of positive roots corresponding to B. Write S for the set of simple roots.

Recall Definition 4.4. We define a map τ: Δ → {1, 2} by

τ(α) = \begin{cases}2 & if α is metaplectic in G̃ \\ 1 & if α is not metaplectic in G̃. \end{cases}

We now fix

(13.1)(a)  ν ∈ 𝔞_{ℂ}^{∗}, ν^{ℓ} = dΨ^{t}(ν) ∈ (𝔞^{ℓ})_{ℂ}^{∗}.

The characteristic property (6.2) of Ψ implies that

(13.1)(b)  τ(α)⟨ν, α^{∨}⟩ = ⟨ν^{ℓ}, (α^{ℓ})^{∨}⟩  (α ∈ Δ).

We now introduce a series of linear operators related to intertwining operators. For the first, suppose (σ, E) is a representation of the Weyl group W, and α ∈ Δ. The reflection s_{α} has order 2, so it acts in E with eigenvalues +1 and −1. We write

E = E(+, α) ⊕ E(−, α)

accordingly. We now define

(13.2)(a)  A(s_{α}, σ, ν^{ℓ}) = (1 + ⟨ν^{ℓ} , α^{∨}⟩)^{−1} (I + ⟨ν^{ℓ}, α^{∨}⟩σ(s_{α}))

This makes sense as long as ⟨ν^{ℓ}, α^{∨}⟩ ≠ −1. A little more explicitly,

A(s_{α}, σ, ν^{ℓ})v = \frac{1 + ϵ⟨ν, α^{∨}⟩}{1 + ⟨ν, α^{∨}⟩} v (v ∈ E(ϵ, α)).

Similarly, define

(13.2)(b)  Ã(s_{α}, σ, ν) = (1 + τ(α)⟨ν, α^{∨}⟩)^{−1} (I + τ(α)⟨ν, α^{∨}⟩σ(s_{α}))

This makes sense as long as τ(α)⟨ν, α^{∨}⟩ ≠ −1. Explicitly,

Ã(s_{α}, σ, ν)v = \frac{1 + ϵτ(α)⟨ν, α^{∨}⟩}{1 + τ(α)⟨ν, α^{∨}⟩} v (v ∈ E(ϵ, α)).

Suppose now that w ∈ W is any element. Choose a reduced decomposition

(13.3)(a)  w = s_{α_{n}}s_{α_{n−1}} ⋯ s_{α_{1}}

where each α_{i} ∈ S. For 1 ≤ i ≤ n define w^{(i)} = s_{α_{i}}s_{α_{i−1}} ⋯ s_{α_{1}}, and set w^{(0)} = 1. Define

(13.3)(b)  A(w, σ, ν^{ℓ}) = ∏_{i=1}^{n} A(s_{α_{i}}, σ, w^{(i−1)}ν^{ℓ})

and

(13.3)(c)  Ã(w, σ, ν) = ∏_{i=1}^{n} Ã(s_{α_{i}}, σ, w^{(i−1)}ν)

These linear operators are defined for ν^{ℓ} and ν not belonging to n hyperplanes. Explicitly, write

Δ^{+}(w) = {β ∈ Δ^{+} | w^{−1}(β) ∈ Δ^{−}}.

This is a set of n positive roots, and it turns out that

Δ^{+}(w) = {(w^{(i−1)})^{−1}(α_{i}) | 1 ≤ i ≤ n}.

It follows that A(w, σ, ν^{ℓ}) is defined as long as

(13.4)(a)  ⟨ν^{ℓ}, β^{∨}⟩ ≠ −1, β ∈ Δ^{+}(w).

Similarly, Ã(w, σ, ν) is defined as long as

(13.4)(b)  τ(β)⟨ν, β^{∨}⟩ ≠ −1, β ∈ Δ^{+}(w).

In particular, the operators are defined whenever ν^{ℓ} and ν have dominant real part. If w_{0} is the long element of W, we will be particularly interested in the operators

(13.5)  A(σ, ν^{ℓ}) = A(w_{0}, σ, ν^{ℓ}),  Ã(σ, ν) = Ã(w_{0}, σ, ν).

Our notation implies that A(w, σ, ν^{ℓ}) is independent of the choice of reduced decomposition of w in (13.3)(a). This is true, and not very difficult to prove; but since we will make no use of the fact, we omit the argument. We are interested in these operators because of their relationship to intertwining operators, and we are interested in intertwining operators because they can be used to construct invariant Hermitian forms on representations. A matrix like A(σ, ν^{ℓ}) will turn out to be part of the matrix of an invariant Hermitian form. To see that this makes sense, we need to know that A(σ, ν^{ℓ}) is (sometimes) a self-adjoint operator. (Because E is the space of the unitary Weyl group representation σ, it carries a natural Hilbert space structure; this is what defines adjoints for operators on E.) Because

σ(s_{α})^{∗} = σ(s_{α})^{−1} = σ(s_{α}),

we get immediately

A(s_{α}, σ, ν^{ℓ})^{∗} = A(s_{α}, σ, \overline{ν}^{ℓ}) = A(s_{α}, σ, −s_{α}(\overline{ν}^{ℓ})),

and similarly for Ã. An easy calculation with (13.3) gives

(13.6) A(w, σ, ν^{ℓ})^{∗} = A(w^{−1}, σ, −w(\overline{ν}^{ℓ})),  Ã(w, σ, ν)^{∗} = Ã(w^{−1}, σ, −w(\overline{ν})).

In particular, A(σ, ν^{ℓ}) is self-adjoint if w_{0}(ν^{ℓ}) = −\overline{ν}^{ℓ}.

Lemma 13.7. Suppose ν ∈ 𝔞_{ℂ}^{∗} and ν^{ℓ} = dΨ^{t}(ν) ∈ (𝔞^{ℓ})_{ℂ}^{∗}. Then for every representation σ of W, and every element w ∈ W, we have

Ã(w, σ, ν) = A(w, σ, ν^{ℓ}).

Proof. This is immediate from (13.1), (13.2), and (13.3). □

We turn now to a description of standard intertwining operators for pseudospherical principal series representations. We need these both for the linear group G^{ℓ} and for the nonlinear group G̃. It is therefore convenient to generalize for a moment to the setting of Definition 4.1, so that G̃ is a (possibly trivial) two-fold cover of a linear group G. Fix a pseudospherical representation δ̃ of M̃, and a weight ν ∈ 𝔞_{ℂ}^{∗}.

For each w ∈ W there is a meromorphic family of standard intertwining operators

T (w, δ̃, ν): I(δ̃, ν) → I(wδ̃, wν),

which we normalize by requiring that they act as the identity on the unique pseudospherical K̃-type μ_{0}. Because δ̃ is pseudospherical, wδ̃ is equivalent to δ̃ (Lemma 4.11); but writing the codomain in this form is convenient for thinking about generalizations to non-pseudospherical principal series. These operators are analytic at least for ν having dominant real part; the precise location of the poles will be clear from Lemma 13.13. We will be particularly interested in the long intertwining operator

T (δ̃, ν) = T (w_{0}, δ̃, ν),

whose image (when the real part of ν is dominant) is J(δ̃, ν). Fix a K̃-type μ. By (10.4)(a), the intertwining operators induce maps

T (w, μ, δ̃, ν) ∈ Hom(V_{μ}[δ̃], V_{μ}[wδ̃]).

Since we are realizing δ̃ on the space of the pseudospherical K̃-type μ_{0}, we get natural identifications

V_{μ}[δ̃] = HomM̃(V_{μ}, V_{μ_{0}}) = V_{μ}[wδ̃].

These identifications give

(13.8)  T (w, μ, δ̃, ν) ∈ End(V_{μ}[δ̃]).

This space inherits a Hilbert space structure from V_{μ} and V_{μ_{0}}. In analogy with (13.6), one can calculate easily that

(13.9)  T (w, μ, δ̃, ν)^{∗} = T (w^{−1}, μ, wδ̃, −\overline{ν}).

If w_{0}ν = −\overline{ν}, it follows that T (μ, δ̃, ν) is self-adjoint. These self-adjoint operators define the invariant Hermitian forms on the spherical representations, giving the following classical and fundamental result.

Theorem 13.10. Suppose that ν ∈ 𝔞^{∗} is real and dominant. Then J(δ̃, ν) admits a non-zero invariant Hermitian form if and only if w_{0}ν = −ν. When the form exists, it is definite if and only if T (μ, δ̃, ν) is positive semi-definite for all K̃-types μ.

If G is a linear group this is standard; see for example [Kn, Chapter VII] or [Ba1]. The same proof holds in the nonlinear case with minor modifications.

Suppose now that Σ is a collection of irreducible representations of K̃.

Definition 13.11. Suppose that J(δ, ν) is an irreducible Hermitian pseudospherical representation of G̃; normalize its invariant Hermitian form to be positive on the pseudospherical K̃-type μ_{0}. We say J(δ̃, ν) is unitary on Σ if its invariant form is positive semi-definite on the μ-isotypic component of J(δ̃, ν) for all μ ∈ Σ.

We say Σ detects unitarity for δ̃ if each Hermitian pseudospherical representation J(δ̃, ν) is unitary if and only if it is unitary on Σ. We say Σ detects spherical unitarity if it detects unitarity for the trivial character δ̃ = 1.

To say that Σ detects unitarity for δ̃ means that for each non-unitary but Hermitian representation J(δ̃, ν) (with ν dominant real), there is a μ ∈ Σ so that the operator T (μ, δ̃, ν) has a strictly negative eigenvalue.

The operator T (w, δ̃, ν) has a factorization analogous to (13.3)(b):

(13.12)  T (w, δ̃, ν) = ∏_{i=1}^{n} T (s_{α_{i}}, w^{(i−1)}δ̃, w^{(i−1)}ν);

here we are using a reduced decomposition of w exactly as in (13.3)(b). This induces a factorization

T (w, δ̃, μ, ν) = ∏_{i=1}^{n} T (s_{α_{i}}, μ, w^{(i−1)}δ̃, w^{(i−1)}ν).

Each factor T (s_{α_{i}}, w^{(i−1)}δ̃, w^{(i−1)}ν) is induced from an operator on a principal series representation of (essentially) SL(2, ℝ) or S̃L(2, ℝ); so we calculate that operator first.

Recall from (4.12) that

Ad(m̃)Z_{α} = ±Z_{α} for all m̃ ∈ M̃.

For k ∈ \frac{1}{2}ℤ, define

V_{μ}(k) = {v ∈ V_{μ} | dμ(iZ_{α})(v) = kv}.

For k ≥ 0, we will also use

V_{μ}(±k) = V_{μ}(k) + V_{μ}(−k).

Recall m̃_{α} = exp(πZ_{α}). Its square m_{α}^{2} is equal to the central element −1 if α is metaplectic, and to 1 if α is not metaplectic (see the discussion after (4.6)). It follows that V_{μ}(±k) is zero unless k ∈ ℤ + \frac{1}{2} if α is metaplectic, or k ∈ ℤ otherwise. Thus (writing \frac{1}{2}ℤ_{+} = \{0, 1/2, 1, … \}) we have

V_{μ} = ∑_{k∈ \frac{1}{2} ℤ_{+}} V_{μ}(±k),

an M̃-invariant decomposition of V_{μ}. Recalling that we are realizing δ̃ on the pseudospherical K̃-type μ_{0}, this gives

V_{μ}[δ] ≃ ⊕_{k∈ \frac{1}{2} ℤ_{+}} HomM̃(V_{μ}(±k), V_{μ_{0}}).

The summands satisfy

HomM̃(V_{μ}(±k), V_{μ_{0}}) = 0 \begin{cases}if α is metaplectic and k ∉ ℤ + \frac{1}{2}, or \\ if α is not metaplectic and k ∉ 2ℤ. \end{cases}

Lemma 13.13. The intertwining operator T (s_{α}, μ, δ̃, ν) acts on

HomM̃(V_{μ}(±k), V_{μ_{0}}) (k ≥ 0)

by 1 if k = 0 or k = 1/2, and for k ≥ 3/2 by the scalar:

(13.14)  ∏_{0≤j≤[(k−1)/2]} \frac{(k − 1 − 2j) − ⟨ν, α^{∨}⟩}{(k − 1 − 2j) + ⟨ν, α^{∨}⟩},

with the product running over integer values of j. In more detail,

(13.15)(a)  \frac{1 − ⟨ν, α^{∨}⟩}{1 + ⟨ν, α^{∨}⟩} · \frac{3 − ⟨ν, α^{∨}⟩}{3 + ⟨ν, α^{∨}⟩} ⋯ \frac{(k − 1) − ⟨ν, α^{∨}⟩}{(k − 1) + ⟨ν, α^{∨}⟩} k ≥ 2 even

(13.15)(b) \frac{1/2 − ⟨ν, α^{∨}⟩}{1/2 + ⟨ν, α^{∨}⟩} · \frac{5/2 − ⟨ν, α^{∨}⟩}{5/2 + ⟨ν, α^{∨}⟩} ⋯ \frac{(k − 1) − ⟨ν, α^{∨}⟩}{(k − 1) + ⟨ν, α^{∨}⟩} k ∈ \frac{3}{2} + 2ℤ_{+}

(13.15)(c) \frac{3/2 − ⟨ν, α^{∨}⟩}{3/2 + ⟨ν, α^{∨}⟩} · \frac{7/2 − ⟨ν, α^{∨}⟩}{7/2 + ⟨ν, α^{∨}⟩} ⋯ \frac{(k − 1) − ⟨ν, α^{∨}⟩}{(k − 1) + ⟨ν, α^{∨}⟩} k ∈ \frac{5}{2} + 2ℤ_{+}

The lemma is proved by a calculation in SL(2), which we omit.

We can decompose V_{μ_{0}} according to eigenspaces of ℤ_{α} just as we did V_{μ}. The assumption that V_{μ_{0}} is pseudospherical means that only V_{μ_{0}}(0) and V_{μ_{0}}(±1/2) can be non-zero. Then

(13.16)  V_{μ}[δ] = ∑_{k,ℓ∈ \frac{1}{2} ℤ_{+}} HomM̃(V_{μ}(±k), V_{μ_{0}}(±ℓ)).

The element m_{α} acts by the scalar i^{2k} on V_{μ}(k) and on V_{μ_{0}}(k). Therefore an element ϕ of HomM̃(V_{μ}, V_{μ_{0}}) must satisfy

ϕ(V_{μ}(k)) ⊂ ∑_{k−ℓ∈2ℤ} V_{μ_{0}}(ℓ).

Since exp(πZ_{α}/2) is a representative of s_{α} ∈ W, we can now calculate that in the representation σ of W on V_{μ}[δ], σ(s_{α}) acts on HomM̃(V_{μ}(±k), V_{μ_{0}}) by the scalar

(13.17)(a)  σ(s_{α}) = \begin{cases}(−1)^{k/2} & (k ∈ 2ℤ) \\ (−1)^{(k−1/2)/2} & (k ∈ \frac{1}{2} + 2ℤ_{+}) \\ (−1)^{(k+1/2)/2} & (k ∈ \frac{3}{2} + 2ℤ_{+} \end{cases}

More explicitly,

(13.17)(b)  σ(s_{α}) = \begin{cases}+1 & on HomM̃(V_{μ}(0), V_{μ_{0}}) \\ −1 & on HomM̃(V_{μ}(±2), V_{μ_{0}}) \\ +1 & on HomM̃(V_{μ}(±1/2), V_{μ_{0}}) \\ −1 & on HomM̃(V_{μ}(±3/2), V_{μ_{0}}) \end{cases}

Proposition 13.18. Suppose δ̃ is pseudospherical, μ is petite, and

ϕ ∈ HomM̃(V_{μ}, V_{μ_{0}})

is an eigenvector of σ(s_{α}). Then

T (s_{α}, μ, δ̃, ν)ϕ = \begin{cases}ϕ & σ(s_{α})(ϕ) = ϕ \\ \frac{1 − ⟨ν, α^{∨}⟩}{1 + ⟨ν, α^{∨}⟩}ϕ & σ(s_{α})(ϕ) = −ϕ, α not metaplectic \\ \frac{1/2 − ⟨ν, α^{∨}⟩}{1/2 + ⟨ν, α^{∨}⟩}ϕ & σ(s_{α})(ϕ) = −ϕ, α metaplectic \end{cases}

In particular, the action of T (s_{α}, μ, δ̃, ν) on V_{μ}[δ] is determined by the Weyl group representation σ.

Proof. This follows from (13.17)(b) and Lemma 13.13.

□

Corollary 13.19. Suppose δ̃ is a pseudospherical representation of M̃, and μ is a petite representation of K̃. Let σ be the representation of W on V_{μ}[δ̃], and suppose w ∈ W. If G̃ is the trivial cover of the linear group G, then

(13.20)(a)  T (w, μ, δ̃, ν) = A(w, σ, ν).

If G̃ is the nonlinear double cover of a simply connected group, and δ̃ is a genuine pseudospherical representation of M̃, then

(13.20)(b)  T (w, μ, δ̃, ν) = Ã(w, σ, ν).

Proof. This follows from corresponding identities for T (s_{α}, μ, δ̃, ν) and A(s_{α}, σ, ν) (cf. (13.2)(a) and Proposition 13.18) and the factorizations (13.3)(b) and (13.12).

□

Now suppose we are in the setting of Conjecture 7.1. In particular we have fixed a pseudospherical representation δ̃ for G̃. We write K^{ℓ}, A^{ℓ}, etc. for objects associated to G^{ℓ}. We restrict consideration to real infinitesimal character, so fix ν ∈ 𝔞_{0}^{∗} and let ν^{ℓ} = dΨ^{t}(ν).

Proposition 13.21. Suppose Σ_{W} (G̃, δ̃) = Σ_{W} (G^{ℓ}, 1) (Definition 10.7). Then J(δ̃, ν) is unitary on ΣK̃(G̃, δ̃) (Definition 13.11) if and only if J(ν^{ℓ}) is unitary on Σ_{K^{ℓ}}(G^{ℓ}, 1).

Proof. Suppose J(δ̃, ν) is unitary on ΣK̃(G̃, δ̃). Then Ã(σ, ν) is positive semidefinite for all σ ∈ Σ_{W} (G̃) by (13.20)(a). By Lemma 13.7, A(σ, ν^{ℓ}) is positive semi-definite for all σ ∈ Σ_{W} (G̃) = Σ_{W} (G^{ℓ}). By (13.20)(b), J(ν^{ℓ}) is unitary on Σ_{K^{ℓ}}(G^{ℓ}, 1). The reverse direction is identical. □

Corollary 13.22. Recall the notation of Conjecture 7.1. Assume that Σ_{W} (G̃, δ̃) = Σ_{W} (G^{ℓ}) (Definition 10.7).

If Σ_{K^{ℓ}}(G^{ℓ}, 1) detects spherical unitarity for G^{ℓ} (Definition 13.11), then

dΨ^{t} (CS(δ̃, G̃)) ⊂ CS(G^{ℓ}).

If ΣK̃(G̃, δ̃) detects unitarity for δ̃, then

CS(G^{ℓ}) ⊂ dΨ^{t} (CS(δ, G̃)) .

Proof. Assume Σ_{K^{ℓ}}(G^{ℓ}, 1) detects spherical unitarity for G^{ℓ}, and suppose J(δ̃, ν) is unitary. By the Proposition J(ν^{ℓ}) is unitary on Σ_{K^{ℓ}}(G^{ℓ}, 1), and therefore unitary since Σ_{K^{ℓ}}(G^{ℓ}, 1) detects spherical unitarity. The other direction is identical. □

Here is the deep result that gives substance to these elementary calculations.

Theorem 13.23 ([Ba1]). If G is a classical linear group, then Σ_{K}(G, 1) detects spherical unitarity.

By Lemma 10.9 and Proposition 12.11, Σ_{W} (G̃, δ̃) = Σ_{W} (G^{ℓ}, 1), so by Corollary 13.22 and Theorem 13.23 we conclude:

Theorem 13.24. Retain the notation of Conjecture 7.1 and assume G is a classical group. Then

dΨ^{t} (CS(δ̃, G̃)) ⊂ CS(G^{ℓ}).

If G = Sp(2n) we will prove equality in the next Section.

14. CS(Spin(n, n + 1)) = CS(δ̃, S̃p(2n, ℝ))

In this section we use the theta correspondence to study the pseudospherical complementary series of S̃p(2n, ℝ). To better understand the limitations of this technique, first suppose that (G_{1}, G′_{1}) is an irreducible reductive dual pair in Sp(2N, ℝ). Write G̃_{1} and G̃′_{1} for the respective preimages in S̃p(2N, ℝ). Then G̃_{1} and G̃′_{1} are both linear except when (up to reordering) G_{1} = O(p, q) with p + q odd and G′_{1} = Sp(2n, ℝ). In this case G̃_{1} = Õ(p, q) is a linear cover of O(p, q), but G̃′_{1} = S̃p(2n, ℝ). Unfortunately, as mentioned above, S̃p(2n, ℝ) is the only nonlinear group arising this way, and the theta correspondence cannot be used to make conclusions about the other nonlinear groups in Conjecture 7.1.

With this in mind, fix a choice ω of one of the two oscillators for S̃p(2N, ℝ), and let Irr(Õ(p, q)) and Irr(S̃p(2n, ℝ)) denote the set of irreducible genuine representations of the indicated groups. The theta correspondence for the dual pair (O(p, q), Sp(2n, ℝ)) may be interpreted as defining maps (depending on the choice of ω)

θ_{n}^{(p,q)} : Irr(S̃p(2n, ℝ)) ⟶ Irr(Õ(p, q)) ∪ {0}
θ_{(p,q)}^{n}: Irr(Õ(p, q)) ⟶ Irr(S̃p(2n, ℝ)) ∪ {0}.

Here θ_{n}^{(p,q)}(π) = 0 if π does not occur in the correspondence, and likewise for θ_{(p,q)}^{n}. When restricted to those representations that do occur, these maps are inverses of each other. The basic reference for these facts is [H2].

For our purposes, it will be convenient to work directly with O(p, q) instead of Õ(p, q). Fix a genuine character of ξ of Õ(p, q) as in (1.2g) of [AB]. Then π ↦ π ⊗ ξ is a bijection from irreducible representations of O(p, q) to irreducible genuine representations of Õ(p, q). The inverse is given by tensoring with ξ^{−1}: if π is a genuine irreducible representation of Õ(p, q), then π ⊗ ξ^{−1} is irreducible and factors to O(p, q). Define

θ̄_{n}^{(p,q)} : Irr(S̃p(2n, ℝ)) ⟶ Irr(O(p, q)) ∪ {0}

by

θ̄_{n}^{(p,q)}(π) = θ_{n}^{(p,q)} ⊗ ξ^{−1}.

Likewise define

θ̄_{(p,q)}^{n}: Irr(O(p, q)) ⟶ Irr(S̃p(2n, ℝ)) ∪ {0}

by

θ̄_{(p,q)}^{n}(π) = θ_{(p,q)}^{n}(π ⊗ ξ).

Now fix a pseudospherical representation δ̃ for S̃p(2n, ℝ). (The notation δ̃_{n} would be better, but we suppress the subscript.) If follows easily from the correspondence of joint harmonics that for any M the oscillator defining θ̄_{n}^{(M+1,M)} may be chosen so that θ̄_{n}^{(M+1,M)} takes δ̃-pseudospherical representations of S̃p(2n, ℝ) to spherical representations of O(M + 1, 1), and likewise that θ̄_{(M+1,M)}^{n} takes spherical representations to pseudospherical ones. (The relevant calculation, together with appropriate reference to the theory of joint harmonics, is recalled in Section 2, and especially Proposition 2.1, of [AB].) In the setting of Definition 1.1, a pseudospherical representation of G̃ is characterized by its pseudospherical K-type and its infinitesimal character (see the comments after Definition 5.5). The following result is now a consequence of Theorem 5.1 of [AB].

Lemma 14.1. Retain the notation as above. Then

θ̄_{(n+1,n)}^{n}(J_{O(n+1,n)}(ν)) = J_{S̃p(2n,ℝ)}(δ̃, ν);

here ν may be interpreted as a parameter for both O(n + 1, n) and S̃p(2n, ℝ) by identifying the dual Cartan subspaces 𝔞^{∗} by the map dΨ^{t} as in Conjecture 7.1.

Theorem 14.2. The theta correspondence restricts to a bijection

θ̄_{(n+1,n)}^{n}: CS(O(n + 1, n)) ⟶ CS(δ̃, S̃p(2n, ℝ)).

Since Lemma 4.20 shows that CS(O(n + 1, 1)) = CS(Spin(n + 1, n)), this establishes Conjecture 7.1 for G = Sp(2n, ℝ).

Proof. We need to introduce some auxiliary notation. Given an infinitesimal character ν for O(n+1, n), let (ν | ρ_{m}) denote the infinitesimal character for O(m+n+ 1, m+n) obtained by augmenting the coordinates of ν by (1/2, 3/2, …, (2m−1)/2). Let J_{O(m+n+1,m+n)}(ν | ρ_{m}) denote the corresponding spherical representation of O(m + n + 1, m + n). The following facts are crucial.

Fact 0(a). If J_{O(n+1,n)}(ν) is unitary, then each real coordinate of ν (in absolute value) is less than or equal to (2n − 1)/2.

Fact 0(b). If J_{S̃p(2n,ℝ)}(δ̃, ν) is unitary, then each real coordinate of ν (in absolute value) is less than or equal to (2n − 1)/2.

Fact 1. If m ≥ n, then J_{O(m+n+1,m+n)}(ν | ρ_{m}) is unitary if and only if J_{O(n+1,n)}(ν) is unitary.

Fact 2. If m ≥ n + 2, then J_{O(m+n+1,m+n)}(ν | ρ_{m}) is a low-rank representation in the sense of [H1] (see also [Li2]). For the moment accept these facts; the proofs will be given below.

Consider the iterated lift θ̄_{n}^{(n+m+1,n+m)}(θ̄_{(n+1,n)}^{n}(J_{O(n+1,n)}(ν))), a representa-

tion of the group O(n+m+1, n+m). Theorem 8.4 of [AB] (the induction principle) and the correspondence of joint harmonics imply

(14.3)  θ̄_{n}^{(n+m+1,n+m)}(θ̄_{(n+1,n)}^{n}(J_{O(n+1,n)}(ν))) = J_{O(m+n+1,m+n)}(ν | ρ_{m}).

Fix ν ∈ CS(O(n + 1, n)). We now show that J_{S̃p(2n,ℝ)}(δ̃, ν) is unitary, and thus establish one half of the theorem. Choose m ≥ n + 2. Consider π := J_{O(m+n+1,m+n)}(ν | ρ_{m}). By Facts 1 and 2, π is a low-rank unitary representation of O(n + m + 1, n + m). The main theorem of [Li2] thus asserts that there exists a character χ of O(n+m+1, n+m) and an irreducible unitary representation τ of some S̃p(2l, ℝ) with l ≤ (m + n)/2 such that

(14.4)  θ̄_{l}^{(n+m+1,n+m)}(τ) = π ⊗ χ.

Our aim is to show that l = n and that τ equals J_{S̃p(2n,ℝ)}(δ̃, ν). Since τ is unitary,

this gives the desired unitarity of J_{S̃p(2n,ℝ)}(δ̃, ν).

Since π is spherical, the correspondence of joint harmonics implies that the character χ in (14.4) must be trivial, and τ must be δ̃-pseudospherical.

Next we argue that the infinitesimal character correspondence dictates that l = n. To see this note that if τ has infinitesimal character λ, then (see for example [Pr]) θ̄_{l}^{(n+m+1,n+m)}(τ) has infinitesimal character (λ | ρ_{n+m−l}). (Here, as above, λ is playing a dual role as an infinitesimal character for O and S̃p; we will no longer remark on this explicitly.) So, as infinitesimal characters

(14.5)  (λ | ρ_{n+m−l}) = (ν | ρ_{m}).

Using the Weyl group, we may assume each real coordinate above is positive, and we may also view the equality in (14.5) as an equality of (n + m)-tuples of complex numbers. First we prove that l ≥ n. Suppose instead that l < n; so n + m − l > m, and the largest coordinate ρ_{n+m−l} is strictly larger than the largest coordinate of ρ_{m}. Equation (14.5) thus implies that the largest coordinate of ρ_{n+m−l} must be a coordinate of ν. Since we have supposed l < n and are assuming m ≥ n + 2 (in particular, m ≥ n), the largest coordinate of ρ_{n+m−l} is strictly larger that the largest coordinate of ρ_{n}, i.e., (2n − 1)/2. But ν ∈ CS(O(n + 1, n)), and thus Fact 0(a) gives a contradiction. So indeed l ≥ n.

Next we suppose l > n and derive a contradiction. If l > n, the largest coordinate of ρ_{m} is greater than the largest coordinate of ρ_{n+m−l} and Equation (14.5) implies that λ must thus contain the largest coordinate of ρ_{m}. Recall that the theorem of [Li2] implies that l ≤ (m + n)/2; so the assumption l > n implies l < m, and the largest coordinate of ρ_{m} is strictly larger than the largest coordinate of ρ_{l}. So λ contains a coordinate strictly larger than (2l − 1)/2. But τ is pseudospherical, contradicting Fact 0(b). Thus l ≤ n.

The previous two paragraphs imply l = n and that the infinitesimal character of τ equals ν. Already we have seen that τ is pseudospherical, and hence (see after Definition 5.5) is characterized by its infinitesimal character. We conclude

τ = J_{S̃p(2n,ℝ)}(δ̃, ν),

as claimed. This proves one half of the theorem.

The other half of the the theorem is easier and does not rely on Facts 0 or 2. Fix ν ∉ CS(O(n + 1, n)); we show that θ̄_{(n+1,n)}^{n}(J_{O(n+1,n)}(ν) is non-unitary. By Fact 1,

θ̄_{n}^{(n+m+1,n+m)}(θ̄_{(n+1,n)}^{n}(J_{O(n+1,n)}(ν))) = J_{O(m+n+1,m+n)}(ν | ρ_{m})

is non-unitary for m ≥ n. But since m ≥ n (the so-called stable range case [Li1]), θ̄_{n}^{(n+m+1,n+m)} takes unitary representations to unitary representations. So, indeed,

θ̄_{(n+1,n)}^{n}(J_{O(n+1,n)}(ν)) = J_{S̃p(2n,ℝ)}(δ̃, ν)

cannot be unitary.

This completes the proof of Theorem 14.2 up to the proofs of Facts 0, 1, and 2. Fact 0 is standard and we begin there. After acting on ν by the Weyl group, we may suppose that the real coordinates of ν are the first m (for some m with 0 ≤ m ≤ n), and that

ν_{1} ≥ ν_{2} ≥ ⋯ ≥ ν_{m} ≥ 0.

We will prove the contrapositive of Fact 0: that if ν_{1} ≥ n − 1/2, then the corresponding pseudospherical representation is not unitary. In fact we will prove non-unitarity under an even weaker condition. Here is the strengthened version of Fact 0(a).

Lemma 14.6. Under the assumptions above on ν, suppose that either (1) ν_{m} > 1/2, or

(2) for some a with 1 ≤ a ≤ m − 1, we have ν_{a} − ν_{a+1} > 1.

Then the spherical principal series representation J_{O(n+1,n)}(ν) is not unitary.

Clearly the assumption that ν_{1} > n − 1/2 implies that at least one of the two conditions in the lemma is satisfied.

Proof. In the first case, set a equal to m. Define

ν_{t} = (t, …, t, 0, …, 0) + ν,

with the first a coordinates equal to t. We consider the family of spherical representations J_{O(n+1,n)}(ν_{t}) for t ≥ 0. What we are trying to show is that J_{O(n+1,n)}(ν_{0}) = J_{O(n+1,n)}(ν) is not unitary. If this representation admits no non-zero invariant Hermitian form, we are done; so we may assume that such a form exists, and normalize it to be positive on the K-fixed vector. Knapp’s “formal symmetry condition” for the existence of such a form is inherited by all ν_{t}; so we may endow all the representations J_{O(n+1,n)}(ν_{t}) (for t ≥ 0) with non-degenerate invariant Hermitian forms positive on the spherical vectors. We will investigate this family of forms on the “𝔭 K-type”: the representation of O(n + 1) × O(n) on ℂ^{n+1} ⊗ ℂ^{n}, which we will call μ_{1}. This is a petite K-type, corresponding to k = 1 in the first case of Lemma 12.7. It is the adjoint representation of K on 𝔭_{ℂ} in the Cartan decomposition 𝔤 = 𝔨 + 𝔭. According to Lemma 12.7, the representation of W on the space V_{μ_{1}}(1) of M-fixed vectors is σ_{1} = (n−1)×(1), which is the reflection representation of W. According to Corollary 13.19, the Hermitian form for J_{O(n+1,n)}(ν_{t}) on the K-type μ_{1} is given by the matrix A(σ_{1}, ν_{t}). What we will show is that for all t ≥ 0, A(σ_{1}, ν_{t}) fails to be positive definite; this will prove in particular that J_{O(n+1,n)}(ν) is not unitary.

The proof that A(σ_{1}, ν_{t}) fails to be positive definite has two steps. First, we will show that the signature of A(σ, ν_{t}) is constant for all t ≥ 0, for any Weyl group representation σ. It therefore suffices to prove that A(σ_{1}, ν_{t}) is indefinite for large t. This we will do by inspection of the Casimir operator for G.

For the first step, let

L = GL(a, ℝ) × O(n − a + 1, n − a)

be the maximal Levi subgroup of G defined by the weight (1, …, 1, 0, …, 0) (with a terms equal to one). Let P = LU be the corresponding standard parabolic subgroup. The roots in U are

e_{i} ± e_{j}, e_{i} (1 ≤ i ≤ a, a + 1 ≤ j ≤ n)

What follows from the hypothesis on a in the lemma is that

(14.7)  ⟨ν, α^{∨}⟩ is strictly greater than 1 or non-real

for all roots α in U. (The first possibility occurs if j is less than or equal to m or absent, and the second if j is greater than m.)

In the factorization (13.3)(b), each factor is invertible operator ⟨ν, α^{∨}⟩ = ±1. This is a consequence of the definition of the factors in Proposition (13.2)(a). Now it follows from (14.7) that the rank of A(σ, t) is independent of t (for t ≥ 0); in fact it is equal to the rank of the corresponding operator for the Weyl group of L. Because the positive and negative parts of the signature are semicontinuous in t, the constancy of the rank (equal to the sum of the positive and negative parts) implies that the signature is constant as well.

It remains to show that A(σ_{1}, t) is not positive definite for large positive t. For this we can invoke [Sal, Lemma 6.3]; since the argument in our case is very simple, we will give a proof. It is most convenient to work with the group representation. Let v_{0}(t) be a non-zero K-fixed unit vector in J_{O(n+1,n)}(ν_{t}). We use the invariant bilinear form on 𝔤 extending our chosen inner product on 𝔞, and let Ω be the corresponding Casimir operator. In a representation π with infinitesimal character λ, we have

π(Ω) = ⟨λ, λ⟩ − ⟨ρ, ρ⟩

(with ρ the half sum of a set of positive roots). In particular, Ω must act on J(ν_{t}) by the scalar

⟨ν_{t}, ν_{t}⟩ − ⟨ρ, ρ⟩.

This is a quadratic polynomial in t with leading term at^{2}, so it certainly has positive real part for t sufficiently large.

Now assume that the invariant Hermitian form ⟨·, ·⟩ is positive on the 𝔭 Ktype μ_{1}. We will compute the Casimir eigenvalue in a different way and get a contradiction. Choose an orthonormal basis {X_{i}} for 𝔭, and an orthogonal basis {Y_{j}} of 𝔨 consisting of vectors of length −1. This is possible, since the invariant form on 𝔤 is positive on 𝔭 and negative on 𝔨. Then

Ω = ∑_{i} X_{i}^{2} − ∑_{j} Y_{j}^{2}.

Since v_{0}(t) is a unit vector in J(ν_{t}), the Casimir eigenvalue is

⟨Ω · v_{0}(t), v_{0}(t)⟩_{t} = ∑_{i}⟨X_{i}^{2} · v_{0}(t), v_{0}(t)⟩_{t} + ∑_{j}⟨Y_{j}^{2} · v_{0}(t), v_{0}(t)⟩_{t}.

Since v_{0}(t) is K-fixed, all terms in the second sum are zero. In the first sum, we can use the invariance of the inner product to move one X_{i} to the right with a sign change, getting

∑_{i} −⟨X_{i} · v_{0}(t), X_{i} · v_{0}(t)⟩_{t}.

Now v_{0}(t) is K-fixed and the X_{i} transform by the representation μ_{1}; so all of the vectors X_{i}·v_{0}(t) must belong to the μ_{1} K-type of J(ν_{t}). By hypothesis the invariant Hermitian form is positive on this K-type, so each vector X_{i} ·v_{0}(t) has non-negative length. The conclusion is that the eigenvalue of the Casimir must be non-positive, contradicting our earlier calculation that the eigenvalue has positive real part for large t. This contradiction proves the invariant form must be non-positive on μ_{1} for large t. □

Here is the strengthened version of Fact 0(b).

Lemma 14.8. Suppose that the real coordinates of ν are the first m, and that

ν_{1} ≥ ν_{2} ≥ ⋯ ≥ ν_{m} ≥ 0.

Assume that either (1) ν_{m} > 1/2, or

(2) for some a with 1 ≤ a ≤ m − 1, we have ν_{a} − ν_{a+1} > 1.

Then the pseudospherical principal series representation J_{S̃p(2n,ℝ)}(ν) is not unitary.

Proof. We follow as closely as possible the proof of Lemma 14.6. We define ν_{t} exactly as in the preceding proof, and consider the family of pseudospherical representations J_{S̃p(2n,ℝ)}(ν_{t}). As before we may assume that J_{S̃p(2n,ℝ)}(ν_{0}) admits an

invariant Hermitian form. Suppose for definiteness that δ̃ = δ̃_{+}, so that all these representations contain the K̃-type det^{1/2}. We normalize the forms ⟨·, ·⟩_{t} to be positive on the pseudospherical K̃-type det^{1/2}. We investigate the forms on the K̃-type μ̃_{1} of highest weight

(1/2, …, 1/2, −3/2).

This is a petite K̃-type, corresponding to k = 1 in the first case of Lemma 12.5. The corresponding representation of W is again σ_{1} = (n − 1) × (1), the reflection representation. According to Corollary 13.19, the Hermitian form for J_{S̃p(2n,ℝ)}(ν_{t})

on the K̃-type μ̃_{1} is given by the matrix Ã(σ_{1}, ν_{t}). According to Lemma 13.7, this is equal to A(σ_{1}, ν_{t}). The fact that this matrix is not positive definite was already proved in Lemma 14.6, so we are done. □

There is a small subtlety here. The proof given for failure of positivity of the matrix A(σ_{1}, ν_{t}) in Lemma 14.6 does not trivially work in the setting of Lemma 14.8. The main difficulty is that the tensor product of det^{1/2} with the representation 𝔭_{ℂ} of K̃ contains not only μ̃_{1}, but also the non-petite representation of K̃ of highest weight (5/2, 1/2, …, 1/2). (One can deal with this difficulty with a little more care in the argument.) But the point is that we do not need a new representationtheoretic argument to prove that A(σ_{1}, ν_{t}) is not positive, since it is exactly the same matrix appearing in the linear case.

Facts 1 and 2 are consequences of the shape of the main results of [Ba1]; here we simply supply a few details.

We begin with a brief summary of [Ba1]. For notational ease, we assume G = O(n + 1, n), O(n, n) or Sp(2n, R). (Of course, the first case is the one relevant for Facts 1 and 2, but it is conceptually easier to work in this slightly more general setting.) It is convenient also to use a possibly disconnected version of the complex dual group, writing

(14.9)  G_{ℂ}^{∨} = \begin{cases}Sp(2n, ℂ) & if G = O(n + 1, n), \\ O(2n, ℂ) & if G = O(n, n), and \\ O(2n + 1, ℂ) & if G = Sp(2n, ℝ). \end{cases}

As usual, the duality relationship between 𝔤 and 𝔤_{ℂ}^{∨} identifies 𝔞_{ℂ}^{∗} as a Cartan subalgebra of 𝔤_{ℂ}^{∨}, in a way that is canonical up to the action of W. In the usual coordinates for these classical groups, the identification comes from the identification of each space with ℂ^{n}.

To each spherical representation J(ν) of G, [Ba1] first attaches a nilpotent orbit O^{∨} in the complex dual Lie algebra 𝔤_{ℂ}^{∨}. Explicitly, O^{∨} is the unique dense orbit in the G_{ℂ}^{∨} saturation of the 1-eigenspace of ad(ν) on 𝔤_{ℂ}^{∨}.) Let p^{∨} = p(O^{∨}) = {p_{1}^{∨} ≥ p_{2}^{∨} ≥ ⋯ } denote the partition obtained from the Jordan normal form of an element of O^{∨}. We break the partition into two pieces p^{∨} = p_{unip}^{∨} ∪ p_{GL}^{∨} as follows:

p_{unip}^{∨} = one copy of each part of p^{∨} having odd multiplicity,

and p_{GL}^{∨} consists of the remaining parts (each therefore appearing with even multiplicity). The partition p_{unip}^{∨} parametrizes a distinguished nilpotent orbit O_{unip}^{∨} in a classical Lie algebra 𝔤(k)_{ℂ}^{∨} of a complex Lie algebra 𝔤(k)_{ℂ} of rank k and the same type as 𝔤_{ℂ}^{∨}. (For classical groups outside of Type A, an orbit is distinguished if and only if the partition corresponding to its Jordan normal form has no repeated entries.) The partition p_{GL}^{∨} may be written

p_{GL}^{∨} = {n_{1}, n_{1} ≥ n_{2}, n_{2} ≥ ⋯ ≥ n_{r}, n_{r}}.

We now define a Levi subgroup

L = G(k) × GL(n_{1}, ℝ) × ⋯ × GL(n_{r}, ℝ) ⊂ G;

here G(k) is a split classical group of rank k and the same type as G. Let π_{unip} denote the spherical representation of G(k) whose infinitesimal character ν_{unip} is one-half the semisimple element of a Jacobson-Morozov triple for O_{unip}^{∨}. (A distinguished nilpotent orbit is even, and ν_{unip} is therefore integral.) In the course of describing how to calculate the partition p^{∨} from ν, [Ba1] defines characters χ_{i} of each GL(n_{i}) factor of L. These characters are chosen so that J(ν) is a composition factor of Ind_{L}^{G}(π_{unip} ⊗ χ_{1} ⊗ ⋯ ⊗ χ_{r}). (Here we really mean parabolic induction of course, but we are being a little sloppy and not specifying the nilradical.)

Write Z(O^{∨})_{ℂ}^{∨} for the centralizer in G_{ℂ}^{∨} of a Jacobson-Morozov triple for O^{∨}. Because G^{∨} is a symplectic or orthogonal group, this centralizer is is a product of symplectic and orthogonal groups ([CMc]):

Z(O^{∨})_{ℂ}^{∨} = ∏_{i} Sp(2a_{i}, ℂ) × ∏_{j} O(2b_{j}, ℂ) × ∏_{k} O(2c_{k} + 1, ℂ).

(The numbers 2a_{i}, 2b_{j}, and 2c_{k} + 1 are the multiplicities of parts of p^{∨}.) Define Z(O^{∨}) to be the split real form of the dual group,

Z(O^{∨}) = ∏_{i} O(a_{i} + 1, a_{i}) × ∏_{j} O(b_{j}, b_{j}) × ∏_{k} Sp(2c_{k}, ℝ).

From the characters χ_{i}, [Ba1] defines an infinitesimal character ν_{Z} for Z(O^{∨}). Since Z(O^{∨}) is a product of orthogonal and symplectic groups, one may run the algorithms just described and extract from ν_{Z} a Levi subgroup, an orbit in the dual, et cetera, with Z(O^{∨}) now playing the role of G in the previous paragraph. Let O_{Z}^{∨} denote the nilpotent orbit in the dual 𝔷(O^{∨})_{ℂ}^{∨} described by the procedure above. One version of the main theorem of [Ba1] is that (14.10)

J_{G}(ν) is unitary if and only if J_{Z(O^{∨})}(ν_{Z}) is unitary and O_{Z}^{∨} is the zero orbit.

We will deduce Facts 1 and 2 by examining how the data described above change when we pass from J_{O(n+1,n)}(ν) to J_{O(n+m+1,n+m)}(ν | ρ_{m}) for m ≥ n. We retain the notation above for the data attached to ν and add primes for the data attached to ν′ = (ν | ρ_{m}). It is not difficult to show that

(p^{∨})′ = p^{∨} augmented by the part 2m.

Since m ≥ n by assumption (and this assumption is crucial), the part 2m is larger than any in p_{unip}^{∨} and the partition (p_{unip}^{∨})′ consists of distinct parts and thus parametrizes a distinguished orbit unless both m = n and p^{∨} = p_{unip}^{∨} = {2n}. We return to this situation below; for now assume we are not in this case. It follows that if L = G(k) × GL(n_{1}) × ⋯ GL(n_{r}) is the Levi factor factor for ν, the L′ for ν′ is simply G(k + m) × GL(n_{1}) × ⋯ GL(n_{r}). The construction in [Ba1] shows that the characters χ′_{i} coincide with the χ_{i}. Because there is a single distinct extra part in (p^{∨})′, the centralizer subgroup Z′((O^{∨})′) is O(1, 0) × Z(O^{∨}); the infinitesimal character ν′_{Z} coincides with ν_{Z}. Thus determining the unitarity of J_{O(n+1,n)}(ν) and determining the unitarity of J_{O(n+m+1,n+m)}(ν | ρ_{m}) reduces via Equation (14.10) to determining the unitarity of the same representation of Z(O^{∨}). (The extra factor of O(1, 0) = ℤ/2ℤ has no effect on unitarity.) This establishes Fact 1.

Return to the case where m = n and p^{∨} = p_{unip}^{∨} = {2n}. Then J_{O(n+1,n)}(ν) is the trivial representation and of course unitary. On the other hand, J_{O(2n+1,2n)}(ν | ρ_{n}) is clearly contained in the representation unitarily induced from the trivial character on the Siegel GL(2n, ℝ) Levi factor, and hence is unitary. So Fact 1 holds in this case as well.

We now turn to Fact 2, beginning with some preliminaries about low rank representations. Given a representation π, let WF(π) denote Howe’s wavefront set of π; for a definition and some basic properties, see [H0] and [BV0]. Define the rank of a set of S of nilpotent matrices in a classical Lie algebra to be max{rank(X) | X ∈ S}; here rank(X) is the rank of X as a matrix. Set

r(N + 1, N) = \begin{cases}N − 1 & if N is odd; \\ N & if N is even. \end{cases}

A representation π of O(N + 1, N) is said to be of low rank if

rank(WF(π)) < r(N + 1, N).

(This is not the original definition of rank given by Howe [H1]. That this definition coincides with Howe’s is an unpublished theorem of Howe (one direction) and Li (the other); see [Li3]. The corresponding theorem for the symplectic group is stated and proved in [Li4], where it is stated that analogous theorems hold for other classical groups. That the formulation given here is the right one for orthogonal groups follows from [Li2, (14)].) In terms of the partition classification of nilpotent orbits, the rank of a nilpotent matrix is equal to the dimension of the vector space minus the number of parts of the partition; that is, the dimension minus the length of the first column. We conclude that a representation π of O(N + 1, N) has low rank if and only if (14.11)

there is a column in a partition for WF(π) of length ≥ \begin{cases}N + 3 & N odd; \\ N + 2 & N even. \end{cases}

In order to establish Fact 2, we must therefore show that

there is a column in WF(J_{O(m+n+1,m+n)}(ν | ρ_{m}) of length at least m + n + 3.

Consider the Levi subgroup

L′ = O(m + 1, m) × GL(1, ℝ)^{n} ⊂ O(m + n + 1, m + n).

Because ρ_{m} is the infinitesimal character of the trivial representation of O(m+1, m), we have

J_{O(m+n+1,m+n)}(ν | ρ_{m}) is a comp. factor of Ind_{L′}^{O(m+n+1,m+n)}(one-diml. char.).

It follows that

WF(J_{O(m+n+1,m+n)}(ν | ρ_{m})) ⊂ closure of Richardson orbit for L′.

Because the length of the longest column in the partition can only increase in an orbit closure, what we want to show comes down to

there is a column in the Richardson orbit for L′ of length at least m + n + 3.

This Richardson orbit is easy to compute: the corresponding partition is (2n + 1)(1)^{2m}. Its longest column has length 2m + 1, which is at least m + n + 3 since m ≥ n + 2. This establishes Fact 2. □

Corollary 14.12. The K̃ types appearing in the proof of Theorem 13.24 detect pseudospherical unitarity for S̃p(2n, ℝ) (Definition 13.11).

Proof. This follows from Theorem 14.2 and Proposition 13.21. □

Remark 14.13. Half of Theorem 14.2 follows from Theorem 13.24; this corresponds to the part of the argument that we gave second, which used only Fact 1. Because the spherical unitary dual of a split real group is the “same” as the spherical unitary dual of split p-adic groups ([Ba1]) and since the results of [Li2] are valid over any local field of characteristic not 2, the proof of Theorem 14.2 (with suitable but essentially obvious modifications) is valid for p-adic orthogonal and symplectic groups (of characteristic not 2). The resulting correspondence of unitary representations should be a special case of the conjecture discussed in Remark 7.3.

References

[A] J. Adams, Nonlinear covers of real groups, Int. Math. Res. Not. 75 (2004), 4031–4047 [AB] J. Adams, D. Barbasch, Genuine representations of the metaplectic group, Compositio

Math. 113 (1998), 23–66.

[BV0] D. Barbasch, D. A. Vogan, Jr., The local structure of characters, J. Func. Anal. 37 (1980), 27–55.

[Ba1] D. Barbasch, Spherical unitary dual of split classical groups, preprint.

       Available at http://www.math.cornell.edu/∼barbasch/.

[Bo] N. Bourbaki, Groupes et Algèbres de Lie, Chapitres 4, 5, et 6, Masson, Paris, 1981.

[CMc] D. Collingwood, W. M. McGovern, Nilpotent orbits in semisimple Lie algebras, Van Nostrand Reinhold Mathematics Series, New York, 1993.

[Ci] D. Ciubotaru, The Iwahori-spherical unitary dual of the split group of type F_{4}, Represent. Theory 9(2005), 94–137.

[CR] C. W. Curtis and I. Reiner, Methods of representation theory. Vol. I, John Wiley & Sons Inc., New York, 1981.

[H0] R. Howe, Wave front sets of representations of Lie groups, in Automorphic Forms, Representation Theory, and Arithmetic, Tata Inst. Fund. Res. Studies in Math., volume 10. Tata Institute for Fundamental Research, Bombay, 1981.

[H1] R. Howe, On a notion of rank for unitary representations of the classical groups, in Harmonic analysis and group representations, 223–331, Liguori (Naples) 1982.

[H2] R. Howe, Transcending classical invariant theory, J. Amer. Math. Soc. 2 (1989), 535–552.

[Hu1] J.-S. Huang, The unitary dual of the universal covering group of GL(n, ℝ), Duke Math. J. 61 (1990), 705–745.

[Hu2] J.-S. Huang, Metaplectic correspondences and unitary representations, Compositio Math. 80 (1991), 309–322.

[Kn] A. W. Knapp, Representation Theory of Semisimple Groups: An Overview Based on Examples, Princeton Mathematical Series 36, Princeton University Press, Princeton, New Jersey, 1986.

[Li1] J.-S. Li, Singular unitary representations of classical groups, Invent. Math. 97 (1989), 237–255.

[Li2] J.-S. Li, On the classification of irreducible low rank unitary representations of classical groups. Compositio Math. 71 (1989), 29–48.

[Li3] J.-S. Li, Unipotent representations attached to small nilpotent orbits, unpublished lecture notes from a 1997 AMS conference in Seattle, available at

       http://www.math.umd.edu/∼jda/seattle proceedings/.

[Li4] J.-S. Li, On the singular rank of a representation, Proc. Amer. Math. Soc. 106 (1989), 567–571.

[Ma] H. Matsumoto, Sur les sous-groupes arithmétiques des groupes semi-simples déployés, Ann. Sci. Ec. Norm. Sup. 2 (1969), 1–62.

[Pr] T. Przebinda, The duality correspondence of infinitesimal characters, Colloq. Math. 70 (1996), 93–102.

[Sal] S. Salamanca, The unitary dual of some classical Lie groups, Compositio Math. 68 (1988), 251–303.

[Sa] G. Savin, Local Shimura correspondence, Math. Ann. 280 (1988), 185–190.

[Sa2] G. Savin, On unramified representations of covering groups, J. Reine Angew. Math. 566 (2004), 111–134.

[Sc] W. Schmid, “On the characters of the discrete series (the Hermitian symmetric case),” Invent. Math. 30 (1975), 47–144.

[St] R. Steinberg, Générateurs, relations, et revêtements de groupes algébriques, 113–127 in Colloq. Théorie des Groupes Algébriques (Bruxelles, 1962). Librairie universitaire, Louvain, 1962.

[V0] D. A. Vogan, Jr., The algebraic structure of the representations of semisimple Lie groups I, em Ann. Math. 109 (1979), 1–60.

[V2] D. A. Vogan, Jr., The unitary dual of GL(n) over an Archimedian field, Invent. Math. 83 (1986), 449–505.

[V3] D. A. Vogan, Jr., The unitary dual of G_{2}, Invent. Math. 116 (1994), 677–791.

[V4] D. A. Vogan, Jr., Representations of Real Reductive Lie Groups, Progress in Mathematics 15, Birkhäuser, Boston, 1981.

[W] N. R. Wallach, Real Reductive Groups II, Academic Press, San Diego, 1992.

Department of Mathematics, University of Maryland, College Park, Maryland 20742

E-mail address: jda@math.umd.edu

Department of Mathematics, Cornell University, Ithaca, New York 14853

E-mail address: barbasch@math.cornell.edu

Department of Mathematics, Western Michigan University, Kalamazoo, Michigan 49008

E-mail address: paula@wmich.edu

Department of Mathematics, University of Utah, Salt Lake City, Utah 84112

E-mail address: ptrapa@math.utah.edu

Department of Mathematics, Massachusetts Institute of Technology, Cambridge, Massachusetts, 02138

E-mail address: dav@math.mit.edu