Isolated unitary representations.

David A. Vogan, Jr. * Department of Mathematics Massachusetts Institute of Technology Cambridge, Massachusetts 02139

In this paper we collect some facts about the topology on the space of irreducible unitary representations of a real reductive group. The main goal is Theorem 10, which asserts that most of the “cohomological” unitary representations for real reductive groups (see [VZ]) are isolated. Many of the intermediate results can be extended to groups over any local field, but we will discuss these generalizations only in remarks. The foundations of the topological theory of the unitary dual are actually more easily available in the nonarchimedean case, particularly in the work of Tadić [Tad1] and [Tad3]. In the archimedean case the best results are due to Miličić, but unfortunately only part of this has been published in [Mil] and [Mil3].

Suppose then that G is a real reductive Lie group. Write Ĝ for the unitary dual of G. The Fell topology on Ĝ is defined as follows. Suppose S ⊂ Ĝ. An irreducible unitary representation π belongs to the closure of S if and only if every matrix coefficient (equivalently, a single non-zero matrix coefficient) of π is the uniform limit on compact sets of matrix coefficients of elements of S. A convenient reference for the definition is [Wal], section 14.7. Write Π(G) for the set of infinitesimal equivalence classes of irreducible admissible representations of G. We regard Ĝ as a subset of Π(G). It is not difficult to impose on Π(G) a topology making Ĝ a closed subspace, but we will have no need to do so. If for example G = A is a vector group, then Â may be identified (topologically) with the real vector space i𝔞_{0}^{∗} of imaginary-valued linear functionals on the Lie algebra 𝔞_{0} of A. Similarly, Π(A) may be identified with the complex vector space 𝔞^{∗} of all complex-valued linear functionals on 𝔞_{0}. If G = K is compact, then K̂ = Π(K) is a discrete space. The general situation combines the features of these extreme cases. The unitary dual Ĝ is more or less a noncompact real polyhedron (some possible local pathologies are explained after Theorem 2), and Π(G) is more or less a complexification of Ĝ.

It is convenient to impose on G the hypotheses of [Green], 0.1.2: essentially that G be a linear group with abelian Cartan subgroups. These hypotheses are satisfied if G is the group of real points of a connected reductive algebraic group defined over ℝ. We fix a maximal compact subgroup K of G, with corresponding Cartan involution θ. By Harish-Chandra’s subquotient theorem, K is a “large compact subgroup of G” in the sense that a fixed irreducible representation τ of K occurs in irreducible unitary representations of G with multiplicity bounded by a constant depending only on τ.

Definition 1. The Hecke algebra of G is the convolution algebra H(G) of compactly supported complexvalued measures μ on G having the following properties.

a) The measure μ is a smooth multiple of Haar measure on G: μ = f(g)dg for some compactly supported smooth function f.

b) The measure μ is left and right K-finite; that is, its left and right translates by K span a finitedimensional space. (It is equivalent to require this of the function f in (a).) Suppose (π, H_{π}) is an irreducible admissible Hilbert space representation of G and μ ∈ H(G). Define

π(μ) = ∫_{G} π(g)dμ(g),

an operator on H_{π}. As a consequence of (b), the operator π(μ) is zero on all but finitely many of the K-isotypic subspaces of H_{π}. In particular, it is of finite rank, and therefore certainly of trace class. The

* Supported in part by NSF grant DMS-9011483. character of π is the linear functional Θ_{π} ∈ H(G)^{∗} defined by

Θ_{π}(μ) = tr(π(μ)).

Every continuous function f on G defines an element of H(G)^{∗} (by sending μ to ∫_{G} fdμ). We therefore refer to elements of H(G)^{∗} as “generalized functions.” This is a slight abuse of terminology: a generalized function usually means a continuous linear functional on the larger space of all test densities on G (in which H(G) is dense). Harish-Chandra showed that characters of irreducible admissible representations are actually generalized functions in this stronger sense, but we will not make explicit use of this fact.

It is not hard to see that the map from π to Θ_{π} defines an embedding of the unitary dual Ĝ in H(G)^{∗}. The point of [Mil] is to describe the topology on Ĝ in terms of this embedding. We can give H(G)^{∗} the topology of weak convergence: a sequence Θ_{j} ∈ H(G)^{∗} converges to Θ if and only if Θ_{j}(μ) converges to Θ(μ) for every μ ∈ H(G). Now points in Ĝ are closed, and the closure of any subset of Ĝ is the set of all limits of convergent sequences in the subset. The following theorem therefore provides the desired description of the topology on Ĝ in terms of characters.

Theorem 2 ([Mil]). Suppose {π_{n}} is a sequence of irreducible unitary representations of G. Assume that the sequence of characters Θ_{π_{n}} ∈ H(G)^{∗} converges to a non-zero element Θ ∈ H(G)^{∗}. Then

a) The generalized function Θ is a finite sum of characters of irreducible unitary representations. That is, there are a non-empty finite set S ⊂ Ĝ, and positive integers {n_{σ}|σ ∈ S}, so that

Θ = ∑_{σ∈S} n_{σ}Θ_{σ}.

b) The sequence {π_{n}} is convergent in Ĝ, and S is precisely the set of all its limit points.

Conversely, suppose {π_{n}} has a limit σ_{0} ∈ Ĝ. Then there is a subsequence {π_{n_{j}}} with the property that Θ_{π_{n_{j}}} ∈ H(G)^{∗} converges to a non-zero element Θ ∈ H(G)^{∗}. In this case Θ must be a sum as in (a), and σ_{0} must belong to S.

The theorem does not directly and completely characterize convergence in Ĝ in terms of characters, but it provides enough information to describe the topology. Two examples will illustrate what is happening. Consider G = SL(2, ℝ), and let σ_{0} be the trivial representation. Let π_{n} be the complementary series representation ρ(t) with parameter t = 1 − 1/n (say for n ≥ 2). Here we think of the complementary series as parametrized by the open interval (0, 1). The characters of these representations converge to the character of a reducible limit representation ρ(1). The character of this limit representation is the sum of the character of the trivial representation and two discrete series representations D^{±}:

Θ_{ρ(1)} = Θ_{σ_{0}} + Θ_{D^{+}} + Θ_{D^{−}}.

The set S in the theorem therefore consists of three representations, and these are the limits of the sequence.

For a second example, we modify the sequence {π_{n}} by replacing half its terms by σ_{0}. This modified sequence still converges to σ_{0}; but the sequence of characters no longer converges. Rather, it has two limit points: Θ_{ρ(1)}, and the character of the trivial representation.

For a more complete discussion of Miličić’ results, we refer to section 1 of [Tad2].

Theorem 3 Suppose σ is an irreducible unitary representation of G, and that σ is not an isolated point in the unitary dual of G. Let {π_{n}} be a sequence of irreducible unitary representations distinct from σ but converging to σ. Then there are a subsequence {π_{n_{j}}}; a parabolic subgroup P = MN of G; an irreducible admissible representation ρ of M; and a sequence of one-dimensional characters {χ_{j}} of M, with the following properties.

a) The characters {χ_{j}} converge to the trivial character of M.

b) The induced representation Ind_{P}^{G}(ρ ⊗ χ_{j}) is infinitesimally equivalent to π_{n_{j}}.

c) The representation σ is a composition factor of Ind_{P}^{G}(ρ).

Under these circumstances, the characters Θ_{π_{n_{j}}} converge to the character Θ of the admissible representation π = Ind_{P}^{G}(ρ). The collection of limit points of the subsequence {π_{n_{j}}} is therefore the set of composition factors of π.

This result is almost certainly true exactly as formulated here for reductive groups over any local field. Proof. Write ℨ(𝔤) for the center of the universal enveloping algebra of the Lie algebra of G. For z ∈ ℨ(𝔤),

write σ(z) for the scalar by which z acts on the smooth vectors of σ. By [BD], the sequence π_{n}(z) converges to σ(z) in ℂ.

Write P_{m} = M_{m}A_{m}N_{m} for a Langlands decomposition of a minimal parabolic subgroup of G. By Harish-Chandra’s subquotient theorem, we can find δ_{n} ∈ M̂_{m} and ν_{n} ∈ Π(A) ≃ 𝔞_{m}^{∗} so that π_{n} is a subquotient of the principal series representation Ind_{P_{m}}^{G}(δ_{n} ⊗ ν_{n}). Now it is easy to calculate the infinitesimal characters of the principal series representations in terms of (the highest weight of) δ_{n} and ν_{n}. From the convergence of infinitesimal characters explained in the preceding paragraph, we deduce first that there are only finitely many different δ_{n}, and second that the sequence ν_{n} is bounded in 𝔞_{m}^{∗}. After passing to a subsequence, we may therefore assume that δ_{n} = δ, and that ν_{n} converges to ν_{0} ∈ 𝔞_{m}^{∗}. Because each principal series representation has only finitely many composition factors, we may also assume that ν_{n} ≠ ν_{0} for every n.

To go further, we need to understand the reducibility of the induced representations I(ν) = Ind_{P_{m}}^{G}(δ⊗ν). Write Δ_{m} = Δ(𝔤, 𝔞_{m}) ⊂ 𝔞_{m}^{∗} for the set of restricted roots of 𝔞_{m} in 𝔤. For each α ∈ Δ_{m}, let M_{α} be the reductive subgroup of G generated by M_{m}A_{m} and the root subgroups for multiples of α. We are interested in the reducibility of the principal series representation I_{α}(ν) = Ind_{P_{m}∩M_{α}}^{M_{α}}(δ ⊗ ν). Because the kernel of α in A_{m} is central in M_{α}, this reducibility occurs along a discrete set of hyperplanes parallel to the kernel of α^{∨} in 𝔞_{m}^{∗}. Define the reducibility set for α by

R(α) = {z ∈ ℂ|I_{α}(ν) is reducible whenever ⟨α^{∨}, ν⟩ = z}.

(Of course this set depends on δ.) It is a discrete set of rational numbers. We will also need

R(α)^{+} = R(α) ∪ {0}.

It follows from the Langlands classification that if there is no root α ∈ Δ_{m} with the property that ⟨α^{∨}, ν⟩ ∈ R(α)^{+}, then I(ν) is irreducible (see [SV], Theorem 3.14). We need a variant of this result.

Lemma 4. In the setting above, there is for every α ∈ Δ_{m} a discrete set R(α)^{++} of rational numbers (depending on δ), with the following property. Let P = MN be a parabolic subgroup of G with M_{m}A_{m} ⊂ M. Suppose that every root α ∈ Δ_{m} with the property that ⟨α^{∨}, ν⟩ ∈ R(α)^{++} is actually a root of 𝔞_{m} in 𝔪. Then induction from P to G carries every irreducible composition factor of Ind_{P_{m}∩M}^{M}(δ ⊗ν) to an irreducible representation of G.

This is a straightforward consequence of the arguments in [SV], particularly Theorem 1.1 and section 3. Probably it suffices to take R(α)^{++} = R(α)^{+}, but this would require a more careful analysis of intertwining operators. In any case we omit the argument.

After passing to a subsequence, we may assume that each restricted root α satisfies exactly one of the following conditions: either

⟨α^{∨}, ν_{n}⟩ ∉ R(α)^{++}

for all n, or

⟨α^{∨}, ν_{n}⟩ = r_{α} ∈ R(α)^{++}

for all n. We call such roots good and bad respectively. Let Δ_{m}(𝔪) be the set of all restricted roots in the rational span of the bad roots. These roots are the restricted root system of a Levi subgroup M of G containing M_{m}A_{m}; choose a corresponding parabolic subgroup P = MN of G. By the construction of M, we have

⟨β^{∨}, ν_{n}⟩ = ⟨β^{∨}, ν⟩ = r_{β}

for every root β of 𝔞_{m} in 𝔪. It follows that ν_{n} − ν extends from A_{m} uniquely to a one-dimensional character χ_{n} of M trivial on M_{m}. Since {ν_{n}} converges to ν, the characters χ_{n} converge to the trivial character of M.

List the composition factors of I_{M}(ν) = Ind_{P_{m}∩M}^{M}(δ ⊗ ν) as {ρ_{1}, …, ρ_{p}}. Then I(ν_{n}) has the same composition factors as Ind_{P}^{G}(I_{M}(ν) ⊗ χ_{n}). By Lemma 4, these are Ind_{P}^{G}(ρ_{i} ⊗ χ_{n}). So one of these must be π_{n}. After passing to a subsequence and relabelling the ρ_{i}, we may assume that

π_{n} = Ind_{P}^{G}(ρ_{1} ⊗ χ_{n})

for all n. It is easy to check that the characters of these representations converge to the character of Ind_{P}^{G}(ρ_{1}). Now all the assertions of Theorem 3 follow from Theorem 2. Q.E.D.

Theorem 5. Suppose P = MN is a parabolic subgroup of G, ρ is an irreducible admissible representation of M, and σ and σ′ are distinct irreducible composition factors of Ind_{P}^{G}(ρ). Then there is a sequence {σ_{0}, …, σ_{n}} of irreducible composition factors of Ind_{P}^{G}(ρ), with the following properties.

a) The first representation σ_{0} is equal to σ, and the last σ_{n} is equal to σ′.

b) For i between 1 and n, there is a non-split extension of σ_{i} by σ_{i−1}.

To understand this result, one should bear in mind that the relation on irreducible admissible representations “there exists a non-split extension of V by W ” is symmetric ([IC1], Lemma 3.18). The theorem would therefore be clear if Ind_{P}^{G}(ρ) were indecomposable, but in general it may have direct summands (for example when a unitarily induced representation is reducible). Again the result should extend without change to groups over other local fields.

This theorem is a routine consequence of a ring-theoretic result of B. J. Müller ([Mu], Theorem 7; I am grateful to J. T. Stafford for providing this reference.) Once the necessary reduction arguments have been sketched, however, it is a simple matter to include a version of Müller’s argument (kindly provided by M. Artin).

Proof. Choose a one-dimensional group of characters {χ_{ν}|ν ∈ ℂ} of M with the property that π(ν) = Ind_{P}^{G}(ρ ⊗ χ_{ν}) is irreducible for small non-zero ν. (Such a line exists because of Lemma 4.) All of these representations (or rather the underlying (𝔤, K) modules) may be realized on a single space V, with a single action of K. For any X ∈ 𝔤, the linear transformations π(ν)(X) depend in an affine way on ν: π(ν)(X) = π_{0}(X) + νπ_{1}(X). The entire family of representations may therefore be described using a single algebra homomorphism

π : U(𝔤) → End(V) ⊗ ℂ[x];

π(ν) is obtained by composition with the evaluation homomorphism

e(ν) : End(V) ⊗ ℂ[x] → End(V)

that replaces x by the complex number ν.

If M is any module for ℂ[x], then End(V) ⊗ ℂ[x] acts on

V_{M} = V ⊗_{ℂ} M

in an obvious way. Composing with π makes V_{M} into a U(𝔤)-module. If we make K act by acting trivially on M, then V_{M} becomes a (𝔤, K)-module. This defines an exact functor from ℂ[x]-modules to (𝔤, K)-modules. (In fact V_{M} is also a ℂ[x]-module, and this action commutes with the (𝔤, K)-module action.) For example, if ℂ_{ν} is the one-dimensional ℂ[x]-module on which x acts by ν, then V_{ℂ_{ν}} is isomorphic to π(ν). We are trying to prove the existence of certain extensions of (𝔤, K)-modules; they will appear as subquotients of certain V_{M}.

Suppose now that the theorem is false. Decompose the set S of irreducible composition factors of π(0) as a disjoint union of non-empty subsets S^{1} and S^{2}, in such a way that no representation σ^{1} ∈ S^{1} has a non-split extension with any representation σ^{2} in S^{2}. For each non-negative integer n, consider the (𝔤, K)-module

V_{(n)} = V ⊗ (ℂ[x]/x^{n}ℂ[x]).

This is an iterated self-extension of π(0), so it has finite length, and all its composition factors belong to S. It therefore has a canonical decomposition

V_{(n)} = V_{(n)}^{1} ⊕ V_{(n)}^{2},

with all the irreducible composition factors of V_{(n)}^{i} belonging to S^{i}. Because they are canonical, these decompositions are compatible with the ℂ[x] action and with the quotient maps V_{(n)} → V_{(m)} (for m ≤ n). Consequently they pass to the inverse limit and define a decomposition

V_{ℂ[[x]]} = V_{ℂ[[x]]}^{1} ⊕ V_{ℂ[[x]]}^{2}.

as ℂ[[x]]-modules and (𝔤, K)-modules. Finally, write Q for the quotient field of ℂ[[x]]. Tensoring with Q gives a decomposition

V_{Q} = V_{Q}^{1} ⊕ V_{Q}^{2}.

We want this decomposition to contradict the generic irreduciblity of the family π(ν).

Fix ν_{0} ∈ ℂ so that π(ν_{0}) is irreducible. Choose a finite set F of representations of K with the property that every irreducible composition factor of π(0) contains a K-type from F (as is possible since there are only finitely many composition factors), and let P (F) be the K-invariant projection of V on the F -isotypic subspace V (F). Choose a basis {v_{1}, …, v_{N}} of V (F). Suppose for simplicity that G is connected, so that U(𝔤) acts irreducibly on π(ν_{0}). (In general one can use instead an appropriate Hecke algebra built from U(𝔤) and operators from K.) By the Jacobson density theorem, we can find for each i and j an element u_{ij} ∈ U(𝔤) with the property that

π(ν_{0})(u_{ij})e_{j} = e_{i}, π(ν_{0})(u_{ij})e_{j′} = 0 (j′ ≠ j).

That is, P (F)π(ν_{0})P (F) is the matrix unit E_{ij}. Define polynomials p_{ij,kl} ∈ ℂ[x] so that

P (F)π(u_{ij})P (F) = ∑_{k,l} p_{ij,kl}E_{kl}.

The determinant of the N^{2} by N^{2} matrix (p_{ij,kl}) is a polynomial δ ∈ ℂ[x]. The value δ(ν_{0}) is 1, so δ is not the zero polynomial. Consequently δ is invertible in the quotient field Q of the formal power series ring. It follows that the Q-span of the operators P (F)π(u_{ij})P (F) is all of End(V_{Q}(F)). This contradicts the decomposition of V_{Q} obtained above, and completes the proof. Q.E.D.

One can give a parallel argument using matrix coefficients of the representations π(ν); for fixed elements of V, the matrix coefficients are holomorphic in ν, and appropriate terms in their power series expansions at zero will generate the extensions we need (under the left action of G on functions).

Theorem 6. Suppose V and V′ are distinct irreducible admissible representations of G, and that there is a non-split extension E of V′ by V:

0 → V → E → V′ → 0.

Assume that the lambda norm of (the lowest K-type of) V is less than or equal to the lambda norm of V′ ([Green], Definition 5.4.1):

∥V ∥_{lambda} ≤ ∥V′∥_{lambda}.

Let I be the standard representation containing V ([Green], Theorem 6.5.12). Then either

a) the inclusion of V in I extends to an embedding of E in I; or

b) V′ is also a Langlands subrepresentation of I.

Proof. Let μ be a lambda-lowest K-type of V. The hypothesis guarantees that μ is also a lambda-lowest K-type of the extension E. The construction in Theorem 6.5.12 of [Green] of a non-trivial map from V into I therefore applies equally well to E, and we get a non-zero map j from E to I. If j is an embedding, then (a) holds and we are done. If not, then (because the extension is not split) the kernel of j must be V. Consequently j is an embedding of V′ in I, and (b) holds. Q.E.D.

Theorem 6 does not make sense for groups over other local fields, because of the hypothesis on lambda norms (which are defined only over ℝ). It may be reformulated in terms of the Langlands classification as follows. Suppose V is the Langlands subrepresentation of an induced representation Ind_{P}^{G}(ρ), with P = MN a parabolic subgroup and ρ a tempered (modulo center) representation of M. Write A for the maximal split torus in the center of M, and X_{∗}(A) for its lattice of rational one-parameter subgroups. Write 𝔞_{0}^{∗} = Hom_{ℤ}(X_{∗}(A), ℝ) for the dual of its real Lie algebra. This real vector space carries a natural positive definite inner product (arising for example from a fixed representation of G as a matrix group). The group A acts in ρ by a complex-valued character; the absolute value of this character corresponds to an element ν(V) ∈ 𝔞_{0}^{∗}. We define

∥V ∥_{Langlands} = ∥ν(V)∥.

This definition makes perfect sense for representations of groups over local fields. In the real case, we have

∥V ∥_{lambda}^{2} + ∥V ∥_{Langlands}^{2} = ∥Re γ∥^{2},

where γ is any weight defining the infinitesimal character ([Green], proof of Lemma 6.6.6). In the setting of Theorem 6, the representations V and V′ must have the same infinitesimal character, so we deduce that

∥V ∥_{lambda}^{2} + ∥V ∥_{Langlands}^{2} = ∥V′∥_{lambda}^{2} + ∥V′∥_{Langlands}^{2}.

The hypothesis on lambda norms in the theorem is therefore equivalent to

∥V′∥_{Langlands} ≤ ∥V ∥_{Langlands}.

Formulated in this way, the result makes sense for groups over any local field, and is probably true. To prove it, one needs to control the asymptotic expansions of matrix coefficients of an extension like E in terms of those of V and V′.

Theorem 7 ([Unit], Theorems 1.2 and 1.3). Let 𝔮 = 𝔩 + 𝔲 be a θ-stable parabolic subalgebra of 𝔤. Fix a Cartan subalgebra 𝔥 ⊂ 𝔩, and a weight λ ∈ 𝔥^{∗}. Assume that

Re⟨α, λ⟩ > 0, all α ∈ Δ(𝔲, 𝔥).

Write ρ(𝔲) ∈ 𝔥^{∗} for half the sum of the roots of 𝔥 in 𝔲, and M(𝔩, L ∩ K)_{λ−ρ(𝔲)} for the category of (𝔩, L ∩ K)- modules of generalized infinitesimal character λ − ρ(𝔲). Finally, write R = R_{𝔮}^{dim 𝔲∩𝔨} for the Zuckerman cohomological induction functor from M(𝔩, L ∩ K)_{λ−ρ(𝔲)} to M(𝔤, K)_{λ}.

Then the functor R is an exact equivalence of categories onto its image, which is a full subcategory of M(𝔤, K)_{λ}. It carries irreducible representations to irreducible representations; standard representations to standard representations; unitary representations to unitary representations; and non-unitary representations to non-unitary representations. The inverse functor is H^{r}(𝔲, ·)_{λ−ρ(𝔲)}, with r = dim 𝔲 ∩ 𝔭 and the subscript indicating the direct summand of infinitesimal character λ − ρ(𝔲).

For our purposes the interest in this theorem arises from the following connection with “cohomological” unitary representations.

Theorem 8 ([VZ]). Suppose π is an irreducible unitary (𝔤, K)-module and H^{∗}(𝔤, K, π ⊗ F) ≠ 0 for some finite-dimensional (𝔤, K)-module F. Then there are a 𝔮 and λ as in Theorem 7, and a one-dimensional unitary module π^{L} in M(𝔩, L ∩ K)_{λ−ρ(𝔲)}, so that π ≃ R(π^{L}). We may choose 𝔮 so that the group L has no compact (non-abelian) simple factors; in that case 𝔮 and π^{L} are uniquely determined up to conjugation by K.

Theorem 8 shows how to construct any cohomological unitary representation from a unitary character. We are interested in the question of when such a representation is isolated in the unitary dual. It is therefore natural to begin by examining that question in the special case of unitary characters.

Theorem 9 (see [Mar], Theorem III.5.6). Suppose π is a one-dimensional unitary character of G. Assume that G has the following properties.

1) The center of G is compact.

2) The group G has no simple factors locally isomorphic to SO(n, 1)(n ≥ 2) or SU(n, 1)(n ≥ 1). Then π is isolated in the unitary dual of G.

This result is due mostly to Kazhdan and to Kostant (see [K] and [Ko], page 642). It is relatively easy to see that conditions (1) and (2) are necessary for Z to be isolated. We will give the argument in the more general context of our main result, to which we now turn.

Theorem 10. Suppose we are in the setting of Theorem 7, and that π^{L} is a one-dimensional unitary module in M(𝔩, L ∩ K)_{λ−ρ(𝔲)}. Fix a θ-stable Cartan subalgebra 𝔥 for 𝔩 as in Theorem 7, and define

Δ^{+}(𝔤, 𝔥) = {α ∈ Δ(𝔤, 𝔥)|Re⟨α, λ⟩ > 0},

a set of positive roots for 𝔥 in 𝔤. Write ρ = ρ(𝔤) for half the sum of the roots in Δ^{+}, and Π = Π(𝔤) for the simple roots for 𝔥 in 𝔤. Suppose that λ satisfies the following strengthening of the positivity hypothesis in Theorem 7:

Re⟨α, λ − ρ⟩ ≥ 0, all α ∈ Δ^{+}.

Assume that the pair (G, 𝔮) has the following properties.

0) The group L has no compact (non-abelian) simple factors.

1) The center of L is compact. (This is automatic if rk G = rk K.)

2) The group L has no simple factors locally isomorphic to SO(n, 1)(n ≥ 2) or SU(n, 1)(n ≥ 1).

3) For every noncompact imaginary root β ∈ Π that is orthogonal to the roots in Π(𝔩), we have

⟨β^{∨}, λ⟩ ≠ 1.

Then the representation π = R(π^{L}) is isolated in the unitary dual of G.

The strengthened positivity hypothesis on λ is automatic for cohomological representations (Theorem 8), when λ − ρ is the highest weight of the finite-dimensional representation F. When rk G = rk K it is a consequence of the linearity assumption on G (and the weaker positivity in Theorem 7). For the (nonlinear) double cover of SL(2, ℝ), the first discrete series representation can be written as R(π^{L}), with all the hypotheses of Theorem 10 satisfied except the strengthened positivity; but this representation is not isolated. I do not know whether there are similar examples for linear groups.

Conditions (1) to (3) are easily seen to be necessary for π to be isolated. If (1) fails, then L has unitary characters converging to the trivial character, and π^{L} may be deformed by tensor product with these; applying R gives a unitary deformation of π. If (2) fails, then π^{L} is a limit of unitary complementary series representations from the SO(n, 1) or SU(n, 1) factor of L, and again we may apply R to write π as a limit point. If (3) fails, consider the θ-stable parabolic 𝔮′ = 𝔩′ + 𝔲′ corresponding to Π(𝔩) ∪ {β}. The Levi subgroup L′ is locally isomorphic to L × SL(2, ℝ) up to center. The cohomological induction functor R factors as R′ ∘ R′′, the inner factor going from L to L′ and the outer from L′ to G. By calculation in SL(2, ℝ), π^{L′} = R′′(π^{L}) is the first discrete series of SL(2, ℝ) (tensored with a one-dimensional character on the rest of L′). Consequently π^{L′} is a limit of unitary complementary series for L′, and we can apply R′ to write π as a limit point.

We also note that condition (1) could be written as “L has no simple factors locally isomorphic to SO(1, 1),” and so subsumed in (2). As the preceding paragraph indicates, however, it is natural to distinguish (1) and (2); they correspond to slightly different local structure in the unitary dual of G.

Before beginning the proof of Theorem 10, we make a few remarks on the strategy. If R(π^{L}) is not isolated in the unitary dual, then it must be a limit point of a sequence {π_{j}} of unitary representations. The easiest possibility is that these representations are themselves constructed by Theorem 7 from unitary representations of L. In that case we will show that the unitary character π^{L} is a limit point of a sequence of unitary representations of L, and apply Theorem 9 to deduce that (1) or (2) must fail. For this we need a criterion for identifying the image of the functor R; it is provided by the theory of lowest K-types.

The difficult possibility is that the representations {π_{j}} are not themselves in the image of R. In this case the lowest K-type criterion mentioned above implies that {π_{j}} has several limit points. Theorems 3 and 5 then provide a non-split extension of π by another unitary representation τ. This situation is controlled not so much by π (and its realization as R(π^{L})) but by τ. We therefore need to realize τ using Theorem 7 (and a different θ-stable parabolic subalgebra). Unfortunately it is far from true that every unitary representation has such a realization; we must use special information about τ.

Suppose for example that π is a “cohomological” representation as in Theorem 8; in other words, that Ext_{𝔤,K}(F^{∗}, π) is non-zero for some finite-dimensional F. The existence of the non-split extension of π by τ means that Ext_{𝔤,K}^{1}(π, τ) ≠ 0. The natural “multiplicative” structure

Ext^{p}(A, B) ⊗ Ext^{q}(B, C) → Ext^{p+q}(A, C)

then suggests that Ext_{𝔤,K}(F^{∗}, τ) should be non-zero — that is, that τ should also be “cohomological.” Theorem 8 would then provide the realization of τ that we want. Unfortunately, these multiplication maps are often zero; it does not seem easy to prove directly that τ is cohomological. When rk G = rk K, one can give a short but very deep proof using the Kazhdan-Lusztig conjectures. When G is complex or G = SL(n, ℝ), Enright and Speh respectively characterized cohomological unitary representations in terms of infinitesimal character alone, so τ is cohomological in these cases as well. These three special cases cover most interesting examples; but of course we would like a general argument (covering also the possibility that π is not cohomological).

For that purpose, we imitate the proof of Theorem 8. From the non-vanishing of Ext_{𝔤,K}^{1}(π, τ) we deduce some crude information about the restriction of τ to K. (This is analogous to the fact that a cohomological representation must contain some K-types in F^{∗} ⊗⋀ 𝔭.) Applying Parthasarathy’s Dirac operator inequality as in [Kum] and [VZ], we can sharpen this information. Finally a lowest K-type criterion identifies τ as R_{𝔮′}(τ^{L′}) in the setting of Theorem 7 (with τ^{L′} a unitary character) as we wished to show.

To complete the proof of Theorem 10, we realize the extension of π by τ as R_{𝔮′} applied to a similar extension of some unitary representation π^{L′} by the unitary character τ^{L′}. It follows that π^{L′} has nonvanishing relative cohomology in degree one, and is therefore of a very special kind. (Such representations exist essentially only for SU(n, 1) and SO(n, 1).) This tells us something about π = R_{𝔮′}(π^{L′}), and ultimately contradicts condition (2) or (3) of Theorem 10.

Proof of Theorem 10. Suppose that condition (0) holds, but that π is not isolated in the unitary dual of G. We must show that one of the conditions (1), (2), or (3) must fail. Choose P = MN, ρ, and {χ_{j}} as in Theorem 3. We consider three cases: first, that Ind_{P}^{G}(ρ) is irreducible; second, that Ind_{P}^{G}(ρ) is reducible, but that π contains a lambda-lowest K-type of the induced representation; and third, that π does not contain a lambda-lowest K-type of the induced representation. We will show (roughly) that each of these cases corresponds to the failure of the corresponding hypothesis in the theorem.

Suppose first that Ind_{P}^{G}(ρ) is irreducible. At least for large j the characters χ_{j} must be trivial on M ∩K; for such j the induced representation π_{j} = Ind_{P}^{G}(ρ ⊗ χ_{j}) must satisfy

π_{j}|_{K} ≃ π|_{K}.

In particular, π_{j} and π share the same lambda-lowest K-types. In order to take advantage of this, we need a long digression involving the classification theorem in [Green], which we now recall.

Theorem 11 (see [Green], section 6.5). Suppose μ is an irreducible representation of K. Write Π(G)(μ) for the set of equivalence classes of irreducible (𝔤, K)-modules containing μ as a lambda-lowest K-type. Attached to μ is a pair (𝔮_{1}, μ^{L_{1}}) (the “classification data”) consisting of a θ-stable parabolic subalgebra 𝔮_{1} = 𝔩_{1} + 𝔲_{1} and an irreducible representation μ^{L_{1}} of L_{1} ∩ K, having the following properties. Put s_{1} = dim 𝔲_{1} ∩ 𝔨. Suppose π^{L_{1}} is an irreducible (𝔩_{1}, L_{1} ∩ K)-module containing the L_{1} ∩ K-type μ^{L_{1}}.

a) The representation μ^{L_{1}} is a lambda-lowest L_{1} ∩ K-type; so it has multiplicity one, and π^{L_{1}} belongs to Π(L_{1})(μ^{L_{1}}).

b) The representation μ has multiplicity one in R_{𝔮_{1}}^{s_{1}}(π^{L_{1}}), and is a lambda-lowest K-type. Consequently R_{𝔮_{1}}^{s_{1}}(π^{L_{1}}) has a unique irreducible composition factor J_{𝔮_{1}}(π^{L_{1}}) in Π(G)(μ).

c) The correspondence J_{𝔮_{1}} of (b) defines a bijection

Π(L_{1})(μ^{L_{1}}) → Π(G)(μ).

Suppose that 𝔮 = 𝔩 + 𝔲 is another θ-stable parabolic subalgebra containing 𝔮_{1}. Then there is a unique irreducible representation μ^{L} of L ∩ K with the property that (𝔮_{1} ∩ 𝔩, μ^{L_{1}}) is the set of classification data attached to L and μ^{L}. Put s = dim(𝔲 ∩ 𝔨), s_{0} = dim(𝔲_{1} ∩ 𝔩 ∩ 𝔨). Suppose π^{L} ∈ Π(L)(μ^{L}). d) The cohomological induction functor R_{𝔮_{1}}^{s_{1}} is naturally equivalent to the composite R_{𝔮}^{s} ∘ R_{𝔮_{1}∩𝔩}^{s_{0}}.

e) The representation μ has multiplicity one in R_{𝔮}^{s}(π^{L}), and is a lambda-lowest K-type. Consequently R_{𝔮}^{s}(π^{L}) has a unique irreducible composition factor J_{𝔮}(π^{L}) in Π(G)(μ).

f) The correspondence J_{𝔮} of (e) defines the top row in a commutative diagram

Π(L)(μ^{L})  ⟶^{J_{𝔮}}  Π(G)(μ)
J_{𝔮_{1}∩𝔩} ↖  ↗ J_{𝔮_{1}}
Π(L_{1})(μ^{L_{1}})

in which all maps are bijections. (The other two arrows arise from (b) applied to (L, μ^{L}) and to (G, μ).)

The main classification theorem in [Green] includes essentially parts (a)–(c) of this theorem. The remainder is a consequence of induction by stages (see [Green], Proposition 6.3.6; one needs also Proposition 6.3.21 to control the spectral sequence). In order to apply this result in our setting, we need to know that the functor of Theorem 7 is a special case of the correspondence in (e) of Theorem 11. This is the content of the following lemma.

Lemma 12. In the setting of Theorem 7, suppose π^{L} is an irreducible (𝔩, L∩K)-module of infinitesimal character λ − ρ(𝔲), and μ^{L} is a lambda-lowest L ∩ K-type of π^{L}. Write (𝔮_{0}, μ^{L_{1}}) for a set of classification data for μ^{L} (Theorem 11), with 𝔮_{0} = 𝔩_{1} + 𝔲_{0}. Define 𝔮_{1} = 𝔮_{0} + 𝔲, a θ-stable parabolic subalgebra of 𝔤. Then there is a lambda-lowest K-type μ of R(π^{L}) for which (𝔮_{1}, μ^{L_{1}}) is a set of classification data.

Proof. Choose a maximal torus T_{0}^{c} in L ∩ K, and write H^{c} = T^{c}A^{c} for the Cartan decomposition of its centralizer in L. This is a fundamental Cartan subgroup of L and of G. Fix also a set of positive roots for 𝔱^{c} in 𝔩 ∩ 𝔨. We may then speak of the highest weights of a representation of L ∩ K; these are characters of T^{c}. By abuse of notation, we also write μ^{L} ∈ (𝔱^{c})^{∗} for the differential of a highest weight of μ^{L}. The proof of Theorem 11 ([Green], Proposition 5.3.3) attaches to μ^{L} another weight λ_{1}^{L} ∈ (𝔱^{c})^{∗}. The parabolic subalgebra 𝔮_{0} in the classification data may be chosen to contain 𝔥^{c}, and is then defined by

Δ(𝔲_{0}, 𝔥^{c}) = {α ∈ Δ(𝔩, 𝔥^{c})|⟨α, λ_{1}^{L}⟩ > 0}

([Green], Definition 5.3.22). The construction of λ_{1}^{L} provides also a set {β_{i}} of orthogonal imaginary roots of 𝔥^{c} in 𝔩_{1}. A fundamental property of this set is that the infinitesimal character of any representation of L of lowest L ∩ K-type μ^{L} is represented by a weight of the form

(λ_{1}^{L} + ∑ ν_{i}β_{i}, ν) ∈ (𝔱^{c})^{∗} × (𝔞^{c})^{∗}

([Green], Corollary 5.4.10). After replacing the Cartan subalgebra and weight in Theorem 7 by conjugates (under Ad(𝔩)), we may therefore assume that

𝔥 = 𝔥^{c}, λ = (λ_{1}^{L} + ∑ ν_{i}β_{i}, ν) + ρ(𝔲).

Write θ_{L} for the automorphism of 𝔥^{c} defined by

θ_{L} = θ · ∏_{i} s_{β_{i}}.

This is an automorphism of order two (since the β_{i} are orthogonal and imaginary) preserving the roots of 𝔥^{c} in 𝔲 (since θ does, and the other factor belongs to the Weyl group of 𝔩). In particular, θ_{L} fixes ρ(𝔲). The action of θ_{L} on λ may now be computed explicitly. It fixes λ_{1}^{L} (since θ does, and the roots β_{i} are orthogonal to λ_{1}^{L}). It acts by −1 on ∑ ν_{i}β_{i} (since θ acts by +1 because the β_{i} are imaginary). It acts by −1 on ν (because θ does, and the β_{i} are orthogonal to ν). Therefore

θ_{L} ((λ_{1}^{L} + ∑ ν_{i}β_{i}, ν) + ρ(𝔲)) = (λ_{1}^{L} − ∑ ν_{i}β_{i}, −ν) + ρ(𝔲).

If α is any root of 𝔥^{c} in 𝔲, it follows from the positivity hypothesis in Theorem 7 that

0 < ⟨α + θ_{L}α, λ⟩
= ⟨α, λ + θ_{L}λ⟩
= 2⟨α, λ_{1}^{L} + ρ(𝔲)⟩

by the calculation of θ_{L} in the preceding paragraph. This inequality includes the main hypothesis of Lemma 6.3.23 of [Green]. The conclusion of that lemma is that there is an irreducible representation μ of K of highest weight μ^{L} + 2ρ(𝔲 ∩ 𝔭). (The ρ-shift is by the representation of L ∩ K on the top exterior power of the −1 eigenspace of θ on 𝔲.) That μ is a lambda-lowest K-type of R(π^{L}) is [Unitariz], Proposition 6.16. That (𝔮_{1}, μ^{L_{1}}) is a set of classification data for μ (more precisely, that λ_{1}^{L} + ρ(𝔲) is the weight associated to μ by Proposition 5.3.3 of [Green]) follows as in Lemma 6.5.4 of [Green] from the positivity property of λ_{1}^{L} + ρ(𝔲) established above. Q.E.D.

We return now to the proof of Theorem 10. Assume that we are in one of the first two cases, so that μ is a lowest K-type of π and of (all but finitely many of) the induced representations π_{j}. Write μ_{L} for the restriction of π^{L} to L ∩ K (which is automatically a lambda-lowest L ∩ K-type of π^{L}). Construct 𝔮_{0} and 𝔮_{1} as in Lemma 12. Theorem 11(e) now applies, and provides unique representations π_{j}^{L} in Π(L)(μ^{L}) with the property that π_{j} is the unique irreducible subquotient of R(π_{j}^{L}) containing μ. Let us parametrize representations using “regular characters” of Cartan subgroups (see for example section 6.6 of [Green]). These parameters may be separated into a “discrete” part (essentially a character of a compact part of a Cartan subgroup) and a “continuous” part (a character of the vector part). Because the representations π_{j}^{L} share the lambda-lowest L ∩ K-type μ^{L}, the parameters may be taken on the same Cartan subgroup H = T A of L, with a common discrete part. As in Theorem 3, the convergence of infinitesimal characters guarantees that the sequence of continuous parameters ν_{j} of π_{j}^{L} is bounded. After passing to a subsequence, we may therefore assume that the ν_{j} converge to some ν_{0}; write π_{0}^{L} for the corresponding representation of L.

The parameters for the representations π_{j} may be computed from those for π_{j}^{L} (cf. [Green], Proposition 8.2.15). They are all associated to the same Cartan subgroup H, with a common discrete part, and continuous part ν_{j}. The continuous part of the parameter for the limit representation π is therefore ν_{0}. (This is very plausible, but not quite trivial to prove. One way to see it is to interpret the Langlands classification in terms of global characters. Roughly speaking, the classification says that a certain term involving exp(ν_{j}) appears in the character of π_{j}. Because the character of a parabolically induced representation depends continuously on that of the inducing representation, it follows that exp(ν_{0}) appears in the character of Ind_{P}^{G}(ρ). Such a large exponential can come only from the character of a Langlands subquotient of the induced representation; so exp(ν_{0}) appears in the character of π. Now it follows that ν_{0} is the continuous part of the parameter for π. We omit the details.)

Now that we know its parameters in the classification, Proposition 8.2.15 of [Green] allows us to conclude that π corresponds under the bijection of Theorem 11(e) to the representation π_{0}^{L}. Consequently π_{0}^{L} = π^{L}. The positivity hypothesis in Theorem 7 on the infinitesimal character of π^{L} is open, and so is satisfied by all but finitely many of the π_{j}^{L}. After passage to a subsequence, we may assume it is satisfied by all of them. Now Theorem 7 guarantees that all of the representations π_{j}^{L} are unitary; so π^{L} is not isolated in the unitary dual of L. Theorem 9 therefore says that either condition (1) or condition (2) of Theorem 10 must fail, as we wished to show. (One can be a little more precise. Section 6 of [Unitariz] shows that the restriction to K of π_{j} is determined precisely by the restriction to L ∩ K of π_{j}^{L}. If π = Ind_{P}^{G}(ρ), it follows that all the π_{j}^{L} are one-dimensional, and therefore that the center of L is noncompact. If π is a proper subquotient of Ind_{P}^{G}(ρ), it follows that the π_{j}^{L} are not one-dimensional, and therefore that the trivial character of L is a limit of infinite-dimensional irreducible unitary representations. In this case L must have simple factors locally isomorphic to SU(n, 1) or SO(n, 1).)

Finally, suppose that π does not contain a lambda-lowest K-type of Ind_{P}^{G}(ρ). We may as well assume that the center of L is compact (since what we are trying to do is establish that either (1), (2), or (3) of Theorem 10 fails). Theorem 5 provides a distinct irreducible composition factor τ of Ind_{P}^{G}(ρ) with the property that there is a non-split extension E of π by τ. Let us assume that the lambda-norm of π is greater than or equal to that of τ. (The other case is similar but much easier; it leads quickly to the conclusion that condition (2) of Theorem 10 fails.) The main difficulty is establishing

Lemma 13. Suppose we are in the setting of Theorem 10; that λ satisfies the strengthened positivity hypothesis there; and that L has compact center. Assume that π = R(π^{L}) admits a non-split extension by a unitary representation τ. Then we can find 𝔮′, λ′ as in Theorem 7 and a one-dimensional unitary character τ^{L′} in M(𝔩′, L′ ∩ K)_{λ′−ρ(𝔲′)} so that τ = R′(τ^{L′}).

We also need

Lemma 14. Suppose that 𝔮, λ and 𝔮′, λ′ satisfy the hypotheses of Theorem 7, that π^{L} is a onedimensional unitary module in M(𝔩, L ∩ K)_{λ−ρ(𝔲)}, and that π^{L′} is a one-dimensional unitary module in M(𝔩′, L′ ∩ K)_{λ′−ρ(𝔲′)}. Assume that

0) The groups L and L′ have no compact (non-abelian) simple factors.

1) The representations R_{𝔮}(π^{L}) and R_{𝔮′}(π^{L′}) are equivalent.

Then the pairs (𝔮, π^{L}) and (𝔮′, π^{L′}) are conjugate by K.

We postpone the proofs of these lemmas for a moment, and continue with the argument for Theorem 10. Fix 𝔮′ and τ^{L′} as in Lemma 13. Write I^{L′} for the standard representation of L′ containing τ^{L′}. Theorem 7 says that the standard representation containing τ is I = R′(I^{L′}). By Theorem 6, the existence of the extension E implies that π is also a subquotient of I. Now Theorem 7 allows us to write π = R′(π^{L′}), with π^{L′} an irreducible unitary representation of L′. The extension E is the image of a non-split extension E^{L′} of π^{L′} by τ^{L′}. Recall that τ^{L′} is a one-dimensional character. The extension E^{L′} represents a non-trivial class in the 1-cohomology of L′ with coefficients in π^{L′} ⊗ (τ^{L′})^{∗}. Non-vanishing 1-cohomology for unitary representations is quite rare; in fact it can happen in only two ways (see [BW], Theorem V.6.1). One possibility is that L′ has noncompact center, and that π^{L′} = τ^{L′}. In this case L is equal to L′, so it fails to satisfy condition (1) of Theorem 10. The more interesting possibility is that L′ has a simple factor of type SO(n, 1) (n ≥ 2) or SU(n, 1) (n ≥ 1), and that π^{L′} is an infinite-dimensional representation differing from τ^{L′} only on this simple factor.

Applying Theorem 8 to π^{L′}, we find 𝔮′′_{0} (a θ-stable parabolic in the SO or SU factor, added to all the other simple factors of 𝔩′) and a one-dimensional unitary character π^{L′′} so that π^{L′} ≃ R_{𝔮′′_{0}}^{s_{0}}(π^{L′′}). The non-vanishing of the first cohomology means that the part of L′′ in the SO or SU factor of L′ must be SO(n − 2, 1) × SO(2) or S(U(n − 1, 1) × U(1)) respectively. Applying R′ to this provides another realization of π from a unitary character in the setting of Theorem 7, this time with Levi subgroup L′′. By Lemma 14, L is conjugate to L′′ by K. (Here the assumption that L has no compact factors is finally used.) The description just given of L′′ shows that L fails to satisfy condition (2) of Theorem 10, unless n = 2 or 3 in the SO case, or n = 1 in the SU case. If n = 3 in the SO case, then L has a simple factor of type SO(1, 1), and so has noncompact center (in violation of condition (1) of Theorem 10).

Since SO(2, 1) and SU(1, 1) are both locally isomorphic to SL(2, ℝ), we are left with the possibility that L′ has a simple factor locally isomorphic to SL(2, ℝ), and that L′′ ≃ L is obtained from L′ by replacing that factor by its compact torus. This factor of L′ corresponds to a noncompact imaginary root β ∈ Π orthogonal to the roots in Π(𝔩). Since τ^{L′} is a one-dimensional character, its infinitesimal character must take the value ±1 on a coroot for the SL(2, ℝ) factor. Therefore ⟨β^{∨}, λ⟩ = 1, in violation of condition (3) of Theorem 10. This completes the proof of the theorem.

Proof of Lemma 13. The assumption on τ means that

Ext_{𝔤,K}^{1}(R_{𝔮}^{s}(π^{L}), τ) ≠ 0.

By Lemma 3.18 of [IC1], we may interchange the arguments of Ext. The Ext group is then the limit of a spectral sequence with E^{2} term

Ext_{𝔩,L∩K}^{p}(H^{r−q}(𝔲, τ), π^{L})

([Green], Corollary 6.3.4; here r = dim 𝔲∩𝔭). Since the limit is non-zero in degree 1, there are two possibilities: either

Ext_{𝔩,L∩K}^{1}(H^{r}(𝔲, τ), π^{L}) ≠ 0,  (15)(a)

or

Hom_{𝔩,L∩K}(H^{r−1}(𝔲, τ), π^{L}) ≠ 0.  (15)(b)

(Actually we need also to know that the spectral sequence is first quadrant. This fact becomes clear in the course of the proof of Theorem 7; we will not stop here to give the argument.)

In case (a), write τ^{L} = H^{r}(𝔲, τ)_{λ−ρ(𝔲)}, a non-zero representation of L. (Recall that the subscript indicates the direct summand of the cohomology on which ℨ(𝔩) acts by the infinitesimal character λ − ρ(𝔲).) Theorem 7 says that τ is isomorphic to R_{𝔮}^{s}(τ^{L}), and therefore that τ^{L} is irreducible and unitary (since τ is). The hypothesis (15)(a) says that τ^{L} is “cohomological” (recall that π^{L} is a unitary character). By Theorem 8, we can find a θ-stable parabolic 𝔮′_{0} = 𝔩′ + 𝔲′_{0} ⊂ 𝔩 and so on, and a one-dimensional unitary character τ^{L′}, so that τ^{L} = R_{𝔮′_{0}}^{s′_{0}}(τ^{L′}). Set 𝔮′ = 𝔮′_{0} + 𝔲, a θ-stable parabolic subalgebra in 𝔤. By induction by stages, we get

τ = R_{𝔮}^{s}(R_{𝔮′_{0}}^{s′_{0}}(τ^{L′})) = R_{𝔮′}^{s′}(τ^{L′}),

as we wished to show; the positivity condition on the weight λ′ is also easy to verify. (In the application to Theorem 10, case (a) corresponds to the easy case that the lambda-norm of τ exceeds that of π.)

Suppose then that (15)(b) holds. Choose T^{c} and H^{c} = T^{c}A^{c} as in the proof of Lemma 12. We may assume that the weight λ of Theorem 7 belongs to (𝔥^{c})^{∗}. Now π^{L} is one-dimensional and has infinitesimal character λ − ρ(𝔲). It follows that the differential of π^{L} is

dπ^{L} = λ − ρ.  (16)(a)

Consequently the restriction of λ to 𝔥^{c}∩[𝔩, 𝔩] must be equal to the half sum ρ(𝔩) of Δ^{+}(𝔩, 𝔥^{c}). After replacing λ by an Ad(𝔩) conjugate, we may assume that Δ^{+}(𝔩, 𝔥^{c}) is preserved by θ. Because of the positivity hypothesis on λ in Theorem 7, the positive system Δ^{+} defined in Theorem 10 is

Δ^{+}(𝔤, 𝔥^{c}) = Δ^{+}(𝔩, 𝔥^{c}) ∪ Δ^{+}(𝔲, 𝔥^{c}).  (16)(b)

This is preserved by θ, so

ρ|_{𝔞^{c}} = ρ(𝔩)|_{𝔞^{c}} = 0.  (16)(c)

Since L is assumed to have compact center, dπ^{L} must also vanish on 𝔞^{c}, so (16)(a) gives

λ|_{𝔞^{c}} = 0.  (16)(d)

After restriction to T^{c}, these roots include a positive system

Δ^{+}(𝔨, 𝔱^{c}) = Δ^{+}(𝔩 ∩ 𝔨, 𝔱^{c}) ∪ Δ^{+}(𝔲 ∩ 𝔨, 𝔱^{c}).

We may therefore speak of highest weights of representations of K or L ∩ K. Write ρ_{c} for half the sum of the roots in Δ^{+}(𝔨, 𝔱^{c}), and ρ_{n} = ρ − ρ_{c}.

These definitions allow us to formulate Parthasarathy’s Dirac operator inequality. Here is the statement.

Lemma 17 ([P], (2.26); [BW], Lemma II.6.11; or [VZ], Lemma 4.2). Suppose Δ^{+}(𝔤, 𝔥^{c}) is a θ-stable system of positive roots for the fundamental Cartan subgroup H^{c} = T^{c}A^{c} of G; define Δ^{+}(𝔨, 𝔱^{c}), ρ_{c}, and ρ_{n} as above. Suppose τ is an irreducible unitary representation of G, and μ ∈ (𝔱^{c})^{∗} is an extremal weight of a representation of K occurring in π. Let w be an element of the Weyl group of K with the property that μ − ρ_{n} is dominant for wΔ^{+}(𝔨, 𝔱^{c}). Finally let c_{0} denote the eigenvalue of the Casimir operator of G in τ. Then

∥μ − ρ_{n} + wρ_{c}∥^{2} ≥ c_{0} + ∥ρ∥^{2}.

Generally one requires not only that μ be an extremal weight, but also that it be dominant for Δ^{(}𝔨, 𝔱^{c}). But replacing μ by the conjugate dominant weight can only decrease the left side of the inequality we want ([VZ], Lemma 4.3); so this version of the inequality follows.

The representation π^{L} contains a unique representation π^{L∩K} of L ∩ K, of highest weight dπ^{L} = λ − ρ. (Equations (16)(c) and (d) allow us to regard λ and ρ as weights in (𝔱^{c})^{∗}.) Equation (15)(b) says that this representation of L∩K must appear in H^{r−1}(𝔲, τ). A Hochschild-Serre spectral sequence ([Green], Theorem 5.2.2) then produces an irreducible representation τ^{K} of K appearing in τ, and non-negative integers x and y with the properties that π^{L∩K} occurs in

∧^{r−x}(𝔲 ∩ 𝔭) ⊗ H^{y}(𝔲 ∩ 𝔨, τ^{K}),

and that x − y = 1. We can analyze this condition as in the proof of Proposition 5.4.2 in [Green]. The conclusion is that there are

—a highest weight μ′ of τ^{K};

—x distinct weights {β_{1}, …, β_{x}} of T^{c} in 𝔲 ∩ 𝔭;

—an element σ ∈ W_{𝔨}^{1} of length y; with the property that

σ(μ′ + ρ_{c}) − ρ_{c} − 2ρ(𝔲 ∩ 𝔭) + ∑ β_{i} = λ − ρ.  (18)(a)

Here W_{𝔨}^{1} is Kostant’s cross section for the cosets of the Weyl group W_{𝔩∩𝔨} (of 𝔱^{c} in 𝔩 ∩ 𝔨) in W_{𝔨}, and the right side is just the differential of the weight π^{L∩K}. (It is at this point that the hypothesis that L has compact center is used: it guarantees that λ vanishes on 𝔞^{c}. In the general case we would have to put λ|_{𝔱^{c}} in (18)(a). In the Dirac operator inequality ((18)(d) below) it would still be λ appearing on the right, and we would no longer be able to draw strong conclusions from the inequality.) We want to apply Lemma 17 to τ and the extremal weight σμ′. For this purpose we define an element w ∈ W_{𝔨} by the requirement that

wΔ^{+}(𝔨, 𝔱^{c}) = {α ∈ Δ(𝔨, 𝔱^{c})|⟨α, σμ′ − ρ_{n}⟩ < 0} ∪ {α ∈ −Δ^{+}(𝔨, 𝔱^{c})|⟨α, σμ′ − ρ_{n}⟩ = 0}.

From (18)(a) we calculate

σμ′ − ρ_{n} + wρ_{c} = (λ − ρ) + ρ − ∑ β_{i} − 2ρ_{n}(𝔩) + (wρ_{c} − σρ_{c}).  (18)(b)

The last term in parentheses is minus the sum of the roots {α_{j}} of 𝔱^{c} in 𝔨 that are positive for σΔ^{+}(𝔨, 𝔱^{c}) but negative for wΔ^{+}(𝔨, 𝔱^{c}). It is an easy consequence of the definition of w that these roots must all belong to Δ^{+}(𝔨, 𝔱^{c}). Therefore

σμ′ − ρ_{n} + wρ_{c} = (λ − ρ) + ρ − [∑ β_{i} + 2ρ_{n}(𝔩) + ∑ α_{j}].  (18)(c)

The last term in brackets is a sum of distinct positive roots of 𝔱^{c} in 𝔤.

Now the eigenvalue of the Casimir operator for G in π (and therefore also in τ, by the assumed existence of the extension of π by τ) is

c_{0} = ⟨λ, λ⟩ − ⟨ρ, ρ⟩.

Lemma 17 and (18)(c) therefore imply that

∥(λ − ρ) + ρ − [∑ β_{i} + 2ρ_{n}(𝔩) + ∑ α_{j}]∥^{2} ≥ ∥λ∥^{2}.  (18)(d)

This inequality can be analyzed as in [VZ], proof of Lemma 4.5. One sees that the left side is actually less than or equal to the right, with equality only if there is an element w′ ∈ W (𝔤, 𝔥) with the following properties:

—w′ commutes with θ; —w′ fixes λ − ρ; and —w′ρ = ρ − [∑ β_{i} + 2ρ_{n}(𝔩) + ∑ α_{j}]. The third condition is equivalent to

{γ ∈ Δ^{+}|γ ∉ w′Δ^{+}} = {β_{i}} ∪ Δ^{+}(𝔩 ∩ 𝔭, 𝔱^{c}) ∪ {α_{j}},  (19)(a)

as well as to the two equations

w′ρ_{c} = ρ_{c} − ∑ α_{j},  w′ρ_{n} = ρ_{n} − ∑ β_{i} − 2ρ_{n}(𝔩).  (19)(b)

We can rewrite (18)(c) as

σμ′ − ρ_{n} + wρ_{c} = (λ − ρ) + w′ρ_{c} + w′ρ_{n}.  (19)(c)

The left side here is dominant and regular for wΔ^{+}(𝔨, 𝔱^{c}), and the right side for w′Δ^{+}(𝔨, 𝔱^{c}). These two positive systems therefore coincide: wρ_{c} = w′ρ_{c}. Combining this with the first equation in (19)(b) and the formula wρ_{c} = σρ_{c} − ∑ α_{j} used earlier, we find that σρ_{c} = ρ_{c}. Therefore σ = 1. Now (19)(c) can be written as

μ′ = (λ − ρ) + ρ_{n} + w′ρ_{n}.  (19)(d)

(We could also conclude for example that x = 1, so that {β_{i}} consists of a single root β_{1}. I do not see how to use this to simplify the rest of the argument, however.) This is precisely the hypothesis for Proposition 5.16 of [VZ] (which is a mild generalization of Kumaresan’s second main result in [Kum]). That result provides the θ-stable parabolic 𝔮′ required by Lemma 13. The rest of Lemma 13 follows from Proposition 6.1 in [VZ]. Q.E.D.

Proof of Lemma 14. We need to show how to recover 𝔮 and π^{L} (up to conjugation by K) from π = R_{𝔮}(π^{L}). One way to do this is using the classification of representations by characters of Cartan subgroups ([Green], Definition 6.6.1). We will first describe how to extract the Cartan subgroup and character from 𝔮 and π^{L}. Let H = T A be a maximally split Cartan subgroup of L, with L = (L ∩ K)AN_{L} a corresponding Iwasawa decomposition. Write ρ(N_{L}) ∈ 𝔞^{∗} for half the sum of the restricted roots with multiplicities. Let M_{L} be the centralizer of A in L ∩ K, and write ρ(M_{L}) ∈ 𝔱^{∗} for half the sum of a set Δ^{+}(𝔪_{L}, 𝔱) of positive roots of 𝔱 in 𝔪_{L}. Let Δ^{+}(𝔩, 𝔥) be the “Iwasawa positive system” containing Δ^{+}(𝔪_{L}, 𝔱) and compatible with N_{L}. The corresponding half sum of positive roots is

ρ(𝔩) = (ρ(M_{L}), ρ(N_{L})) ∈ 𝔱^{∗} × 𝔞^{∗}.  (20)(a)

After replacing the Cartan subalgebra of Theorem 10 by an Ad(𝔩)-conjugate, we may assume that it is the one just described. Similarly, we may assume that the positive system Δ^{+} of Theorem 10 contains the Iwasawa positive system just chosen. Therefore

ρ = (ρ(𝔲) + ρ(M_{L}), ρ(N_{L})) ∈ 𝔱^{∗} × 𝔞^{∗}.  (20)(b)

As in (16), we conclude that the differential of π^{L} is λ − ρ. As the differential of a unitary character, this satisfies

(λ − ρ)|_{𝔞} ∈ i𝔞_{0}^{∗},  (20)(c)

which has purely imaginary inner product with any root. With this notation, the representation π is attached to the Cartan subgroup H, and to a character with differential λ. (This is implicit in Theorem 11 above, and explicit in [VZ], Theorem 6.16.) In particular,

λ|_{𝔞} = (λ − ρ)|_{𝔞} + ρ(N_{L}).  (20)(d)

Since the data of the classification are determined up to conjugation by K, we must show how to reverse this calculation to recover 𝔮 and π^{L} from H and the character with differential λ. Now 𝔮 must contain the “classification parabolic” 𝔮_{1} of Theorem 11, and this is just the parabolic defined by λ|𝔱:

Δ(𝔮_{1}, 𝔥) = {α ∈ Δ(𝔤, 𝔥)|⟨α, λ|_{𝔱}⟩ ≥ 0}.  (21)(a)

([Green], Proposition 6.6.2). The classification also provides a representation π^{L_{1}} of L_{1} corresponding to π (Theorem 11). So we only have to recover the Levi subalgebra 𝔩; π^{L} will be the representation of L corresponding to π^{L_{1}} (Theorem 11 again).

We claim next that

Δ(𝔫_{L}, 𝔥) = {α ∈ Δ(𝔤, 𝔥)|Re⟨α, λ⟩ and Re⟨θα, λ⟩ have opposite signs}.  (21)(b)

To see this, notice first that the positivity hypothesis on λ in Theorem 7 guarantees that the roots on the right must all be roots in 𝔩. Such a root is orthogonal to λ − ρ and to ρ(𝔲); so the set on the right is

{α ∈ Δ(𝔩, 𝔥)|⟨α, ρ(𝔩)⟩ and ⟨α, θρ(𝔩)⟩ have opposite signs}.

This in turn is equal to Δ^{+}(𝔩) ∩ (−θΔ^{+}(𝔩), which is obviously Δ(𝔫_{L}, 𝔥).

We have shown how to recover the roots Δ(𝔫_{L}, 𝔥) from the classification data. Because 𝔩 has no compact simple factors, its root system is spanned by Δ(𝔫_{L}, 𝔥). That is,

Δ(𝔩, 𝔥) = ℤΔ(𝔫_{L}, 𝔥) ∩ Δ(𝔤, 𝔥).  (21)(c)

As we remarked earlier, 𝔮 is generated by the classification parabolic 𝔮_{1} and 𝔩:

Δ(𝔮, 𝔥) = Δ(𝔮_{1}, 𝔥) ∪ Δ(𝔩, 𝔥).  (21)(d)

This completes the proof of Lemma 14. Q.E.D.

References.

[BD] P. Bernat and J. Dixmier, “Sur le dual d’un groupe de Lie,” C. R. Acad. Sci. Paris 250 (1960), 1778–1779.

[BW] A. Borel and N. Wallach, Continuous Cohomology, Discrete Subgroups, and Representations of Reductive Groups. Princeton University Press, Princeton, New Jersey, 1980.

[K] D. Kazhdan, “Connection of the dual space of a group with the structure of its closed subgroups,” Funct. Anal. Appl. 1 (1967), 63–65.

[Ko] B. Kostant, “On the existence and irreducibility of certain series of representations,” Bull. Amer. Math. Soc. 75 (1969), 627–642.

[Kum] S. Kumaresan, “On the canonical 𝔨-types in the irreducible unitary 𝔤-modules with non-zero relative cohomology,” Invent. Math. 59 (1980), 1–11.

[Mar] G. Margulis, Discrete Subgroups of Semisimple Lie Groups. Springer-Verlag, Berlin-Heidelberg-New York, 1991.

[Mil] D. Miličić, “On C^{∗}-algebras with bounded trace,” Glasnik Mat. 8 (28) (1973), 7–22.

[Mil2] D. Miličić, “On structure of dual of semisimple groups,” Ph.D. dissertation, University of Zagreb, 1973.

[Mil3] D. Miličić, “The dual spaces of almost connected reductive groups,” Glasnik Mat. 9 (29) (1974), 273–288.

[Mu] B. J. Müller, “Localization in non-commutative Noetherian rings,” Canad. J. Math. 28 (1976), 600–610.

[P] R. Parthasarathy, “Criteria for the unitarizability of some highest weight modules,” Proc. Indian Acad. Sci. 89 (1980), 1–24.

[SV] B. Speh and D. Vogan, “Reducibility of generalized principal series representatons,” Acta Math. 145 (1980), 227–299.

[Tad1] M. Tadić, “The topology of the dual space of a reductive group over a local field,” Glasnik Mat. 18 (38) (1983), 259–279.

[Tad2] M. Tadić, “Topology of unitary dual of nonarchimedean GL(n),” Duke Math. J. 55 (1987), 385–422.

[Tad3] M. Tadić, “Geometry of dual spaces of reductive groups (non-archimedean case),” J. Analyse Math. 51 (1988), 139–181.

[IC1] D. Vogan, “Irreducible characters of semisimple Lie groups I,” Duke Math. J. 46 (1979), 61–108.

[Green] D. Vogan, Representations of Real Reductive Lie Groups. Birkhäuser, Boston-Basel-Stuttgart, 1981.

[Unit] D. Vogan, “Unitarizability of certain series of representations,” Ann. of Math. 120 (1984), 141–187.

[VZ] D. Vogan and G. Zuckerman, “Unitary representations with non-zero cohomology,” Compositio Math. 53 (1984), 51–90.

[Wal] N. Wallach, Real Reductive Groups II. Academic Press, San Diego, 1992.