Unitary Representations and Complex Analysis

David A. Vogan, Jr ^{⋆}

Department of Mathematics, Massachusetts Institute of Technology Cambridge, Massachusetts 02139, U.S.A.

dav@math.mit.edu

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 Compact groups and the Borel-Weil theorem . . . . . . . . . . . . . 6 3 Examples for SL(2, ℝ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 4 Harish-Chandra modules and globalization . . . . . . . . . . . . . . . . 16 5 Real parabolic induction and the globalization functors … 26 6 Examples of complex homogeneous spaces . . . . . . . . . . . . . . . . 36 7 Dolbeault cohomology and maximal globalizations . . . . . . . . 44 8 Compact supports and minimal globalizations . . . . . . . . . . . . . 60 9 Invariant bilinear forms and maps between representations 69 10 Open questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

1 Introduction

Much of what I will say depends on analogies between representation theory and linear algebra, so let me begin by recalling some ideas from linear algebra. One goal of linear algebra is to understand abstractly all possible linear transformations T of a vector space V. The simplest example of a linear transformation is multiplication by a scalar on a one-dimensional space. Spectral theory seeks to build more general transformations from this example. In the case of infinite-dimensional vector spaces, it is useful and interesting to introduce a topology on V, and to require that T be continuous. It often happens

⋆ Supported in part by NSF grant DMS-9721441 Mathematics Subject Classification: Primary 22E46

(as in the case when T is a differential operator acting on a space of functions) that there are many possible choices of V, and that choosing the right one for a particular problem can be subtle and important.

One goal of representation theory is to understand abstractly all the possible ways that a group G can act by linear transformations on a vector space V. Exactly what this means depends on the context. For topological groups (like Lie groups), one is typically interested in continuous actions on topological vector spaces. Using ideas from the spectral theory of linear operators, it is sometimes possible (at least in nice cases) to build such representations from irreducible representations, which play the role of scalar operators on one-dimensional spaces in linear algebra. Here is a definition.

Definition 1.1. Suppose G is a topological group. A representation of G is a pair (π, V) with V a complete locally convex topological vector space, and π a homomorphism from G to the group of invertible linear transformations of V. We assume that the map

G × V → V, (g, v) ↦ π(g)v

is continuous.

An invariant subspace for π is a closed subspace W ⊂ V with the property that π(g)W ⊂ W for all g ∈ G. The representation is said to be irreducible if there are exactly two invariant subspaces (namely V and 0).

The flexibility in this definition—the fact that one does not require V to be a Hilbert space, or the operators π(g) to be unitary—is a very powerful technical tool, even if one is ultimately interested only in unitary representations. Here is one reason. There are several important classes of groups (including reductive Lie groups) for which the classification of irreducible unitary representations is still an open problem. One way to approach the problem (originating in the work of Harish-Chandra, and made precise by Knapp and Zuckerman in [KZ77]) is to work with a larger class of “admissible” irreducible representations, for which a classification is available. The problem is then to identify the (unknown) unitary representations among the (known) admissible representations. Here is a formal statement.

Problem 1.2. Given an irreducible representation (π, V), is it possible to impose on V a Hilbert space structure making π a unitary representation? Roughly speaking, this question ought to have two parts.

(1.2)(A)  Does V carry a G-invariant Hermitian bilinear form ⟨, ⟩_{π}?

Assuming that such a form exists, the second part is this.

(1.2)(B)  Is the form ⟨, ⟩_{π} positive definite?

The goal of these notes is to look at some difficulties that arise when one tries to make this program precise, and to consider a possible path around them. The difficulties have their origin exactly in the flexibility of Definition 1.1. Typically we want to realize a representation of G on a space of functions. If G acts on a set X, then G acts on functions on X, by

[π(g)f](x) = f(g^{−1} · x).

The difficulty arises when we try to decide exactly which space of functions on X to consider. If G is a Lie group acting smoothly on a manifold X, then one can consider

C(X) = continuous functions on X,

C_{c}(X) = continuous functions with compact support,

C_{c}^{∞}(X) = compactly supported smooth functions.

C^{−∞}(X) = distributions on X.

If there is a reasonable measure on X, then one gets various Banach spaces like L^{p}(X) (for 1 ≤ p < ∞), and Sobolev spaces. Often one can impose various other kinds of growth conditions at infinity. All of these constructions give topological vector spaces of functions on X, and many of these spaces carry continuous representations of G. These representations will not be “equivalent” in any simple sense (involving isomorphisms of topological vector spaces); but to have a chance of getting a reasonable classification theorem for representations, one needs to identify them.

When G is a reductive Lie group, Harish-Chandra found a notion of “infinitesimal equivalence” that addresses these issues perfectly. Inside every irreducible representation V is a natural dense subspace V_{K}, carrying an irreducible representation of the Lie algebra of G. (Actually one needs for this an additional mild assumption on V, called “admissibility.”) Infinitesimal equivalence of V and W means algebraic equivalence of V_{K} and W_{K} as Lie algebra representations. (Some details appear in section 4.)

Definition 1.3. Suppose G is a reductive Lie group. The admissible dual of G is the set Ĝ of infinitesimal equivalence classes of irreducible admissible representations of G. The unitary dual of G is the set Ĝ_{u} of unitary equivalence classes of irreducible unitary representations of G.

Harish-Chandra proved that each infinitesimal equivalence class of admissible irreducible representations contains at most one unitary equivalence class of irreducible unitary representations. That is,

(1.4)  Ĝ_{u} ⊂ Ĝ.

This sounds like great news for the program described in Problem 1.2. Even better, he showed that the representation (π, V) is infinitesimally unitary if and only if the Lie algebra representation V_{K} admits a positive-definite invariant Hermitian form ⟨, ⟩_{π,K}.

The difficulty is this. Existence of a continuous G-invariant Hermitian form ⟨, ⟩_{π} on V implies the existence of ⟨, ⟩_{π,K} on V_{K}; but the converse is not true. Since V_{K} is dense in V, there is at most one continuous extension of ⟨, ⟩_{π,K} to V, but the extension may not exist. In section 3, we will look at some examples, in order to understand why this is so. What the examples suggest, and what we will see in section 4, is that the Hermitian form can be defined only on appropriately “small” representations in each infinitesimal equivalence class. In the example of the various function spaces on X, compactly supported smooth functions are appropriately small, and will often carry an invariant Hermitian form. Distributions, on the other hand, are generally too large a space to admit an invariant Hermitian form.

Here is a precise statement. (We will write V^{∗} for the space of continuous linear functionals on V, endowed with the strong topology (see section 8).)

Theorem 1.5. (Casselman, Wallach, and Schmid; see [Cas89], [Sch85], and section 4). Suppose (π, V) is an admissible irreducible representation of a reductive Lie group G on a reflexive Banach space V. Define

(π^{ω}, V^{ω}) = analytic vectors in V ,
(π^{∞}, V^{∞}) = smooth vectors in V ,
(π^{−∞}, V^{−∞}) = distribution vectors in V = dual of (V′)^{∞}, and
(π^{−ω}, V^{−ω}) = hyperfunction vectors in V = dual of (V′)^{ω}.

Each of these four representations is a smooth representation of G in the infinitesimal equivalence class of π, and each depends only on that equivalence class. The inclusions

V^{ω} ⊂ V^{∞} ⊂ V ⊂ V^{−∞} ⊂ V^{−ω}

are continuous, with dense image.

Any invariant Hermitian form ⟨, ⟩_{K} on V_{K} extends uniquely to continuous G-invariant Hermitian forms ⟨, ⟩_{ω} and ⟨, ⟩_{∞} on V^{ω} and V^{∞}.

The assertions about Hermitian forms will be proven in Theorem 9.16. The four representations appearing in Theorem 1.5 are called the minimal

globalization, the smooth globalization, the distribution globalization, and the maximal globalization respectively. Unless π is finite-dimensional (so that all of the spaces in the theorem are the same) the Hermitian form will not extend continuously to the distribution or maximal globalizations V^{−∞} and V^{−ω}.

We will be concerned here mostly with representations of G constructed using complex analysis, on spaces of holomorphic sections of vector bundles and generalizations. In order to use these constructions to get unitary representations, we need to do the analysis in such a way as to get the minimal or smooth globalizations; this will ensure that the Hermitian forms we seek will be defined on the representations. A theorem of Hon-Wai Wong (see [Won99] or Theorem 7.21 below) says that Dolbeault cohomology leads to the maximal globalizations in great generality. This means that there is no possibility of finding invariant Hermitian forms on these Dolbeault cohomology representations except in the finite-dimensional case.

We therefore need a way to modify the Dolbeault cohomology construction to produce minimal globalizations rather than maximal ones. Essentially we will follow ideas of Serre from [Ser55], arriving at realization of minimal globalization representations first obtained by Tim Bratten in [Bra97]. Because of the duality used to define the maximal globalization, the question amounts to this: how can one identify the topological dual space of a Dolbeault cohomology space on a (noncompact) complex manifold? The question is interesting in the simplest case. Suppose X ⊂ ℂ is an open set, and H(X) is the space of holomorphic functions on X. Make H(X) into a topological vector space, using the topology of uniform convergence of all derivatives on compact sets. What is the dual space H(X)′?

This last question has a simple answer. Write C_{c}^{−∞}(X, densities) for the space of compactly supported distributions on X. We can think of this as the space of compactly supported complex 2-forms (or (1, 1)-forms) on X, with generalized function coefficients. (A brief review of these ideas will appear in section 8). More generally, write

A_{c}^{(p,q),−∞}(X) =compactly supported (p, q)-forms

on X with generalized function coefficients.

The Dolbeault differential \overline{∂} maps (p, q)-forms to (p, q+1) forms and preserves support; so

\overline{∂}: A_{c}^{(1,0),−∞}(X) → A_{c}^{(1,1),−∞}(X) = C_{c}^{−∞}(X, densities).

Then (see [Ser55], Théorème 3)

(1.6)  H(X)′ ≃ A_{c}^{(1,1),−∞}(X)/\overline{\overline{∂}A_{c}^{1,0}(X)}.

Here the overline denotes closure. For X open in ℂ the image of \overline{∂} is automatically closed, so the overline is not needed; but this formulation has an immediate extension to any complex manifold X (replacing 1 and 0 by the dimension n and n − 1). Here is Serre’s generalization.

Theorem 1.7. (Serre; see [Ser55], Théorème 2 or Theorem 8.13 below). Suppose X is a complex manifold of dimension n, V is a holomorphic vector bundle on X, and Ω is the canonical line bundle (of (n, 0)-forms on X). Define

A^{0,p}(X, V) = smooth V-valued (0, p)-forms on X

A_{c}^{(0,p),−∞}(X, V) = compactly supported V-valued (0, p)-forms

with generalized function coefficients.

Define the topological Dolbeault cohomology of X with values in V as

H_{top}^{0,p}(X, V) = [kernel of \overline{∂} on A^{0,p}(X, V)]/\overline{\overline{∂}A^{p−1,0}(X, V)};

this is a quotient of the usual Dolbeault cohomology. It carries a natural locally convex topology. Similarly, define

H_{c,top}^{0,p}(X, V) = [kernel of \overline{∂} on A_{c}^{(0,p),−∞}(X, V)]/\overline{\overline{∂}A_{c}^{(p−1,0),−∞}(X, V)},

the topological Dolbeault cohomology with compact supports. Then there is a natural identification

H_{top}^{0,p}(X, L)^{∗} ≃ H_{c,top}^{0,n−p}(X, Ω ⊗ L^{∗}).

Here L^{∗} is the dual holomorphic vector bundle to L.

When X is compact, then the subscript c adds nothing, and the \overline{∂} operators automatically have closed range. One gets in that case the most familiar version of Serre duality.

In Corollary 8.14 we will describe how to use this theorem to obtain Bratten’s result, constructing minimal globalization representations on Dolbeault cohomology with compact supports.

Our original goal was to understand invariant bilinear forms on minimal globalization representations. Once the minimal globalizations have been identified geometrically, we can at least offer a language for discussing this problem using standard functional analysis. This is the subject of section 9.

There are around the world a number of people who understand analysis better than I do. As an algebraist, I cannot hope to estimate this number. Nevertheless I am very grateful to several of them (including Henryk Hecht, Sigurdur Helgason, David Jerison, and Les Saper) who helped me patiently with very elementary questions. I am especially grateful to Tim Bratten, for whose work these notes are intended to be an advertisement. For the errors that remain, I apologize to these friends and to the reader.

2 Compact groups and the Borel-Weil theorem

The goal of these notes is to describe a geometric framework for some basic questions in representation theory for noncompact reductive Lie groups. In order to explain what that might mean, I will recall in this section the simplest example: the Borel-Weil theorem describing irreducible representations of a compact group. Throughout this section, therefore, we fix a compact connected Lie group K, and a maximal torus T ⊂ K. (We will describe an example in a moment.) We fix also a K-invariant complex structure on the homogeneous space K/T. In terms of the structure theory of Lie algebras, this amounts to a choice of positive roots for the Cartan subalgebra 𝔱 = Lie(T)_{ℂ} inside the complex reductive Lie algebra 𝔨 = Lie(K)_{ℂ}. For more complete expositions of the material in this section, we refer to [Kna86], section V.7, or [Hel94], section VI.4.3, or [Vog87], chapter 1.

Define

(2.1)(a)  T̂ = lattice of characters of T;

these are the irreducible representations of T. Each μ ∈ T̂ may be regarded as a homomorphism of T into the unit circle, or as a representation (μ, ℂ_{μ}) of T. Such a representation gives rise to a K-equivariant holomorphic line bundle

(2.1)(b)  L_{μ} → K/T.

Elements of T̂ are often called weights.

I do not want to recall the structure theory for K in detail, and most of what I say will make some sense without the details. With that warning not to pay attention, fix a simple root α of T in K, and construct a corresponding three-dimensional subgroup

(2.2)(a)  ϕ_{α}: SU(2) → K,  K_{α} = ϕ_{α}(SU(2)),  T_{α} = K_{α} ∩ T.

Then K_{α}/T_{α} is the Riemann sphere ℂℙ^{1}, and we have a natural holomorphic embedding

(2.2)(b)  ℂℙ^{1} ≃ K_{α}/T_{α} ↪ K/T.

The weight μ ∈ T̂ is called antidominant if for every simple root α,

(2.2)(c))  L_{μ}|_{K_{α}/T_{α}} has non-zero holomorphic sections.

This is a condition on ℂℙ^{1}, about which we know a great deal. The sheaf of germs of holomorphic sections of L_{μ}|_{K_{α}/T_{α}} is O(−⟨μ, α^{∨}⟩); here α^{∨} is the coroot for the simple root α, and ⟨μ, α^{∨}⟩ is an integer. The sheaf O(n) on ℂℙ^{1} has non-zero sections if and only if n ≥ 0. It follows that μ is antidominant if and only if for every simple root α,

(2.2)(d))  ⟨μ, α^{∨}⟩ ≤ 0.

Theorem 2.3. (Borel-Weil, Harish-Chandra; see [HC56], [Ser59]). Suppose K is a compact connected Lie group with maximal torus T; use the notation of (2.1) and (2.2) above.

(1) Every K-equivariant holomorphic line bundle on K/T is equivalent to L_{μ}, for a unique weight μ ∈ T̂.

(2) The line bundle L_{μ} has non-zero holomorphic sections if and only if μ is antidominant.

(3) If μ is antidominant, then the space Γ (L_{μ}) of holomorphic sections is an irreducible representation of K.

(4) This correspondence defines a bijection from antidominant characters of T onto K̂.

As the references indicate, I believe that this theorem is due independently to Harish-Chandra and to Borel and Weil. Nevertheless I will follow standard practice and refer to it as the Borel-Weil theorem.

Before saying anything about a proof, we look at an example. Set

(2.4)(a)  K = U(n) = n × n complex unitary matrices
= {u = (u_{1}, …, u_{n}) | u_{i} ∈ ℂ^{n}, ⟨u_{i}, u_{j}⟩ = δ_{i,j}} .

Here we regard ℂ^{n} as consisting of column vectors, so that the u_{i} are the columns of the matrix u; δ_{i,j} is the Kronecker delta. This identifies U(n) with the set of orthonormal bases of ℂ^{n}. As a maximal torus, we choose

(2.4)(b)  T = U(1)^{n} = diagonal unitary matrices

= \{(\begin{matrix}e^{iϕ_{1}} \\ & . & . & . \\ & & & & e^{iϕ_{n}} \end{matrix} ) | ϕ_{j} ∈ ℝ\} .

As a basis for the lattice of characters of T, we can choose

(2.4)(c)  χ_{j} (\begin{matrix}e^{iϕ_{1}} \\ & . & . & . \\ & & & & e^{iϕ_{n}} \end{matrix} ) = e^{iϕ_{j}},

the action of T on the jth coordinate of ℂ^{n}.

We want to understand the homogeneous space K/T = U(n)/U(1)^{n}. Recall that a complete flag in ℂ^{n} is a collection of linear subspaces

(2.5)(a)  F = (0 = F_{0} ⊂ F_{1} ⊂ ⋯ ⊂ F_{n} = ℂ^{n}),  dim F_{j} = j.

The collection of all such complete flags is a complex projective algebraic variety

(2.5)(b)  X = complete flags in ℂ^{n},

of complex dimension n(n − 1)/2. (When we need to be more precise, we may write X^{GL(n)}.) We claim that

(2.5)(c)  U(n)/U(1)^{n} ≃ X.

The map from left to right is

(2.5)(d)  (u_{1}, …, u_{n})U(1)^{n} ↦ F = (F_{j}),  F_{j} = span(u_{1}, …, u_{j}).

Right multiplication by a diagonal matrix replaces each column of u by a scalar multiple of itself; so the spans in this definition are unchanged, and the map is well-defined on cosets. For the map in the opposite direction, we choose an orthonormal basis u_{1} of the one-dimensional space F_{1}; extend it by Gram-Schmidt to an orthonormal basis (u_{1}, u_{2}) of F_{2}; and so on. Each u_{j} is determined uniquely up to multiplication by a scalar e^{iϕ_{j}}, so the coset (u_{1}, …, u_{n})U(1)^{n} is determined by F.

It is often useful to notice that the full general linear group K_{ℂ} = GL(n, ℂ) (the complexification of U(n)) acts holomorphically on X. For this action the isotropy group at the base point is the Borel subgroup B_{ℂ} of upper triangular matrices:

X = K_{ℂ}/B_{ℂ}.

The fact that X is also homogeneous for the subgroup K corresponds to the group-theoretic facts

K_{ℂ} = KB_{ℂ}, K ∩ B_{ℂ} = T.

Now the definition of X provides a number of natural line bundles on X. For 1 ≤ j ≤ n, there is a line bundle L_{j} whose fiber at the flag F is the one-dimensional space F_{j}/F_{j−1}:

(2.6)(a)  L_{j}(F) = F_{j}/F_{j−1}  (F ∈ X).

This is a U(n)-equivariant (in fact K_{ℂ}-equivariant) holomorphic line bundle. For any μ = (m_{1}, …, m_{n}) ∈ ℤ^{n}, we get a line bundle

(2.6)(b)  L_{μ} = L_{1}^{m_{1}} ⊗ ⋯ ⊗ L_{n}^{m_{n}}

For example, if p ≤ q, and F is any flag, then F_{q}/F_{p} is a vector space of dimension q − p. These vector spaces form a holomorphic vector bundle V_{q,p}. Its top exterior power ⋀^{q−p} V_{q,p} is therefore a line bundle on X. Writing

(2.6)(c)  μ_{q,p} = (\underbrace{0, …, 0}_{p terms}, \underbrace{1, …, 1}_{q − p terms}, \underbrace{0, …, 0}_{n − q terms}),

we find

(2.6)(d)  L_{μ_{q,p}} ≃ ⋀^{q−p} V_{q,p}.

One reason for making all these explicit examples is that it shows how some of these bundles can have holomorphic sections. The easiest example is L_{n}, whose fiber at F is the quotient space ℂ^{n}/F_{n−1}. Any element v ∈ ℂ^{n} defines a section σ_{v} of 𝕃_{n}, by the formula

(2.6)(e)  σ_{v}(F) = v + F_{n−1} ∈ ℂ^{n}/F_{n−1} = L_{n}(F).

Notice that this works only for L_{n}, and not for the other L_{j}. In a similar way, taking q = n in (2.6)(c), we find that any element ω ∈ ⋀^{n−p} ℂ^{n} defines a section σ_{ω} of L_{μ_{q,p}}, by

σ_{ω}(F) = \overline{ω} ∈ ⋀^{n−p}(ℂ^{n}/F_{p}).

By multiplying such sections together, we can find non-zero holomorphic sections of any of the bundles L_{μ}, as long as

(2.6)(e)  0 ≤ m_{1} ≤ ⋯ ≤ m_{n}.

In case m_{1} = ⋯ = m_{n} = m, the sections we get are related to the function det^{m} on K (or K_{ℂ}). Since that function vanishes nowhere on the group, its inverse provides holomorphic sections of the bundle corresponding to (−m, ⋯, −m). Multiplying by these, we finally have non-zero sections of L_{μ} whenever

(2.6)(f)  m_{1} ≤ ⋯ ≤ m_{n}.

Here is what the Borel-Weil theorem says for U(n).

Theorem 2.7. (Borel-Weil, Harish-Chandra). Use the notation of (2.4)– (2.6).

(1) Every U(n)-equivariant holomorphic line bundle on the complete flag manifold X is equivalent to L_{μ} = L_{1}^{m_{1}} ⊗ ⋯ ⊗ L_{n}^{m_{n}}, for a unique

μ = (m_{1}, …, m_{n}) ∈ ℤ^{n}.

(2) The line bundle L_{μ} has non-zero holomorphic sections if and only if μ is antidominant, meaning that

m_{1} ≤ ⋯ ≤ m_{n}.

(3) If μ is antidominant, then the space Γ (L_{μ}) of holomorphic sections is an irreducible representation of U(n).

(4) This correspondence defines a bijection from increasing sequences of integers onto { \atop U(n)}.

Here are some remarks about proofs for Theorems 2.3 and 2.7. Part (1) is very easy: making anything G-equivariant on a homogeneous space G/H is the same as making something H-equivariant. (Getting precise theorems of this form is simply a matter of appropriately specifying “anything” and “something,” then following your nose.)

For part (2), “only if” is easy to prove using reduction to ℂℙ^{1}: if μ fails to be antidominant, then there will not even be sections on some of those projective lines. The “if” part is more subtle. We proved it for U(n) in (2.6)(f), essentially by making use of a large supply of known representations of U(n) (the exterior powers of the standard representation, the powers of the determinant character, and tensor products of these). For general K, one can do something similar: once one knows the existence of a representation of lowest weight μ, it is a simple matter to use matrix coefficients of that representation to construct holomorphic sections of L_{μ}. This is what Harish-Chandra did. I cannot tell from the account in [Ser59] exactly what argument Borel and Weil had in mind. In any case it is certainly possible to construct holomorphic sections of L_{μ} (for antidominant μ) directly, using the Bruhat decomposition of K/T. It is easy to write a holomorphic section on the open cell (for any μ); then one can use the antidominance condition to prove that this section extends to all of K/T.

Part (3) and the injectivity in part (4) are both assertions about the space of intertwining operators

Hom_{K}(Γ (L_{μ_{1}}), Γ (L_{μ_{2}})).

We will look at such spaces in more generality in section 9 (Corollary 9.13).

Finally, the surjectivity in part (4) follows from the existence of lowest weights for arbitrary irreducible representations. This existence is a fairly easy part of algebraic representation theory. I do not know of a purely complex analysis proof.

Before we abandon compact groups entirely, here are a few comments about how to generalize the linear algebra in (2.4)–(2.6). A classical compact group is (in the narrowest possible definition) one of the groups U(n), O(n) (of real orthogonal matrices), or Sp(2n) (of complex unitary matrices also preserving a standard symplectic form on ℂ^{2n}). For each of these groups, there is a parallel description of K/T as a projective variety of certain complete flags in a complex vector space. One must impose on the flags certain additional conditions involving the bilinear form that defines the group. Here are some details.

Suppose first that K = O(n), the group of linear transformations of ℝ^{n} preserving the standard symmetric bilinear form

B(x, y) = ∑_{j=1}^{n} x_{j}y_{j} (x, y ∈ ℝ^{n}).

This form extends holomorphically to ℂ^{n}, where it defines the group

(2.8)(a)  K_{ℂ} = O(n, ℂ).

If W ⊂ ℂ^{n} is a p-dimensional subspace, then

W^{⊥} = {y ∈ ℂ^{n} | B(x, y) = 0, all x ∈ W }

is a subspace of dimension n − p. We now define the complete flag variety for O(n) to be (2.8)(b)

X = X^{O(n)} = {F = (F_{j}) complete flags | F_{p}^{⊥} = F_{n−p}, 0 ≤ p ≤ n}.

Notice that this definition forces the subspaces F_{q} with 2q ≤ n to satisfy F_{q}^{⊥} ⊂ F_{q}; that is, the bilinear form must vanish on these F_{q}. Such a subspace is called isotropic. Knowledge of the isotropic subspaces F_{q} (for q ≤ n/2) determines the remaining subspaces, by the requirement F_{n−q} = F_{q}^{⊥}. We get an identification

(2.8)(c) X^{O(n)} ≃ chains of isotropic subspaces (F_{q}) = (F_{0} ⊂ F_{1} ⊂ ⋯),
with dim F_{q} = q for all q ≤ n/2.

The orthogonal group is

(2.8)(d)  O(n) = {v = (v_{1}, …, v_{n})|v_{p} ∈ ℝ^{n}, B(v_{p}, v_{q}) = δ_{p,q}},

the set of orthonormal bases of ℝ^{n}. As a maximal torus T in K, we can take SO(2)^{[n/2]}, embedded in an obvious way. We claim that

(2.8)(e)  O(n)/SO(2)^{[n/2]} ≃ X^{O(n)}.

The map from left to right is (2.8)(f) (v_{1}, …, v_{n})SO(2)^{n} ↦ F = (F_{p}), F_{p} = span(v_{1} + iv_{2}, …, v_{2p−1} + iv_{2p}).

We leave to the reader the verification that this is a well-defined bijection, and an extension of the ideas in (2.6) to this setting. (Notice only that O(n) is not connected, that correspondingly X has two connected components, and that the irreducibility assertion in Theorem 2.3(3) can fail.) The space X is also homogeneous for the (disconnected) reductive algebraic group K_{ℂ} = O(n, ℂ), the isotropy group being a Borel subgroup of the identity component.

Finally, consider the standard symplectic form on ℂ^{2n},

ω(x, y) = ∑_{p=1}^{n} x_{p}y_{n+p} − x_{n+p}y_{p}.

The group of linear transformations preserving this form is

(2.9)(a)  K_{ℂ} = Sp(2n, ℂ);

the corresponding compact group may be taken to be

K = K_{ℂ} ∩ U(2n).

(Often it is easier to think of K as a group of n × n matrices with entries in the quaternions. This point of view complicates slightly the picture of K_{ℂ}, and so I will not adopt it.) Just as for the symmetric form B, we can define

W^{⊥} = {y ∈ ℂ^{2n} | ω(x, y) = 0, all x ∈ W };

if W has dimension p, then W^{⊥} has dimension 2n − p. The complete flag variety for Sp(n) is (2.9)(b)

X = X^{Sp(2n)} = {F = (F_{j}) complete flags | F_{p}^{⊥} = F_{2n−p}, 0 ≤ p ≤ 2n}.

Again the definition forces F_{q} to be isotropic for q ≤ n, and we can identify

(2.9)(c)  X ≃ chains of isotropic subspaces (F_{q}) = (F_{0} ⊂ F_{1} ⊂ ⋯),
with dim F_{q} = q for all q ≤ n.

The complex symplectic group K_{ℂ} = Sp(2n, ℂ) acts holomorphically on the projective variety X.

The complex symplectic group is (2.9)(d)

Sp(2n, ℂ) = {v = (v_{1}, …, v_{2n}) | v_{p} ∈ ℂ^{2n}, ω(v_{p}, v_{q}) = δ_{p,q−n} (p ≤ q)},

This identifies K_{ℂ} with the collection of standard symplectic bases for ℂ^{2n}. The compact symplectic group is identified with standard symplectic bases that are also orthonormal for the standard Hermitian form ⟨, ⟩ on ℂ^{2n}:

(2.9)(e) Sp(2n) = {(v_{1}, …, v_{2n}) | v_{p} ∈ ℂ^{2n}, ω(v_{p}, v_{q}) = δ_{p,q−n}, ⟨v_{p}, v_{q}⟩ = δ_{p,q} (p ≤ q)},

As a maximal torus in K, we choose the diagonal subgroup

(2.9)(f)  T = \{(\begin{matrix}e^{iϕ_{1}} \\ & . & . & . \\ & & & & e^{iϕ_{n}} \\ & & & & & e^{−iϕ_{1}} \\ & & & & & & . & . & . \\ & & & & & & & & & e^{−iϕ_{n}} \end{matrix} )\} ≃ U(1)^{n}.

We claim that

(2.9)(g)  Sp(2n)/U(1)^{n} ≃ X^{Sp(2n)}.

The map from left to right is (2.9)(h)

(v_{1}, …, v_{2n})U(1)^{n} ↦ F = (F_{p}), F_{p} = span(v_{1}, …, v_{p}) (0 ≤ p ≤ n).

Again we leave to the reader the verification that this is a well-defined bijection, and the task of describing the equivariant line bundles on X.

3 Examples for SL(2, ℝ)

In this section we will present some examples of representations of SL(2, ℝ), in order to develop some feeling about what infinitesimal equivalence, minimal globalizations, and so on look like in examples. More details can be found in [Kna86], pages 35–41.

In fact it is a little simpler for these examples to consider not SL(2, ℝ) but the isomorphic group

(3.1)(a)  G = SU(1, 1) = \{(\begin{matrix}α & β \\ \overline{β} & \overline{α} \end{matrix}) | α, β ∈ ℂ, |α|^{2} − |β|^{2} = 1\} .

This is the group of linear transformations of ℂ^{2} preserving the standard Hermitian form of signature (1, 1), and having determinant 1. We will be particularly interested in a maximal compact subgroup:

(3.1)(b)  K = \{(\begin{matrix}e^{iθ} & 0 \\ 0 & e^{−iθ} \end{matrix}) | θ ∈ ℝ\} .

The group G acts on the open unit disc by linear fractional transformations:

(3.2)(a)  (\begin{matrix}α & β \\ \overline{β} & \overline{α} \end{matrix}) · z = \frac{\overline{α}z + \overline{β}}{βz + α}.

It is not difficult to check that this action is transitive:

(3.2)(b)  D = {z | |z| < 1} = G · 0 ≃ G/K.

The last identification comes from the fact that K is the isotropy group for the action at the point 0.

The action of G on D preserves complex structures. Setting

(3.2)(c)  V^{−ω} = holomorphic functions on D,

we therefore get a representation π of G on V^{−ω} by

(3.2)(d)  [π(g)f](z) = f(g^{−1} · z) = f (\frac{αz − \overline{β}}{−βz + \overline{α}}.)

The representation (π, V^{−ω}) is not irreducible, because the one-dimensional closed subspace of constant functions is invariant. Nevertheless (as we will see in section 4) the Casselman-Wallach-Schmid theory of distinguished globalizations still applies. As the notation suggests, V^{−ω} is a maximal globalization for the corresponding Harish-Chandra module

(3.2)(e)  V^{K} = polynomials in z

of K-finite vectors.

In order to describe other (smaller) globalizations of V^{K}, we can control the growth of functions near the boundary circle of D. The most drastic possibility is to require the functions to extend holomorphically across the boundary of D:

(3.3)(a)  V^{ω} = holomorphic functions on \overline{D}

This space can also be described as the intersection (over positive numbers ϵ) of holomorphic functions on discs of radius 1 + ϵ. Restriction to the unit circle identifies V^{ω} with real analytic functions on the circle whose negative Fourier coefficients all vanish. There is a natural topology on V^{ω}, making it a representation of G by the action π of (3.2)(d). As the notation indicates, this representation is Schmid’s minimal globalization of V^{K}.

A slightly larger space is

(3.3)(b) V^{∞} = holomorphic functions on D with smooth boundary values.

More or less by definition, V^{∞} can be identified with smooth functions on the circle whose negative Fourier coefficients vanish. The identification topologizes V^{∞}, and it turns out that the resulting representation of G is the Casselman-Wallach smooth globalization of V^{K}. Larger still is

(3.3)(c) V^{(2)} = holomorphic functions on D with L^{2} boundary values.

This is a Hilbert space, the square-integrable functions on the circle whose negative Fourier coefficients vanish. The representation of G on this Hilbert space is continuous but not unitary (because these linear fractional transformations of the circle do not preserve the measure). Of course there are many other function spaces on the circle that can be used in a similar way; I will mention only (3.3)(d)

V^{−∞} = holomorphic functions on D with distribution boundary values.

This is the Casselman-Wallach distribution globalization of V^{K}.

We therefore have

V^{ω} ⊂ V^{∞} ⊂ V^{(2)} ⊂ V^{−∞} ⊂ V^{−ω}.

These inclusions of representations are continuous with dense image. Holomorphic functions on the disc all have Taylor expansions

f(z) = ∑_{n=0}^{∞} a_{n}z^{n}.

We can describe each space by conditions on the coefficients a_{n}; these descriptions implicitly specify the topologies very nicely.

V^{−ω} ↔ {(a_{n}) | ∑_{n=0}^{∞} |a_{n}|(1 − ϵ)^{n} < ∞, 0 < ϵ ≤ 1}.
V^{−∞} ↔ {(a_{n}) | for some N > 0, |a_{n}| < C_{N}(1 + n)^{N}}.
V^{(2)} ↔ {(a_{n}) | ∑_{n=0}^{∞} |a_{n}|^{2} < ∞}.
V^{∞} ↔ {(a_{n}) | for every N > 0, |a_{n}| < C_{N}(1 + n)^{−N}}.
V^{ω} ↔ {(a_{n}) | ∑_{n=0}^{∞} |a_{n}|(1 + ϵ)^{n} < ∞, some ϵ > 0}.

4 Harish-Chandra modules and globalization

In this section we will recall very briefly some general facts about representations of real reductive groups. The first problem is to specify what groups we are talking about. A Lie algebra (over any field of characteristic zero) is called semisimple if it is a direct sum of non-abelian simple Lie algebras. It is natural to define a real Lie group to be semisimple if it is connected, and its Lie algebra is semisimple. Such a definition still allows some technically annoying examples (like the universal cover of SL(2, ℝ), which has no non-trivial compact subgroups). Accordingly there is a long tradition of working with connected semisimple groups having finite center. There are several difficulties with that. As we will see, there are many results relating the representation theory of G to representation theory of subgroups of G; and the relevant subgroups are rarely themselves connected and semisimple. Another difficulty comes from the demands of applications. One of the most important applications of representation theory for Lie groups is to automorphic forms. In that setting the most fundamental example is GL(n, ℝ), a group which is neither connected nor semisimple.

Most of these objections can be addressed by working with algebraic groups, and considering always the group of real points of a connected reductive algebraic group defined over ℝ. (The group GL(n) is a connected algebraic group, even though its group of real points is disconnected as a Lie group.) The difficulty with this is that it still omits some extremely important examples. Some of the most interesting representation theory lives on the nonlinear double cover Mp(2n, ℝ) of the algebraic group Sp(2n, ℝ) (consisting of linear transformations of ℝ^{2n} preserving a certain symplectic form). The “oscillator representation” of this group is fundamental to mathematical physics, to the theory of automorphic forms, and to classical harmonic analysis. (Such an assertion needs to be substantiated, and I won’t do that; but here at least are some interesting references: [Wei64], [How88], [How89].)

So we want to include at least finite covering groups of real points of connected reductive algebraic groups. At some point making a definition along these lines becomes quite cumbersome. I will therefore follow the path taken by Knapp in [Kna86], and take as the definition of reductive a property that usually appears as a basic structure theorem. The definition is elementary and short, and it leads quickly to some fundamental facts about the groups. One can object that it does not extend easily to groups over other local fields, but for the purposes of these notes that will not be a problem.

The idea is that the most basic example of a reductive group is the group GL(n, ℝ) of invertible n × n real matrices. We will recall a simple structural fact about GL(n) (the polar decomposition of Proposition 4.2 below). Then we will define a reductive group to be (more or less) any subgroup of some GL(n) that inherits the polar decomposition.

If g ∈ G = GL(n, ℝ), define

θg = ^{t}g^{−1},  (4.1)(a

the inverse of the transpose of g. The map θ is an automorphism of order 2, called the Cartan involution of GL(n). Write O(n) = GL(n)^{θ} for the subgroup of fixed points of θ. This is the group of n × n real orthogonal matrices, the orthogonal group. It is compact.

Write 𝔤𝔩(n, ℝ) = Lie(GL(n, ℝ)) for the Lie algebra of GL(n) (the space of all n × n real matrices). The automorphism θ of G differentiates to an involutive automorphism of the Lie algebra, defined by

(4.1)(b)  (dθ)(X) = −^{t}X.

Notice that if X happens to be invertible, then (dθ)(X) and θ(X) are both defined, and they are not equal. Despite this potential for confusion, we will follow tradition and abuse notation by writing simply θ for the differential of θ. The −1-eigenspace of θ on the Lie algebra is

(4.1)(c)  𝔭_{0} = n × n symmetric matrices.

Proposition 4.2. (Polar or Cartan decomposition for GL(n, ℝ)). Suppose G = GL(n, ℝ), K = O(n), and 𝔭_{0} is the space of n × n symmetric matrices. Then the map

O(n) × 𝔭_{0} → GL(n) (k, X) ↦ k exp(X)

is an analytic diffeomorphism of O(n) × 𝔭_{0} onto GL(n).

Definition 4.3. A linear reductive group is a subgroup G ⊂ GL(n, ℝ) such that

(1) G is closed (and therefore G is a Lie group).

(2) G has finitely many connected components.

(3) G is preserved by the Cartan involution θ of GL(n, ℝ) (cf. (4.1)(a)).

Of course the last requirement means simply that the transpose of each element of G belongs again to G. The restriction of θ to G (which we still write as θ) is called the Cartan involution of G. Define

K = G ∩ O(n) = G^{θ},

a compact subgroup of G. Write

𝔤_{0} = Lie(G) ⊂ 𝔤𝔩(n, ℝ).

Finally, define

𝔰_{0} = symmetric matrices in 𝔤_{0},

the −1-eigenspace of θ.

Proposition 4.4. (Cartan decomposition for linear real reductive groups). Suppose G ⊂ GL(n, ℝ) is a linear reductive group, K = G ∩ O(n), and 𝔰_{0} is the space of symmetric matrices in the Lie algebra of G. Then the map

K × 𝔰_{0} → G, (k, X) ↦ k exp(X)

is an analytic diffeomorphism of K × 𝔰_{0} onto G.

One immediate consequence of this proposition is that K is a maximal compact subgroup of G; that is, that any subgroup of G properly containing K must be noncompact.

Here is a result connecting this definition with a more traditional one.

Proposition 4.5. Suppose H is a reductive algebraic group defined over ℝ, and π: H → GL(V) is a faithful representation defined over ℝ. Then we can choose a basis of V = V(ℝ) in such a way that the corresponding embedding

π: H(ℝ) → GL(n, ℝ)

has image a linear reductive group in the sense of Definition 4.3.

Conversely, suppose G is a linear reductive group in the sense of Definition 4.3. Then we can choose H and π as above in such a way that

π(H(ℝ))_{0} = G_{0};

that is, these two groups have the same identity component.

Here at last is the main definition.

Definition 4.6. A real reductive group is a Lie group G̃ endowed with a surjective homomorphism

π: G̃ → G ⊂ GL(n, ℝ)

onto a linear reductive group, such that ker π is finite. Use the differential of π to identify

𝔤̃_{0} = Lie(G̃) ≃ Lie(G) = 𝔤_{0} ⊂ 𝔤𝔩(n, ℝ).

This identification makes θ into an automorphism θ̃ of 𝔤̃_{0}. Define

K̃ = π^{−1}(K) ⊂ G̃,

a compact subgroup (since π has finite kernel). Finally, define

𝔰̃_{0} = −1-eigenspace of θ̃ on 𝔤̃_{0}.

In the next statement we will use tildes to distinguish elements of G̃ and the exponential map of G̃ for clarity; this is also helpful in writing down the (very easy) proof based on Proposition 4.5. But thereafter we will drop all the tildes.

Proposition 4.7. (Cartan decomposition for real reductive groups). Suppose G̃ is a real reductive group as in Definition 4.6. Then the map

K̃ × 𝔰_{0} → G̃, (k̃, X̃) ↦ k̃ exp^{∼}(X̃)

is an analytic diffeomorphism of K̃ × 𝔰̃_{0} onto G̃. Define a diffeomorphism θ̃ of G̃ by

θ̃(k̃ exp^{∼}(X̃)) = k̃ exp^{∼}(−X̃).

Then θ̃ is an automorphism of order two, with fixed point group K̃.

We turn now to the problem of exploiting this structure for understanding representations of a reductive group G. Recall from Definition 1.1 the notion of representation of G.

Definition 4.8. Suppose (π, V) is a representation of G. A vector v ∈ V is said to be smooth (respectively analytic) if the map

G → V, g ↦ π(g) · v

is smooth (respectively analytic). Write V^{∞} (respectively V^{ω}) for the space of smooth (respectively analytic) vectors in V. When the group G is not clear from context, we may write for example V^{∞,G}.

Each of V^{∞} and V^{ω} is a G-stable subspace of V; we write π^{∞} and π^{ω} for the corresponding actions of G. Each of these representations differentiates to a representation dπ of the Lie algebra 𝔤_{0}, and hence also of the enveloping algebra U(𝔤). (We will always write

(4.9)  𝔤_{0} = Lie(G),  𝔤 = 𝔤_{0} ⊗_{ℝ} ℂ,

and use analogous notation for other Lie groups.) Each of V^{∞} and V^{ω} has a natural complete locally convex topology, making the group representations continuous. In the case of V^{∞}, this topology can be given by seminorms

v ↦ ρ(dπ(u)v)

with ρ one of the seminorms defining the topology of V, and u ∈ U(𝔤). Since the enveloping algebra has countable dimension, it follows at once that V^{∞} is Fréchet (topologized by countably many seminorms) whenever V is Fréchet. The condition that the function π(g)v be real analytic may be expressed in terms of the existence of bounds on derivatives of the function: that if X_{1}, X_{2}, …, X_{m} is a basis of 𝔤_{0}, and g_{0} ∈ G, then there should exist ϵ > 0 and a neighborhood U of g_{0} so that for any seminorm ρ on V,

ρ(dπ(X^{I})π(g)v) ≤ C_{ϵ}I! ϵ^{−|I|}

for all multiindices I = (i_{1}, …, i_{m}) and all g ∈ U. Here we use standard multiindex notation, so that

X^{I} = X_{1}^{i_{1}} ⋯ X_{m}^{i_{m}}, |I| = ∑_{j=1}^{m} i_{j},

and so on. This description suggests how to define the topology on V^{ω} as an inductive limit (over open coverings of G, with positive numbers ϵ(U) attached to each set in the cover).

It is a standard theorem (due to Gårding, and true for any Lie group) that V^{∞} is dense in V. I am not certain in what generality the density of V^{ω} in V is known; we will recall (in Theorem 4.13) Harish-Chandra’s proof of this density in enough cases for our purposes.

One of Harish-Chandra’s fundamental ideas was the use of relatively easy facts in the representation theory of compact groups to help in the study of representations of G. Here are some basic definitions.

Definition 4.10. Suppose (π, V) is a representation of a compact Lie group K. A vector v ∈ V is said to be K-finite if it belongs to a finite-dimensional K-invariant subspace. Write V^{K} for the space of all K-finite vectors in V.

Suppose (μ, E_{μ}) is an irreducible representation of K. (Then E_{μ} is necessarily finite-dimensional, and carries a K-invariant Hilbert space structure.) The μ-isotypic subspace V (μ) is the span of all copies of E_{μ} inside V.

Proposition 4.11. Suppose (π, V) is a representation of a compact Lie group K. Then

V^{K} ⊂ V^{ω,K} ⊂ V^{∞,K} ⊂ V;

V^{K} is dense in V. There is an algebraic direct sum decomposition

V^{K} = ∑_{μ∈K̂} V (μ).

Each subspace V (μ) is closed in V, and so inherits a locally convex topology. There is a unique continuous operator

P (μ): V → V (μ)

commuting with K and acting as the identity on V (μ). For any v ∈ V and μ ∈ K̂, we can therefore define

v_{μ} = P (μ)v ∈ V (μ).

If v ∈ V^{∞,K}, then

v = ∑_{μ∈K̂} v_{μ},

an absolutely convergent series.

Finally, define

V^{−K} = ∏_{μ∈K̂} V (μ),

the algebraic direct product. The operators P (μ) define an embedding

V ↪ V^{−K}, v ↦ ∏ v_{μ}.

There are natural complete locally convex topologies on V^{K} and V^{−K} making all the inclusions here continuous, but we will have no need of this.

Returning to the world of reductive groups, here is Harish-Chandra’s basic definition. The definition will refer to

ℨ^{G}(𝔤) = Ad(G)-invariant elements of U(𝔤).

If G is connected, this is just the center of the enveloping algebra. Schur’s lemma suggests that ℨ^{G}(𝔤) ought to act by scalars on an irreducible representation of G, but there is no general way to make this suggestion into a theorem. (Soergel gave an example of an irreducible Banach representation of SL(2, ℝ) in which ℨ^{G}(𝔤) does not act by scalars.) Nevertheless the suggestion is correct for most representations arising in applications, so Harish-Chandra made it into a definition.

Definition 4.12. Suppose G is real reductive with maximal compact subgroup K, and (π, V) is a representation of G. We say that π is admissible if π has finite length, and either of the following equivalent conditions is satisfied:

(1) for each μ ∈ K̂, the isotypic space V (μ) is finite-dimensional.

(2) each v ∈ V^{∞} is contained in a finite-dimensional subspace preserved by dπ(ℨ^{G}(𝔤)).

The assumption of “finite length” means that V has a finite chain of closed invariant subspaces in which successive quotients are irreducible. Harish-Chandra actually called the first condition “admissible” and the second “quasisimple,” and he proved their equivalence. It is the term admissible that has become standard now, perhaps because it carries over almost unchanged to the setting of p-adic reductive groups. Harish-Chandra also proved that a unitary representation of finite length is automatically admissible.

Theorem 4.13. (Harish-Chandra). Suppose (π, V) is an admissible representation of a real reductive group G with maximal compact subgroup K.

(1) V^{K} ⊂ V^{ω} ⊂ V^{∞}.

(2) The subspace V^{K} is preserved by the representations of 𝔤 and K.

(3) There is a bijection between the set

{closed G-stable subspaces W ⊂ V }

and the set

{arbitrary (𝔤, K)-stable subspaces W^{K} ⊂ V^{K}} .

Here W corresponds to its subspace W^{K} of K-finite vectors, and W^{K} corresponds to its closure W in V.

The structure carried by V^{K} is fundamental, and has a name of its own.

Definition 4.14. Suppose G is a real reductive group with complexified Lie algebra 𝔤 and maximal compact subgroup K. A (𝔤, K)-module is a vector space X endowed with actions of 𝔤 and of K, subject to the following conditions.

(1) Each vector in X belongs to a finite-dimensional K-stable subspace, on which the action of K is continuous.

(2) The differential of the action of K (which exists by the first condition) is equal to the restriction of the action of 𝔤.

(3) The action map

𝔤 × X → X, (Z, x) ↦ Z · x

is equivariant for the actions of K. (Here K acts on 𝔤 by Ad.)

Harish-Chandra’s Theorem 4.13 implies that if (π, V) is an admissible irreducible representation of G, then V^{K} is an irreducible (𝔤, K)-module. We call V^{K} the Harish-Chandra module of π. We say that two such representations (π, V) and (ρ, W) are infinitesimally equivalent if V^{K} ≃ W^{K} as (𝔤, K)- modules.

Theorem 4.15. (Harish-Chandra). Every irreducible (𝔤, K)-module arises as the Harish-Chandra module of an irreducible admissible representation of G on a Hilbert space.

(Actually Harish-Chandra proved this theorem only for linear reductive groups G. The general case was completed by Lepowsky.) In light of this theorem and the preceding definitions, we define

Ĝ_{adm} =infinitesimal equivalence classes of

irreducible admissible representations

=equivalence classes of irreducible (𝔤, K)-modules.

A continuous group representation with Harish-Chandra module X is called a globalization of X.

We fix now a finite length (𝔤, K)-module

(4.16)(a)  X = ∑_{μ∈K̂} X(μ).

In addition, we fix a Hilbert space globalization

(4.16)(b)  (π^{Hilb}, X^{Hilb})

(For irreducible X, such a globalization is provided by Theorem 4.15. For X of finite length, the existence of X^{Hilb} is due to Casselman.) For the purposes of the theorems and definitions that follow, any reflexive Banach space globalization will serve equally well; with minor modifications, one can work with a reflexive Fréchet representation of moderate growth (see [Cas89], Introduction). The Hilbert (or Banach) space structure restricts to a norm

(4.16)(c)  ∥ ∥_{μ}: X(μ) → ℝ

We will need also the dual Harish-Chandra module

X^{dual} = K-finite vectors in the algebraic dual of X.

The contragredient representation of G on the dual space (X^{Hilb})′, defined by

(4.16)(d)  (π^{Hilb})′(g) = ^{t}(π^{Hilb}(g^{−1}))

has Harish-Chandra module X^{dual}. (We will discuss the transpose of a linear map and duality in general in more detail in section 8.) In particular, (X^{Hilb})′(μ) = X(μ)′ inherits the norm

(4.16)(e)  ∥ ∥′_{μ}: X(μ) → ℝ

from (X^{Hilb})′. This is unfortunately not precisely the dual norm to ∥ ∥_{μ}, but the difference can be controlled^{2}: there is a constant C ≥ 1 so that

2 The difference is that the dual norm to ∥ ∥_{μ} involves the size of a linear functional only on elements of X(μ), whereas ∥ ∥′_{μ} involves the size of a linear functional on the whole space. The second inequality in (4.16)(f) is obvious. The constant in the first inequality is an estimate for the norm of the projection operator P (μ) from Proposition 4.11. The estimate comes from the standard formula for P (μ) as an integral over K of π(k) against the character of μ.

(4.16)(f)  (C · dim μ)^{−1}∥ ∥′_{μ} ≤ (∥ ∥_{μ})′ ≤ ∥ ∥′_{μ}.

A Hilbert space globalization is technically valuable in the subject, but it has a very serious weakness: it is not canonically defined, even up to a bounded operator. More concretely, suppose that X happens to be unitary, so that there is a canonical Hilbert space globalization X^{Hilb^{∼}} (coming from the unitary structure). The nature of the problem is that the infinitesimal equivalence of X^{Hilb} and X^{Hilb^{∼}} need not be implemented by a bounded operator from X^{Hilb} to X^{Hilb^{∼}}. A consequence is that the invariant Hermitian form on X giving rise to the unitary structure need not be defined on all of X^{Hilb}: we cannot hope to look for unitary structures by looking for Hermitian forms on random Hilbert space globalizations. Here is the technical heart of the work of Wallach, Casselman, and Schmid addressing this problem.

Theorem 4.17. (Casselman-Wallach; see [Cas89]). In the setting of (4.16), the norm ∥ ∥_{μ} is well-defined up to a polynomial in |μ| (which means the length of the highest weight of the representation μ of K): if ∥ ∥_{μ}^{∼} is the collection of norms arising from any other Hilbert (or reflexive Banach) globalization of X, then there are a positive integer M and a constant C_{M} so that

∥ ∥_{μ} ≤ C_{M}(1 + |μ|)^{M}∥ ∥_{μ}^{∼}.

The proof of this result given by Wallach and Casselman is quite complicated and indirect; indeed it is not entirely easy even to extract the result from their papers. It would certainly be interesting to find a more direct approach: beginning with the Harish-Chandra module X, to construct the various norms ∥ ∥_{μ} (defined up to inequalities like those in Theorem 4.17); and then to prove directly that the topological vector spaces constructed in Theorems 4.18, 4.20 carry smooth representations of G. The first step in this process (defining the norms) is perhaps not very difficult. The second seems harder.

Theorem 4.18. (Casselman-Wallach; see [Cas89]). In the setting of (4.16), the space of smooth vectors of X^{Hilb} is

X^{Hilb,∞} = { ∑_{μ∈K̂} x_{μ} | x_{μ} ∈ X(μ), ∥x_{μ}∥_{μ} rapidly decreasing in |μ|} .

Here “rapidly decreasing” means that for every positive integer M there is a constant C_{N} so that

∥x_{μ}∥_{μ} ≤ C_{N}(1 + |μ|)^{−N}.

The minimum possible choice of C_{N} defines a seminorm; with these seminorms, X^{Hilb,∞} is a nuclear Fréchet space.

Regarded as a collection of sequences of elements chosen from X(μ), the space of smooth vectors and its topology are independent of the choice of the globalization X^{Hilb}.

Definition 4.19. Suppose that X is any Harish-Chandra module of finite length. The Casselman-Wallach smooth globalization of X is the space of smooth vectors in any Hilbert space globalization of X. We use Theorem 4.18 to identify it as a space of sequences of elements of X, and denote it X^{∞}.

There is a parallel description of analytic vectors.

Theorem 4.20. (Schmid; see [Sch85]). In the setting of (4.16), the space of analytic vectors of X^{Hilb} is

X^{Hilb,ω} = { ∑_{μ∈K̂} x_{μ} | x_{μ} ∈ X(μ), ∥x_{μ}∥_{μ} exponentially decreasing in |μ|} .

Here “exponentially decreasing” means that there are an ϵ > 0 and a constant C_{ϵ} so that

∥x_{μ}∥_{μ} ≤ C_{ϵ}(1 + ϵ)^{−|μ|}.

The minimum choice of C_{ϵ} defines a Banach space structure on a subspace, and X^{Hilb,ω} has the inductive limit topology, making it the dual of a nuclear Fréchet space.

Regarded as a collection of sequences of elements chosen from X(μ), the space of analytic vectors and its topology are independent of the choice of the globalization X^{Hilb}.

Definition 4.21. Suppose that X is any Harish-Chandra module of finite length. Schmid’s minimal or analytic globalization of X is the space of analytic vectors in any Hilbert space globalization of X. We use Theorem 4.18 to identify it as a space of sequences of elements of X, and denote it X^{ω}.

Finally, we will need the duals of these two constructions.

Definition 4.22. Suppose that X is any Harish-Chandra module of finite length. The Casselman-Wallach distribution globalization of X is the continuous dual of the space of smooth vectors in any Hilbert space globalization of X^{dual}. We can use Theorem 4.18 to identify it as a space of sequences of elements of X, and denote it X^{−∞}. Explicitly,

X^{−∞} = { ∑_{μ∈K̂} x_{μ} | x_{μ} ∈ X(μ), ∥x_{μ}∥_{μ} slowly increasing in |μ|} .

Here “slowly increasing” means that there is a positive integer N and a constant C_{N} so that

∥x_{μ}∥_{μ} ≤ C_{N}(1 + |μ|)^{N}.

This exhibits X^{−∞} as an inductive limit of Banach spaces, and the dual of the nuclear Fréchet space X^{dual,∞}.

Definition 4.23. Suppose that X is any Harish-Chandra module of finite length. Schmid’s maximal or hyperfunction globalization of X is the continuous dual of the space of analytic vectors in any Hilbert space globalization of X^{dual}. We can use Theorem 4.19 to identify it as a space of sequences of elements of X, and denote it X^{−ω}. Explicitly,

X^{−ω} = { ∑_{μ∈K̂} x_{μ} | x_{μ} ∈ X(μ),

∥x_{μ}∥_{μ} less than exponentially increasing in |μ|} .

Here “less than exponentially increasing” means that for every ϵ > 0 there is a constant C_{ϵ} so that

∥x_{μ}∥_{μ} ≤ C_{ϵ}(1 + ϵ)^{|μ|}.

This exhibits X^{−ω} as a nuclear Fréchet space.

When we wish to emphasize the K-finite nature of X, we can write the space as X^{K} = ∑_{μ∈K̂} X(μ). It is also convenient to write

X^{−K} = ∏_{μ∈K̂} X(μ) = (X^{dual})^{∗}.

Our various globalizations now appear as sequence spaces, with gradually weakening conditions on the sequences:

(4.24)  X^{K} ⊂ X^{ω} ⊂ X^{∞} ⊂ X^{−∞} ⊂ X^{−ω} ⊂ X^{−K}.

The conditions on the sequences are: almost all zero, exponentially decreasing, rapidly decreasing, slowly increasing, less than exponentially increasing, and no condition. All of the conditions are expressed in terms of the norms chosen in (4.16), and the naturality of the definitions depends on Theorem 4.17. We could also insert our Hilbert space X^{Hilb} in the middle of the list (between X^{∞} and X^{−∞}), corresponding to sequences in ℓ^{2}. I have not done this because that sequence space does depend on the choice of X^{Hilb}.

Even the spaces X^{K} and X^{−K} carry natural complete locally convex topologies (still given by the sequence structure); the representations of 𝔤 and K are continuous for these topologies. (The group G will not act on either of them unless X is finite-dimensional.)

5 Real parabolic induction and the globalization functors

In order to get some feeling for the various globalization functors defined in section 4, we are going to compute them in the setting of parabolically induced representations. Logically this cannot be separated from the definition of the functors: the proof by Wallach and Casselman of Theorem 4.15 proceeds by embedding arbitrary representations in parabolically induced representations, and computing there. But we will ignore these subtleties, taking the results of section 4 as established.

Throughout this section, G will be a real reductive group with Cartan involution θ and maximal compact subgroup K, as in Definition 4.6. We want to construct representations using parabolic subgroups of G, so the first problem is to say what a parabolic subgroup is. In part because of the possible disconnectedness of G, there are several possible definitions. We want to take advantage of the fact that the complexified Lie algebra 𝔤 (cf. (4.9)) is a complex reductive Lie algebra, for which lots of structure theory is available.

Definition 5.1. A real parabolic subgroup of the real reductive group G is a Lie subgroup P ⊂ G with the property that 𝔭 = Lie(P)_{ℂ} is a parabolic subalgebra of 𝔤. Write

𝔲 = nil radical of 𝔭;

because this is an ideal preserved by all automorphisms of 𝔭 as a real Lie algebra, it is the complexification of an ideal 𝔲_{0} of 𝔭_{0}. Let U be the connected Lie subgroup of P with Lie algebra 𝔲_{0}; it is a nilpotent Lie group, normal in P.

This is the most liberal possible definition of real parabolic subgroup. The most restrictive would require in addition that P be the normalizer of 𝔭 (under the adjoint action) in G.

The quotient Lie algebra 𝔭/𝔲 is reductive, and is always represented by a subalgebra (a Levi factor) of 𝔭. But the Levi factor is not unique, and picking a good one is often a slightly delicate matter. In the present setting this problem is solved for us. Because θ is an automorphism of G, θP is another parabolic subgroup. Define

(5.2)  L = P ∩ θP,

a θ-stable Lie subgroup of G.

Proposition 5.3. Suppose P is a parabolic subgroup of the real reductive group G, and L = P ∩ θP.

(1) The subgroups P, L, and U are all closed in G.

(2) Multiplication defines a diffeomorphism

L × U → P, (l, u) ↦ lu.

In particular, L ≃ P/U.

(3) L is a reductive subgroup of G, with Cartan involution θ|_{L}.

(4) The exponential map is a diffeomorphism from 𝔲_{0} onto U.

(5) Every element of G is a product (not uniquely) of an element of K and an element of P: G = KP. Furthermore

P ∩ K = L ∩ K

is a maximal compact subgroup of L and of P. Consequently there are diffeomorphisms of homogeneous spaces

G/P ≃ K/L ∩ K, G/K ≃ P/L ∩ K.

The first of these is K-equivariant, and the second P -equivariant.

(6) The map

K × (𝔩_{0} ∩ 𝔰_{0}) × U → G, (k, X, u) ↦ k · exp(X) · u

is an analytic diffeomorphism.

The last assertion interpolates between the Iwasawa decomposition (the case when P is a minimal parabolic subgroup) and the Cartan decomposition (the case when P is all of G, or more generally when P is open in G). We saw in section 2 (after 2.5) an example of the diffeomorphism in (5), with G = GL(n, ℂ), P the Borel subgroup of upper triangular matrices, and K = U(n). In this case L is the group of diagonal matrices in GL(n, ℂ), and L ∩ K = U(1)^{n}.

How does one find parabolic subgroups? The easiest examples are “block upper-triangular” subgroups of GL(n, ℝ). I will assume that if you’ve gotten this far, those subgroups are more or less familiar, and look only at more complicated reductive groups.

It’s better to ask instead how to find the homogeneous spaces G/P, in part because construction of representations by induction really takes place on the whole homogeneous space and not just on the isotropy group P. A good answer is that one begins with the corresponding homogeneous spaces related to the complex Lie algebra 𝔤, and looks for appropriate orbits of G on those spaces. We will give some more details about this approach in section 6; but here is one example.

Suppose p and q are non-negative integers, and n = p + q. The standard Hermitian form of signature (p, q) on ℂ^{n} is

⟨v, w⟩_{p,q} = ∑_{j=1}^{p} v_{j}\overline{w}_{j} − ∑_{k=1}^{q} v_{p+k}\overline{w}_{p+k}.

The group G = U(p, q) of complex linear transformations preserving this form is a real reductive group, with maximal compact subgroup

(5.4)(a)  K = U(p) × U(q).

The group G does not have obvious “block upper-triangular” subgroups; but here is a way to make a parabolic. Fix a non-negative integer r ≤ p, q, and define

(5.4)(b)  f_{j} = e_{j} + ie_{p+j}, g_{j} = e_{j} − ie_{p+j}  (1 ≤ j ≤ r)

The subspace

I_{r} = span(f_{1}, …, f_{r})

is an r-dimensional isotropic plane (for the form ⟨v, w⟩_{p,q}), and so is its complex conjugate

\overline{I}_{r} = span(g_{1}, …, g_{r}).

Define

(5.4)(c)  P_{r} = stabilizer of I_{r} in U(p, q);

this will turn out to be a parabolic subgroup of U(p, q). One checks easily that

θP_{r} = stabilizer of \overline{I}_{r} in U(p, q).

Writing ℂ^{p,q} for ℂ^{n} endowed with the Hermitian form ⟨v, w⟩_{p,q}, we find a natural vector space decomposition

ℂ^{p,q} = I_{r} ⊕ \overline{I}_{r} ⊕ ℂ^{p−r,q−r}.

This decomposition provides an embedding

(5.4)(d)  GL(r, ℂ) × U(p − r, q − r) ↪ U(p, q) :

a matrix g in GL(r, ℂ) acts as usual on the basis \{f_{j}\} of I_{r}, by ^{t}\overline{g}^{−1} on the basis \{g_{j}\} of \overline{I}_{r}, and trivially on ℂ^{p−r,q−r}. Now it is easy to check that

(5.4)(e)  L_{r} = P_{r} ∩ θP_{r} = GL(r, ℂ) × U(p − r, q − r).

(I will leave to the reader the problem of describing the group U_{r} explicitly. As a hint, my calculations indicate

dim U_{r} = r(2[(p − r) + (q − r)] + r).

If my calculations are incorrect, please disregard this hint.)

The example shows that

(5.4)(f)  G/P_{r} ≃ r-dimensional isotropic subspaces of ℂ^{p,q}.

As r varies from 1 to min(p, q), we get in this way all the maximal proper parabolic subgroups of U(p, q). Smaller parabolic subgroups can be constructed directly in similar ways, or by using the following general structural fact.

Proposition 5.5. Suppose G is a real reductive group and P = LU is a parabolic subgroup. Suppose Q_{L} = M_{L}N_{L} is a parabolic subgroup of L. Then

Q = Q_{L}U = M_{L}(N_{L}U)

is a parabolic subgroup of G, with Levi factor M_{L} and unipotent radical N = N_{L}U. This construction defines a bijection

parabolic subgroups of L ↔ parabolic subgroups of G containing P.

If you believe that the P_{r} are all the maximal parabolics in U(p, q), and if you know about parabolic subgroups of GL(r, ℂ), then you see that the conjugacy classes of parabolic subgroups of U(p, q) are parametrized by sequences (possibly empty) r = (r_{1}, …, r_{s}) of positive integers, with the property that

r = ∑ r_{j} ≤ min(p, q).

The Levi subgroup L_{r} of P_{r} is

L_{r} = GL(r_{1}, ℂ) × ⋯ × GL(r_{s}, ℂ) × U(p − r, q − r).

Parallel analyses can be made for all the classical groups (although the possibilities for disconnectedness can become quite complicated).

We turn now to representation theory.

Definition 5.6. Suppose P = LU is a parabolic subgroup of the reductive group G. A representation (τ, Y) of P is called admissible if its restriction to L is admissible (Definition 4.12). In this case the Harish-Chandra module of Y is the (𝔭, L ∩ K)-module Y^{L∩K} of L ∩ K-finite vectors in Y.

The notation here stretches a bit beyond what was defined in section 4, but I hope that is not a serious problem. The easiest way to get admissible representations of P is from admissible representations of L, using the isomorphism L ≃ P/U. That is, we extend a representation of L to P by making U act trivially. Any irreducible admissible representation of P is of this form. On any admissible representation (τ, Y) of P, the group U must act unipotently, in the following strong sense: there is a finite chain

0 = Y_{0} ⊂ Y_{1} ⊂ ⋯ ⊂ Y_{m} = Y

of closed P -invariant subspaces, with the property that U acts trivially on each subquotient Y_{i}/Y_{i−1}. One immediate consequence is that the action of U is analytic on all of Y, so that (for example) the P -analytic vectors for Y are the same as the L-analytic vectors.

The main reason for allowing representations of P on which U acts nontrivially is for the Casselman-Wallach proof of Theorem 4.17. They show that (for P minimal) any admissible representation of G can be embedded in a representation induced from an admissible representation of P. This statement is not true if one restricts to representations trivial on U.

So how do we pass from a representation of P to a representation of G? Whenever G is a topological group, H a closed subgroup, and (τ, Y) a representation of H, the induced representation of G is defined on a space like

(5.7)(a)  X = {f: G → Y | f(xh) = τ(h)^{−1}f(x) (x ∈ G, h ∈ H)}.

The group G acts on such functions by left translation:

(5.7)(b)  (π(g)f)(x) = f(g^{−1}x).

To make the definition precise, one has to decide exactly which functions to use, and then to topologize X so as to make the representation continuous. Depending on exactly what structures are available on G, H, and Y, there are many possibilities: continuous functions, smooth functions, analytic functions, measurable functions, integrable functions, distributions, and many more.

To be more precise in our setting, let us fix

(5.8)(a)  Y^{L∩K} = admissible (𝔭, L ∩ K)-module;

by “admissible” we mean that Y^{L∩K} should have finite length as an (𝔩, L∩K)- module. The theory of globalizations in section 4 extends without difficulty to cover admissible (𝔭, L ∩ K)-modules. This means first of all that Y^{L∩K} is the Harish-Chandra module of an admissible Hilbert space representation

(5.8)(b)  (τ^{Hilb}, Y^{Hilb}).

Using this Hilbert space representation, we can construct the subrepresentations of smooth and analytic vectors, and dually the distribution and hyperfunction vectors (duals of the smooth and analytic vectors in a Hilbert globalization of Y^{L∩K,dual}). In the end, just as in (4.24), we have

(5.8)(c)  Y^{L∩K} ⊂ Y^{ω} ⊂ Y^{∞} ⊂ Y^{Hilb} ⊂ Y^{−∞} ⊂ Y^{−ω} ⊂ Y^{−L∩K}.

These are complete locally convex topological vector spaces; the inclusions are continuous with dense image. All but the first and last carry irreducible representations of P, which we denote τ^{ω}, etc. When Y^{L∩K} is finite-dimensional (as is automatic for P minimal), all of these spaces are the same.

We now want to use these representations of P and the general idea of (5.7) to construct representations of G. That is, we want to begin with one of the representations Y of (5.8)(a), and define an appropriate space of functions

(5.9)(a)  X = {f: G → Y | f(xp) = τ(p)^{−1}f(x) (x ∈ G, p ∈ P }.

What we will use constantly is Proposition 5.3(5). This provides an identification

(5.9)(b)  X ≃ {f: K → Y | f(kl) = τ(l)^{−1}f(k) (k ∈ K, l ∈ L ∩ K}.

The description of X in (5.9)(a) is called the “induced picture”; we may write X_{ind} to emphasize that. The description in (5.9)(b) is the “compact picture,” and may be written X_{cpt}. The great advantage of the first picture is that the action of G (by left translation) is apparent. The great advantage of the second is that many questions of analysis come down to the compact group K. Eventually we will need to understand at least the action of the Lie algebra 𝔤 in the compact picture; a formula appears in (5.14)(d). For the moment, notice that an element of X_{ind} is continuous (respectively measurable) if and only if the corresponding element of X_{cpt} is continuous (respectively measurable). If the representation τ is smooth (respectively analytic), then the same is true of smooth (respectively analytic) functions.

The classical setting for induction is unitary representations. In the setting of (5.7), suppose Y is a Hilbert space, with Hilbert space norm ∥·∥_{Y} preserved by H. We will choose the space X in (5.7)(a) to consist of certain measurable functions from G to Y. If f is such a function, then

(5.10)(a)  g ↦ ∥f(g)∥_{Y}

is a non-negative real-valued measurable function on G. Because of the transformation law on f imposed in (5.7)(a), this function is actually right-invariant under H:

(5.10)(b)  ∥f(gh)∥_{Y} = ∥f(g)∥_{Y}  (g ∈ G, h ∈ H)

We want to find a Hilbert space structure on some of these functions f. A natural way to do that is to require ∥f(g)∥_{Y} to be square-integrable in some sense. Because of the H-invariance in (5.9)(b), what is natural is to integrate over the homogeneous space G/H. That is, we define a Hilbert space norm on these functions by

(5.10)(c)  ∥f∥_{X}^{2} = ∫_{G/H} ∥f(g)∥_{Y}^{2} d\overline{g}.

Here d\overline{g} is some measure on G/H; the integrand is actually a function on G/H by (5.10)(b). The Hilbert space for the G representation is then

(5.10)(d)  X = {f as in (5.7)(a) measurable, ∥f∥_{X} < ∞}.

The group G will preserve this Hilbert space structure (that is, the representation will be unitary) if d\overline{g} is a G-invariant measure.

Let us see how to use this idea and our Hilbert space representation Y^{Hilb} of P to construct a Hilbert space representation of G. There are two difficulties. First, the representation Y^{Hilb} need not be unitary for P, so (5.10)(b) need not hold: the function ∥f(g)∥_{Y^{Hilb}} need not descend to G/P. Second, the homogeneous space G/P carries no nice G-invariant measure (unless P is open in G); so we cannot hope to get a unitary representation of G even if τ^{Hilb} is unitary.

Mackey found a very general way to address the second problem, essentially by tensoring the representation τ by a certain one-dimensional character of P defining the bundle of “half-densities” on G/P. This is the source of a strange exponential term (for example the “ρ” in section VII.1 of [Kna86]) in many formulas for induced representations. There is a long-winded explanation in Chapter 3 of [Vog87]. Because we will not be using parabolic induction to construct unitary representations, we will ignore this problem (and omit the “ρ” from the definition of parabolic induction). If the action of G changes the measure d\overline{g} in a reasonable way, we can still hope that G will act by bounded operators on the Hilbert space of (5.10)(d).

The first problem is more serious, since it seems to prevent us even from writing down an integral defining a Hilbert space. The function we want to integrate is defined on all of G, but it is dangerous to integrate over G: if the representation of P were unitary, the function would be constant on the cosets of P, so the integral (at least with respect to Haar measure on G) would not converge. This suggests using instead of Haar measure some measure on G that decays at infinity in some sense. (One might at first be tempted to use the delta function, assigning the identity element of G the measure 1 and every other element the measure zero. This certainly takes care of convergence problems, but this measure behaves so badly under translation by G that G fails to act continuously on the corresponding Hilbert space).

A reasonable resolution is hiding in Proposition 5.3(5).

Proposition 5.11. Suppose P is a parabolic subgroup of the real reductive group G, and (τ^{Hilb}, Y^{Hilb}) is an admissible representation of P on a Hilbert space. Define

X_{cpt}^{Hilb} = {f: K → Y^{Hilb} measurable |

f(kl) = τ(l)^{−1}f(k), ∫_{K} ∥f(k)∥_{Y^{Hilb}}^{2}dk < ∞}.

Here dk is the Haar measure on K of total mass 1; the norm on X_{cpt}^{Hilb} is the square root of the integral in the definition. Define X_{ind}^{Hilb} to be the corresponding space of functions on G, using the identification in (5.9). Then X_{ind}^{Hilb} is preserved by left translation by G. The corresponding representation π^{Hilb} of G is continuous and admissible; its restriction to K is unitary.

It may seem strange that we have obtained a unitary representation of K even though we did not assume that τ^{Hilb} was unitary on L ∩ K. This is possible because we have integrated over K rather than over K/L ∩ K. If we apply this proposition with P = G (so that τ^{Hilb} is a representation of G), then X^{Hilb} = Y^{Hilb} as a topological vector space, but the Hilbert space structures ∥ · ∥_{Y^{Hilb}}^{2} and ∥ · ∥_{X^{Hilb}}^{2} are different: the latter is obtained by averaging the former over K.

We now have a Hilbert space globalization of a Harish-Chandra module for G, so the machinery of section 4 can be applied. To begin, it is helpful to write down the Harish-Chandra module for G explicitly. This is

(5.12)(a) X^{K} = {f: G → Y^{Hilb} | f(xp) = τ(p)^{−1}f(x), and f left K-finite}.

In order to understand this as a vector space, it is most convenient to use the “compact picture” of (5.9)(b): (5.12)(b)

X_{cpt}^{K} = {f: K → Y^{Hilb} | f(kl) = τ(l)^{−1}f(k), and f left K-finite}.

Now a function f in X_{cpt}^{K} can transform on the left according to a representation μ of K only if it transforms on the right according to representations of L ∩ K appearing in the restriction of the dual of μ. It follows that the functions in X_{cpt}^{K} must take values in Y^{L∩K}. (This is not true of the corresponding functions in the induced picture (5.12)(a).) Therefore (5.12)(c)

X_{cpt}^{K} = {f: K → Y^{L∩K} | f(kl) = τ(l)^{−1}f(k), and f left K-finite}.

As in (5.9), the drawback of this description of X^{K} is that the action of the Lie algebra 𝔤 is not as clear as in (5.12)(a).

We turn next to the determination of X^{∞}, the space of smooth vectors in X^{Hilb}. Recall that “smooth” refers to the differentiability of the action of G, not directly to smoothness as functions on G. What is more or less obvious (from standard theorems saying that functions on compact manifolds with lots of L^{2} derivatives are actually smooth) is this:

(5.13)(a) f ∈ X_{cpt}^{Hilb} is smooth for the representation of K if and only if

it is smooth as a function on K with values in Y^{Hilb}.

Smoothness of a function on K may be tested by differentiating by Lie algebra elements either on the left or on the right. Because of the transformation property imposed under L ∩ K on the right, it therefore follows that the K-smooth vectors in X^{Hilb} must take values in Y^{∞}: (5.13)(b) X^{Hilb,K-smooth} = {f: K → Y^{∞} | f(kl) = τ(l)^{−1}f(k), and f smooth on K}.

(Implicitly there is a Fréchet topology here, with seminorms like

sup_{k∈K} ν(λ(u) · f);

here u ∈ U(𝔨) is acting by differentiation on the left (this is λ), and ν is a seminorm defining the topology of Y^{∞}.) We will show that X^{Hilb,K-smooth} is precisely the set of smooth vectors of X^{Hilb}. In order to do that, we must show that the left translation action of G on this space (as a subspace of X^{Hilb}) is smooth. This means that we need to describe explicitly the action of the Lie algebra 𝔤 in the compact picture.

So suppose Z ∈ 𝔤. The action of Z is by differentiation on the left:

(5.14)(a)  dπ(Z)f = λ(Z)f  (f ∈ X^{Hilb}).

Now differentiation on the left by an element Z of the Lie algebra (which we have written λ(Z)) is related to differentiation on the right (written ρ(Z)) by the adjoint action:

(5.14)(b)  [λ(Z)f](g) = [ρ(− Ad(g^{−1})Z)f](g).

We are interested in the restriction of f to K. By Proposition 5.3(6), any Lie algebra element W ∈ 𝔤 has a unique decomposition

(5.14)(c)  W = W_{𝔨} + W_{𝔭},  (W_{𝔨} ∈ 𝔨, W_{𝔭} ∈ 𝔩 ∩ 𝔰 + 𝔲 ⊂ 𝔭.

We apply this decomposition to the element − Ad(k^{−1})Z in (5.14)(b), and use the transformation property of f on the right under τ. The conclusion is

(5.14)(d) [dπ(Z)f](k) = [ρ((− Ad(k^{−1})Z))_{𝔨}f](k) + [dτ(Ad(k^{−1})Z))_{𝔭}(f(k))].

This is a kind of first order differential operator on functions on K with values in Y: the first term is a first derivative, and the second (zeroth order) term is just a linear operator on the values of f. We can if we like move the derivative back to the left: (5.14)(e) [dπ(Z)f](k) = [λ(Ad(k)(((− Ad(k^{−1})Z))_{𝔨})f](k) + [dτ(Ad(k^{−1})Z))_{𝔭}(f(k))].

The space of K-smooth vectors in X^{Hilb} was defined by seminorms involving the left action of U(𝔨), which is analogous to constant coefficient differential operators. We have seen in (5.14)(e) that the action of 𝔤 is given by something like variable coefficient differential operators on K. Because the coefficient functions are smooth and bounded on K, this proves that the action of G on the K-smooth vectors of X^{Hilb} is in fact differentiable. That is,

(5.15)(a) X^{∞} = {f: K → Y^{∞} | f(kl) = τ(l)^{−1}f(k), f smooth on K}.

A parallel argument identifies the analytic vectors

(5.15)(b) X^{ω} = {f: K → Y^{ω} | f(kl) = τ(l)^{−1}f(k), f analytic on K}.

Finally, there are the distribution and hyperfunction globalizations to consider. Each of these requires a few more soft analysis remarks. For example, if V is reflexive topological vector space with dual space V^{∗}, then the space of “generalized functions” on a manifold M with values in V is by definition

C^{−∞}(M, V) = [C_{c}^{∞}(M, V^{∗} ⊗ (densities on M))]^{∗},

the topological dual of the space of compactly supported smooth “test densities” on M with values in V^{∗}. (Topologies on the dual space are discussed in section 8; we will be interested most of all in the strong dual topology.) We can then define (5.15)(c) X^{−∞} = {f: K → Y^{−∞} | f(kl) = τ(l)^{−1}f(k), f generalized function on K}.

This is the Casselman-Wallach distribution globalization of X. Similarly, we can make sense of (5.15)(d)

X^{−ω} = {f: K → Y^{−ω} | f(kl) = τ(l)^{−1}f(k), f hyperfunction on K}, Schmid’s maximal globalization of X. We have in the end a concrete version of (4.24):

(5.16)(a)  X^{K} ⊂ X^{ω} ⊂ X^{∞} ⊂ X^{Hilb} ⊂ X^{−∞} ⊂ X^{−ω} ⊂ X^{−K}.

This time each space may be regarded as “functions” on K with values in Y^{−L∩K}, with weakening conditions on the functions: first K-finite, then analytic, then smooth, then L^{2}, then distribution-valued, and so on. (Beginning with X^{−∞}, these are not literally “functions” on K.) It is natural and convenient to write (5.16)(b)

X^{K} = (Ind_{P}^{G})^{K}(Y),  X^{ω} = (Ind_{P}^{G})^{ω}(Y),  X^{∞} = (Ind_{P}^{G})^{∞}(Y),

and so on.

6 Examples of complex homogeneous spaces

In this section we will begin to examine the complex homogeneous spaces for reductive groups that we will use to construct representations. We are going to make extensive use of the structure theory for complex reductive Lie algebras, and for that purpose it is convenient to have at our disposal a complex reductive group. (This means a complex Lie group that is also a reductive group in the sense of Definition 4.6.)

Definition 6.1. A complexification of G is a complex reductive group G_{ℂ}, endowed with a Lie group homomorphism

j: G → G_{ℂ},

subject to the following conditions.

(1) The map j has finite kernel.

(2) The corresponding Lie algebra map

dj: 𝔤_{0} → Lie(G_{ℂ})

identifies 𝔤_{0} as a real form of Lie(G_{ℂ}). More explicitly, this means that

Lie(G_{ℂ}) = dj(𝔤_{0}) ⊕ idj(𝔤_{0}),

with i the complex multiplication on the complex Lie algebra Lie(G_{ℂ}). Using this, we identify Lie(G_{ℂ}) with the complexified Lie algebra 𝔤 henceforth.

(3) The Cartan involutions of G and G_{ℂ} are compatible via the map j.

It is possible to construct a complexification that actually contains the linear reductive group im(π) in Definition 4.6, so that j may be taken to be the composition of an inclusion with the finite covering π. The complexification of G is not unique, but the ambiguity will cause us no problems. If G is the group of real points of a reductive algebraic group, we can of course take for G_{ℂ} the group of complex points; this is perhaps the most important case.

We need notation for the maximal compact subgroup of G_{ℂ}. It is fairly common to refer to this group as U (perhaps in honor of the case of U(n) ⊂ GL(n, ℂ)). Since we will also be discussing parabolic subgroups and their unipotent radicals, the letter U will not be convenient. So we will write

C_{G} = maximal compact subgroup of G_{ℂ}.

Hypothesis (3) in Definition 6.1 guarantees that

K = C_{G} ∩ G.

The complex homogeneous spaces we want will be coverings of (certain) open orbits of G on (certain) complex homogeneous spaces for G_{ℂ}. Here first are the homogeneous spaces for G_{ℂ} that we want.

Definition 6.2. In the setting of (6.1), a partial flag variety for G_{ℂ} is a homogeneous space

X = G_{ℂ}/Q_{ℂ},

with Q_{ℂ} a parabolic subgroup of G_{ℂ} (Definition 5.1). (Recall that this means

𝔮 = Lie(Q_{ℂ}) ⊂ Lie(G_{ℂ}) = 𝔤

is a parabolic subalgebra.) It will sometimes be helpful to write

Q_{ℂ}^{max} = {g ∈ G_{ℂ} | Ad(g)𝔮 = 𝔮},
Q_{ℂ}^{min} = connected subgroup with Lie algebra 𝔮
= identity component of Q_{ℂ}^{max}.

It follows from standard structure theory for complex groups that

Q_{ℂ}^{max} ∩ identity component of G_{ℂ} = Q_{ℂ}^{min}.

Each element of the partial flag variety X^{min} = G_{ℂ}/Q_{ℂ}^{max} may be identified with a parabolic subalgebra of 𝔤, by

gQ_{ℂ}^{max} ↦ Ad(g)(𝔮).

Each element of X^{max} = G_{ℂ}/Q_{ℂ}^{min} may be identified with a pair consisting of a parabolic subalgebra of 𝔤 and a connected component of G_{ℂ}.

Write L_{ℂ} for the Levi factor of Q_{ℂ} defined in Definition 5.1, and

C_{L} = C_{G} ∩ Q_{ℂ}.

Then Proposition 5.3(5) says that

X = C_{G}/C_{L},

a compact homogeneous space for C_{G}.

Theorem 6.3. (Wolf [Wol69]). Suppose G_{ℂ} is a complexification of the real reductive group G, and

X = G_{ℂ}/Q_{ℂ} = C_{G}/C_{L}

is a partial flag variety for G_{ℂ} (Definition 6.2). Then X is a compact complex manifold. The group G acts on X with finitely many orbits; so the finitely many open orbits of G on X are complex homogeneous spaces for G.

Up to covering, the spaces on which we wish to construct representations of G are certain of these open orbits. It remains to say which ones. For that, it is helpful to think about what an arbitrary G orbit on X can look like. We may as well look only at the orbit of the base point eQ_{ℂ}. This G-orbit is

(6.4)(a)  G · (eQ_{ℂ}) ≃ G/H,  (H = G ∩ Q_{ℂ}).

Let us compute the Lie algebra of the isotropy group. Write bar for the complex conjugation defining the real form 𝔤_{0} = Lie(G) of 𝔤:

(6.4)(b)  \overline{A + iB} = A − iB  (A, B ∈ 𝔤_{0}).

Then bar is an involutive automorphism of 𝔤, with fixed points 𝔤_{0}. It follows that \overline{𝔮} is another parabolic subalgebra of 𝔤, and that the complexified Lie algebra 𝔥 of H is

(6.4)(c)  𝔥 = 𝔮 ∩ \overline{𝔮}.

So understanding 𝔥 means understanding the intersection of the two parabolic subalgebras 𝔮 and \overline{𝔮}. The key to analyzing this in general is the fact that the intersection of any two parabolic subalgebras must contain a Cartan subalgebra; this is essentially equivalent to the Bruhat decomposition. (In our case it is even true that the intersection of 𝔮 and \overline{𝔮} must contain the complexification of a Cartan subalgebra of 𝔤_{0}; but we will not use this.) Once one has chosen a Cartan in both parabolics, the analysis of the intersection comes down to combinatorics of sets of roots. There are many interesting possibilities, but we will be looking only at two extreme cases. One extreme is 𝔮 = \overline{𝔮}. In this case H is a real parabolic subgroup of G. This is the case we looked at in section 5. The following definition describes the opposite extreme. (The terminology “nice” is entirely artificial, and not to be taken seriously.)

Definition 6.5. Suppose G is a real reductive group with complexified Lie algebra 𝔤. A parabolic subalgebra 𝔮 ⊂ 𝔤 is called nice if 𝔮∩\overline{𝔮} is a Levi subalgebra 𝔩 of 𝔮. In this case the group

L^{max} = {g ∈ G | Ad(g)𝔮) = 𝔮} = j^{−1}(Q_{ℂ}^{max})

is a real reductive subgroup of G. The G orbit

X_{0}^{min} = G · (eQ_{ℂ}) ≃ G/L^{max} ⊂ G_{ℂ}/Q_{ℂ}^{max} = X^{min}

is open, and therefore inherits a G-invariant complex structure.

Define

L^{min} = identity component of L^{max}.

A real Levi factor for 𝔮 is by definition any subgroup L such that L^{min} ⊂ L ⊂ L^{max}. A measurable complex partial flag variety for G is by definition any homogeneous space X = G/L, endowed with the complex structure pulled back by the covering map

X_{0} = G/L → G/L^{max} = X_{0}^{min} ⊂ X^{min}.

(An explanation of the term “measurable” may be found in [Wol69].)

We say that 𝔮 is very nice if it is nice, and in addition 𝔮 is preserved by the complexified Cartan involution θ. In this case every real Levi factor L is also preserved by θ, so that L ∩ K is a maximal compact subgroup of L.

Obviously the condition of being “nice” is constant on Ad(G)-orbits of parabolic subalgebras. It turns out that every nice parabolic subalgebra is conjugate by Ad(G) to a very nice one; so we may confine our attention to those.

If G is a compact group, then every parabolic subalgebra 𝔮 of 𝔤 is very nice, and measurable complex partial flag varieties for G are exactly the same thing as partial flag varieties for the (canonical) complexification of G. We will begin to look at some noncompact examples in a moment.

Proposition 6.6. Suppose G is a real reductive group, and 𝔮 is a very nice parabolic subalgebra of 𝔤 (Definition 6.5). Let L be a real Levi factor for 𝔮, so that

X_{0} = G/L

is a measurable complex partial flag variety for G. Then L ∩ K is a real Levi factor for the (automatically nice) parabolic subalgebra 𝔮 ∩ 𝔨 of 𝔨, so

Z = K/L ∩ K

is a (compact partial) flag variety for K and for K_{ℂ}. The inclusion

Z = K/L ∩ K ↪ G/L = X_{0}

is holomorphic, and meets every connected component of X_{0} exactly once.

We turn now to some examples for classical groups. Recall from section 2 that a classical complex group G_{ℂ} is a group of linear transformations of ℂ^{n}, perhaps preserving some standard symmetric or skew-symmetric bilinear form. A real form G is the subgroup defined by some kind of reality condition on the matrices. Just as we saw in section 2 for complete flag varieties, a partial flag variety for G_{ℂ} will be a space of partial flags in ℂ^{n}, subject to some conditions involving the bilinear form defining the group. We need to analyze the orbits of G on such flags, which is usually a matter of linear algebra.

Here is an example. Suppose G = GL(n, ℝ) and G_{ℂ} = GL(n, ℂ). A partial flag variety for G_{ℂ} is determined by a collection m of integers

(6.7)(a)  0 = m_{0} < m_{1} < ⋯ < m_{r} = n.

The variety X_{m} is the collection of all possible partial flags

(6.7)(b) X_{m} = {F = (F_{j}) | 0 = F_{0} ⊂ F_{1} ⊂ ⋯ ⊂ F_{r} = ℂ^{n}, dim F_{j} = m_{j}}.

Here each F_{j} is a linear subspace of ℂ^{n}. There is a standard flag F^{std}, with

(6.7)(c)  F_{j}^{std} = ℂ^{m_{j}} ⊂ ℂ^{n},

embedded in the first m_{j} coordinates. The group G_{ℂ} acts transitively on X_{m}; the isotropy group at the base point F^{std} consists of block upper triangular matrices

(6.7)(d)  P_{r}^{std} = \{(\begin{matrix}A_{1} & ∗ & ∗ \\ 0 & ... & ∗ \\ 0 & 0 & A_{r} \end{matrix}) | A_{j} ∈ GL((m_{j} − m_{j−1}), ℂ)\} .

So how does one understand the orbits of GL(n, ℝ) on this space? If F is a flag of type m, then so is \overline{F}. (If W is a subspace of ℂ^{n}, then \overline{W} consists of all the complex conjugates of vectors in W.) The collection of dimensions

(6.7)(e)  dim_{ℂ}(F_{j} ∩ \overline{F}_{k})

is obviously constant on GL(n, ℝ) orbits in X_{m}; and it is not very hard to show that these invariants specify the GL(n, ℝ) orbits completely. It is also not so difficult to describe exactly what sets of dimensions are possible. Roughly speaking, the open orbits of GL(n, ℝ) should be “generic,” and so should be characterized by having all the dimensions in (6.7)(e) as small as possible. It is an excellent exercise for the reader to work this out in detail for complete flags; the conclusion in that case is that the GL(n, ℝ) orbits correspond to elements of order 2 in the symmetric group S_{n}. (For complete flags in general split real groups G, there is a surjective map from G orbits to elements of order 2 in the Weyl group, but the map can have non-trivial fibers.)

We will analyze instead the much simpler case

(6.8)(a)  m = (0, m, n).

In this case

(6.8)(b)  X_{m} = {F ⊂ ℂ^{n} | dim F = m}

is the Grassmann variety of m-planes in ℂ^{n}. The unique invariant of a GL(n, ℝ) orbit on X_{m} is the integer

(6.8)(c)  r = dim(F ∩ \overline{F}) ≤ m.

The integer r is a measure of the “reality” of F: F is the complexification of a real subspace if and only if r = m. The GL(n, ℝ) orbit corresponding to r = m is the real Grassmann variety of m-planes in ℝ^{n}, and the isotropy groups are examples of the real parabolic subgroups studied in section 5.

We are interested now in the opposite case, when r is as small as possible. How small is that? The constraint comes from the fact that dim(F + \overline{F}) ≤ n. The sum has dimension 2m − r, so we find 2m − r ≤ n, or equivalently r ≥ 2m − n. The conclusion is that possible values of r are

(6.8)(d)  min(0, 2m − n) ≤ r ≤ m.

The unique open orbit of GL(n, ℝ) on X_{m} corresponds to the smallest possible value of r; it is

(6.8)(e) X_{m,0} = \{F ⊂ ℂ^{n} | dim F = m, dim(F ∩ \overline{F}) = min(0, 2m − n)\},

a complex homogeneous space for GL(n, ℝ).

For definiteness, let us now concentrate on the case

(6.8)(f)  2m ≥ n.

In this case we are looking at subspaces F ⊂ ℂ^{n} such that

(6.8)(g)  0 ⊂ \underbrace{F ∩ \overline{F}}_{dimension 2m − n} ⊂ \underbrace{F}_{dimension m} ⊂ F + \overline{F} = ℂ^{n}.

Let us now look at the corresponding parabolic subalgebra 𝔮, the stabilizer of F. We can choose a basis of ℂ^{n} so that

(6.8)(h)  F ∩ \overline{F} = span of middle 2m − n basis vectors
F = span of first m basis vectors
\overline{F} = span of last m basis vectors

In these coordinates, we compute

𝔮 = \{(\begin{matrix}∗ & ∗ & ∗ \\ ∗ & ∗ & ∗ \\ 0 & 0 & ∗ \end{matrix})\} , \overline{𝔮} = \{(\begin{matrix}∗ & 0 & 0 \\ ∗ & ∗ & ∗ \\ ∗ & ∗ & ∗ \end{matrix})\} .

Here the blocks correspond to the first n − m, middle 2m − n, and last n − m coordinates. The intersection of these two parabolic subalgebras is

𝔮 ∩ \overline{𝔮} = \{(\begin{matrix}∗ & 0 & 0 \\ ∗ & ∗ & ∗ \\ 0 & 0 & ∗ \end{matrix})\} .

The nil radical of this Lie algebra is

\{(\begin{matrix}0 & 0 & 0 \\ ∗ & 0 & ∗ \\ 0 & 0 & 0 \end{matrix})\} ,

which has dimension 2(n − m)(2m − n). It follows that 𝔮 ∩ \overline{𝔮} is not reductive when n/2 < m < n: in these cases, the complex homogeneous space X_{m,0} is not “measurable” in the sense of Definition 6.5. (The same conclusion applies to the cases 0 < m < n/2.)

We now look more closely at the case n = 2m. Recall that a complex structure on a real vector space V is a linear map J such that J^{2} = −I.

Proposition 6.9. Suppose n = 2m is a positive even integer. Define

X = X_{m,0} = \{F ⊂ ℂ^{2m} | dim F = m, F ∩ \overline{F} = 0\}.

Then X is a measurable complex partial flag variety for GL(2m, ℝ). Its points may be identified with complex structures on ℝ^{2m}; the identification sends a complex structure J to the +i-eigenspace of J acting on (ℝ^{2m})_{ℂ} = ℂ^{2m}.

The isotropy group at a subspace F corresponding to the complex structure J_{F} consists of all linear automorphisms of ℝ^{2m} commuting with the complex structure J_{F}; that is, of complex-linear automorphisms of the corresponding m-dimensional complex vector space. In particular, if we choose as a base point of X_{m,0} the standard complex structure, then the isotropy group is

L = GL(m, ℂ) ⊂ GL(2m, ℝ).

This base point is “very nice” in the sense of Definition 6.5. The corresponding O(n) orbit is

\underbrace{Z = O(2m)/U(m)}_{dim_{ℂ}=(m^{2}−m)/2} ⊂ \underbrace{GL(2m, ℝ)/GL(m, ℂ) = X}_{dim_{ℂ}=m^{2}} .

This compact subvariety consists of all orthogonal complex structures on ℝ^{2m} (those for which multiplication by i preserves length).

We conclude this section with an easier example: the case of U(p, q). We begin with non-negative integers p and q, and write n = p + q. There is a standard Hermitian form

(6.10)(a)  ⟨v, w⟩_{p,q} = ∑_{i=1}^{p} v_{i}\overline{w}_{i} − ∑_{j=1}^{q} v_{p+j}\overline{w}_{p+j}

of signature (p, q) on ℂ^{n}. The indefinite unitary group of signature (p, q) is (6.10)(b)

U(p, q) = {g ∈ GL(n, ℂ) | ⟨g · v, g · w⟩_{p,q} = ⟨v, w⟩_{p,q} (v, w ∈ ℂ^{n})}

Just as in the case of U(n), it is easy to check that every n×n complex matrix Z can be written uniquely as Z = A + iB, with A and B in Lie(U(p, q)). It follows that GL(n, ℂ) is a complexification of G. Let us fix a partial flag variety X_{m} as in (6.7), and try to understand the orbits of U(p, q) on X_{m}. Consider a flag F = (F_{j}) in X_{m}. The orthogonal complement F_{k}^{⊥} (with respect to the form ⟨·.·⟩_{p,q}) is a subspace of dimension n − m_{k}; we therefore get a partial flag consisting of the subspaces F_{j} ∩ F_{k}^{⊥} inside F_{j}. The dimensions of these subspaces are invariants of the U(p, q) orbit of F. We are interested in open orbits, where the dimensions are as small as possible. The minimum possible dimensions are

(6.10)(c)  dim F_{j} ∩ F_{k}^{⊥} = \begin{cases}m_{j} − m_{k}, & k ≤ j \\ 0, & j ≤ k. \end{cases}

Looking in particular at the case k = j, we see that on an open orbit, F_{j}∩F_{j}^{⊥} = 0. This means that the restriction of ⟨·.·⟩_{p,q} to F_{j} will be a non-degenerate Hermitian form, which will therefore have some signature (p(F_{j}), q(F_{j})) = (p_{j}, q_{j}). These non-negative integers must satisfy the conditions (6.10)(d)

p_{j}+q_{j} = m_{j},  0 = p_{0} ≤ p_{1} ≤ ⋯ ≤ p_{r} = p,  0 = q_{0} ≤ q_{1} ≤ ⋯ ≤ q_{r} = q.

These sequences (p, q) are invariants of the U(p, q) orbit of F. Conversely, if F′ is any other flag giving rise to the same sequence of signatures, then it is easy to find an element of U(p, q) carrying F to F′. The following proposition summarizes this discussion, and some easy calculations.

Proposition 6.11. Suppose X_{m} is a partial flag variety for GL(n, ℂ) as in (6.7). The open orbits of U(p, q) on X_{m} are in one-to-one correspondence with pairs of sequences (p, q) as in (6.10)(d). Write X_{p,q} for the corresponding orbit. Each of these orbits is measurable (Definition 6.5). The corresponding real Levi factor (Definition 6.5) is isomorphic to

∏_{j=1}^{r} U(p_{j} − p_{j−1}, q_{j} − q_{j−1}).

The orbit of K = U(p) × U(q) through a very nice point is isomorphic to

[U(p)/ ∏_{j=1}^{r} U(p_{j} − p_{j−1})] × [U(q)/ ∏_{j=1}^{r} U(q_{j} − q_{j−1})] ≃ X_{p} × X_{q}.

This is a compact complex subvariety of X_{p,q}.

7 Dolbeault cohomology and maximal globalizations

The central idea in these notes is this: we want to construct representations of a real reductive group G by starting with a measurable complex flag variety X = G/L (Definition 6.5) and using G-equivariant holomorphic vector bundles on X. For G compact connected, the Borel-Weil theorem (Theorem 2.3) says that all irreducible representations of G arise in this way, as spaces of holomorphic sections of holomorphic line bundles. In order to get some feeling for what to expect about noncompact groups, we look first at the example of U(1, 1). In the language of Proposition 6.11, let us take r = 2 and consider the complete flag variety for GL(2, ℂ), corresponding to

(7.1)(a)  m = (0, 1, 2).

Explicitly, X_{m} is just the projective space ℂℙ^{1} of lines in ℂ^{2}. We identify

(7.1)(b)  (ℂ ∪ ∞) ≃ X_{m}, z ↦ line through (\begin{matrix}1 \\ z \end{matrix})

We consider the open U(1, 1) orbit X_{p,q} with

p = (0, 1, 1), q = (0, 0, 1).

Explicitly, these are the lines in ℂ^{2} on which the Hermitian form ⟨·.·⟩_{1,1} is strictly positive. Because

⟨(\begin{matrix}1 \\ z \end{matrix}) , (\begin{matrix}1 \\ z \end{matrix})⟩_{1,1} = 1 − |z|^{2},

it follows that the identification of (7.1)(b) gives

(7.1)(c)  X_{p,q} ≃ {z ∈ ℂ | |z| < 1},

the unit disc. The action of U(1, 1) on the disc is by linear fractional transformations as in (3.2); the reason is

(\begin{matrix}α & β \\ \overline{β} & \overline{α} \end{matrix}) (\begin{matrix}1 \\ z \end{matrix}) = (\begin{matrix}βz + α \\ \overline{α}z + β \end{matrix}) = c · (\begin{matrix}1 \\ (\overline{α}z + β)/(βz + α) \end{matrix}) .

The standard base point is the origin z = 0, where the isotropy group is U(1) × U(1). It follows that equivariant holomorphic line bundles on X_{p,q} are in one-to-one correspondence with characters

(7.1)(d)  μ = (m_{1}, m_{2}) ∈ (U(1) × U(1)) ≃ ℤ^{2}.

Write L_{μ} for the holomorphic line bundle corresponding to μ. Because μ extends to a holomorphic character of the group of complex upper triangular matrices, L_{μ} extends to a GL(2, ℂ)-equivariant holomorphic line bundle on the Riemann sphere X_{m}.

The most straightforward analogy with the Borel-Weil theorem suggests defining

(7.1)(e)  H_{μ} = holomorphic sections of L_{μ}.

If we endow this space with the topology of uniform convergence on compact sets, then it is a complete topological vector space, and the action π_{μ} of G = U(1, 1) by left translation is continuous.

Proposition 7.2. The representation π_{μ} of U(1, 1) is always infinite-dimen-sional. It is irreducible unless μ is antidominant; that is, unless m_{1} ≤ m_{2}. If μ is antidominant, there is exactly one proper closed G-invariant subspace: the (m_{2} − m_{1} + 1)-dimensional space of sections extending holomorphically to the entire Riemann sphere X_{m}.

Here are some hints about proofs. Any holomorphic line bundle on the disc is holomorphically (although not equivariantly!) trivial, so the space of sections may be identified with holomorphic functions on the disc; this is certainly infinite-dimensional. For μ anti-dominant, Theorem 2.3 provides a finite-dimensional subspace of sections extending to the Riemann sphere. The dimension calculation is a standard fact about U(2), and the invariance of this subspace is clear.

For the remaining assertions, examining Taylor series expansions shows that every U(1) × U(1) weight of H_{μ} is of the form (m_{1} + k, m_{2} − k), with k a non-negative integer; and that each of these weights has multiplicity one. Now one can apply facts about Verma modules for 𝔤𝔩(2) to finish.

Proposition 7.2 is a bit discouraging with respect to the possibility of extending Theorem 2.3 to noncompact groups. The case of U(2, 1) is even worse. Let us look at

(7.3)(a)  m = (0, 1, 2, 3),  p = (0, 1, 1, 2),  q = (0, 0, 1, 1).

Then X_{p,q} consists of complete flags F with the property that the Hermitian form is positive on F_{1} and of signature (1, 1) on F_{2}. The isotropy group at the standard base point is U(1)^{3}, and its characters are given by triples

(7.3)(b)  μ = (m_{1}, m_{2}, m_{3}) ∈ ℤ^{3}

Write L_{μ} for the corresponding equivariant holomorphic line bundle on X_{p,q} (which automatically extends to be GL(3, ℂ)-equivariant on X_{m}) and (π_{μ}, H_{μ}) for the representation of U(2, 1) on its space of holomorphic sections.

Proposition 7.4. In the setting of (7.3), the representation π_{μ} of U(2, 1) is zero unless μ is antidominant; that is, unless m_{1} ≤ m_{2} ≤ m_{3}. In that case it is finite-dimensional, and all holomorphic sections extend to the full flag variety X_{m}.

Again one can use Taylor series to relate the Harish-Chandra module of H_{μ} to a highest weight module. What one needs to know is that if V is an irreducible highest weight module for 𝔤𝔩(3), and the non-simple root 𝔤𝔩(2) subalgebra acts in a locally finite way, then V is finite-dimensional. Again we omit the details.

The behavior in Proposition 7.4 is typical. Holomorphic sections of vector bundles on measurable complex partial flag varieties rarely produce anything except finite-dimensional representations of G. One way to understand this is that the varieties fail to be Stein, so we should not expect to understand them looking only at holomorphic sections: we must also consider “higher cohomology.” We begin with a brief review of Dolbeault cohomology.

Suppose X is a complex manifold, with complexified tangent bundle T_{ℂ}X. The complex structure on X provides a decomposition

(7.5)(a)  T_{ℂ}X = T^{1,0} ⊕ T^{0,1}

into holomorphic and antiholomorphic tangent vectors. These may be understood as the +i and −i eigenspaces of the complex structure map J (defining “multiplication by i” in the real tangent space.) The two subspaces are interchanged by complex conjugation. The space T^{0,1} consists of the tangent vectors annihilating holomorphic functions: the Cauchy-Riemann equations are in T^{0,1}.

There is a terminological dangerous bend here. One might think that a smooth section of T^{1,0} should be called a “holomorphic vector field,” but in fact this terminology should be reserved only for holomorphic sections (once those are defined). We will call a smooth section a vector field of type (1, 0). On ℂ, the vector field

x\frac{∂}{∂z} = \frac{x}{2} (\frac{∂}{∂x} − i\frac{∂}{∂y} )

is of type (1, 0), but is not holomorphic. If we replace the coefficient function x by 1 (or by any holomorphic function), we get a holomorphic vector field.

Write

(7.5)(b)  A^{m} = complex-valued differential forms of degree m on X,

= ∑_{p+q=m} A^{p,q}.

Here A^{p,q} consists of differential forms that vanish on sets of p′ type (1, 0) vector fields and q′ type (0, 1) vector fields unless p′ = p and q′ = q. The de Rham differential

d: A^{m} → A^{m+1}

satisfies

(7.5)(c)  d(A^{p,q}) ⊂ A^{p+1,q} ⊕ A^{p,q+1}.

This follows by inspection of the formula

dω(Y_{0}, …, Y_{m}) = ∑_{i=0}^{m}(−1)^{i}Y_{i} · ω(Y_{0}, …, Ŷ_{i}, …, Y_{m})
+ ∑_{i<j}(−1)^{i+j}ω([Y_{i}, Y_{j}], Y_{0}, …, Ŷ_{i}, …, Ŷ_{j}, …, Y_{m}),

and the fact the Lie bracket of two type (1, 0) (respectively type (0, 1)) vector fields is type (1, 0) (respectively type (0, 1)). Now the decomposition in (7.5)(c) allows us to write d = ∂ + \overline{∂}, with

(7.5)(d)  ∂: A^{p,q} → A^{p+1,q},  \overline{∂}: A^{p,q} → A^{p,q+1}

The fact that d^{2} = 0 implies that

(7.5)(e)  ∂^{2} = \overline{∂}^{2} = 0,  ∂\overline{∂} + \overline{∂}∂ = 0.

If we try to write explicit formulas for ∂ and \overline{∂}, the only difficulty arises from terms involving [Y, \overline{Z}], with Y a vector field of type (1, 0) and \overline{Z} of type (0, 1). The bracket is again a vector field, so it decomposes as

[Y, \overline{Z}] = [Y, \overline{Z}]_{1,0} + [Y, \overline{Z}]_{0,1}.

The first summand will appear in a formula for ∂, and the second in a formula for \overline{∂}. One way to avoid this unpleasantness is to notice that if Y is actually a holomorphic vector field, then the first summand [Y, \overline{Z}]_{1,0} is automatically zero; one can take this as a definition of a holomorphic vector field on X. If \overline{Z} is antiholomorphic, then the second summand vanishes.

Here are the formulas that emerge.

Proposition 7.6. Suppose X is a complex manifold, ω ∈ A^{p,q} is a complexvalued differential form of type (p, q) (cf. (7.5)), (Y_{0}, … Y_{p}) are holomorphic vector fields, and (\overline{Z}_{0}, …, \overline{Z}_{q}) are antiholomorphic vector fields. Then

∂ω(Y_{0}, …, Y_{p}, \overline{Z}) = ∑_{i=0}^{p}(−1)^{i}Y_{i} · ω(Y_{0}, …, Ŷ_{i}, …, Y_{p}, \overline{Z})

+ ∑_{i<j}(−1)^{i+j}ω([Y_{i}, Y_{j}], Y_{0}, …, Ŷ_{i}, …, Ŷ_{j}, …, Y_{p}, \overline{Z}).

In this formula,

\overline{Z} = \overline{Z}_{1}, …, \overline{Z}_{q}.

Similarly,

(−1)^{p}\overline{∂}ω(Y, \overline{Z}_{1}, …, \overline{Z}_{q}) = ∑_{i=0}^{q}(−1)^{i}\overline{Z}_{i} · ω(Y, \overline{Z}_{1}, …, ˆ \overline{Z}_{i}, …, \overline{Z}_{q})

+ ∑_{i<j}(−1)^{i+j}ω(Y, [\overline{Z}_{i}, \overline{Z}_{j}], \overline{Z}_{0}, …, ˆ \overline{Z}_{i}, …, ˆ \overline{Z}_{j}, …, \overline{Z}_{q}).

Here

Y = Y_{1}, …, Y_{p}.

Definition 7.7. Suppose X is a complex manifold. The (p, q)-Dolbeault cohomology of X is by definition

H^{p,q}(X) = (kernel of \overline{∂} on A^{p,q})/(image of \overline{∂} from A^{p,q−1}).

This makes sense because of (7.5)(e). The space A^{p,0} consists of smooth sections of the bundle Ω^{p} of holomorphic p-forms on X; and it is easy to check that

H^{p,0} = kernel of \overline{∂} on A^{p,0}

= holomorphic p-forms on X.

In particular, H^{0,0} = holomorphic functions on X.

Suppose now that V is a holomorphic vector bundle on X. One cannot apply the de Rham differential to forms with values in a bundle, because there is no canonical way to differentiate sections of a bundle by a vector field. However, we can apply type (0, 1) vector fields canonically to smooth sections of a holomorphic vector bundle. Here is how this looks locally. Suppose \overline{Z} is a type (0, 1) vector field (near x ∈ X), and v is a smooth section of V (defined near x). Choose a basis (v_{1}, …, v_{d}) of holomorphic sections of V (still near x) and write

(7.8)(a)  v = ∑ g_{i}v_{i},

with g_{i} smooth on X (near x). Finally, define

(7.8)(b)  \overline{Z} · v = ∑(\overline{Z} · g_{i})v_{i}.

Why is this well-defined? If we choose a different basis (v′_{1}, …, v′_{d}), then it differs from the first by an invertible matrix B_{ij} of holomorphic functions on X (near x):

v_{i} = ∑_{j} B_{ij}v′_{j}.

If we expand v in the new basis, the coefficient functions g′_{i} are

g′_{j} = ∑_{i} g_{i}B_{ij}.

Applying the vector field \overline{Z} and using the Leibnitz rule gives

\overline{Z} · g′_{j} = ∑_{i}[(\overline{Z} · g_{i})B_{ij} + g_{i}(\overline{Z} · B_{ij})].

The second terms all vanish, because \overline{Z} is a type (0, 1) vector field and B_{ij} is a holomorphic function. What remains says (after multiplying by v′_{j} and summing over j) that

∑_{j}(\overline{Z} · g′_{j})v′_{j} = ∑_{i}(\overline{Z} · g_{i})v_{i};

that is, that our definition of \overline{Z} · v is well-defined.

What follows from (7.8) is that the Dolbeault \overline{∂} operator can be defined on (p, q) forms with values in a holomorphic vector bundle on X. Here is an explicit account. Write

(7.9)(a)  A^{p,q}(V) = smooth (p, q) forms on X with values in V.

An element of this space attaches to p type (1, 0) vector fields and q type (0, 1) vector fields a smooth section of V. The Dolbeault operator

(7.9)(b)  \overline{∂}: A^{p,q}(V) → A^{p,q+1}(V)

is defined by the formula in Proposition 7.6, with the terms of the form

(7.9)(c)  \overline{Z}_{i} · (smooth section of V)

defined by (7.8). If we need to be more explicit, we may write this operator as \overline{∂}^{p,q}(V). Just as in (7.5)(e), we have

\overline{∂}^{2} = 0.

Definition 7.10. Suppose X is a complex manifold, and V is a holomorphic vector bundle on X. The (p, q)-Dolbeault cohomology of X with coefficients in V is by definition

H^{p,q}(X, V) = (kernel of \overline{∂} on A^{p,q}(V))/(image of \overline{∂} from A^{p,q−1}(V)).

This makes sense because of (7.5)(e). The space A^{0,0}(V) consists of smooth sections of V, and

H^{0,0}(X, V) = kernel of \overline{∂} on A^{0,0}(V)

= holomorphic p-forms on X.

In particular, H^{0,0} = holomorphic sections of V.

As a first application, we can understand the dependence of Dolbeault cohomology on p. Recall that Ω^{p} is the bundle of holomorphic p-forms on X. It is easy to see that

A^{p,q}(V) ≃ A^{0,q}(Ω^{p} ⊗ V),

and that this isomorphism respects the \overline{∂} operators (up to a factor of (−1)^{p}). It follows that

H^{p,q}(X, V) ≃ H^{0,q}(X, Ω^{p} ⊗ V).

Here is the central fact about Dolbeault cohomology.

Theorem 7.11. (Dolbeault, Serre [Ser55]). Suppose V is a holomorphic vector bundle on a complex manifold X. Write O_{V} for the sheaf of germs of holomorphic sections of V. Then there is a canonical isomorphism

H^{0,q}(X, V) ≃ H^{q}(X, O_{V}).

On the right is the Čech cohomology of X with coefficients in the sheaf O_{V}.

It may be helpful to see how Dolbeault cohomology looks on a homogeneous space. For this we can allow G to be any Lie group and L any closed subgroup. Write

(7.12)(a)  X = G/L,  𝔤 = Lie(G)_{ℂ} ⊃ Lie(L)_{ℂ} = 𝔩.

A G-invariant complex structure on G/L corresponds to a complex Lie subalgebra 𝔮 ⊂ 𝔤 satisfying

(7.12)(b)  Ad(L)𝔮 = 𝔮,  𝔮 + \overline{𝔮} = 𝔤,  𝔮 ∩ \overline{𝔮} = 𝔩.

In terms of the decomposition in (7.5)(a), 𝔮 corresponds to the antiholomorphic tangent vectors:

(7.12)(c)  T_{eL}^{0,1}(G/L) = 𝔮/𝔩,  T_{eL}^{1,0}(G/L) = \overline{𝔮}/𝔩.

(All of this is described for example in [TW71] or in [Vog87], Proposition 1.19.) A complex-valued smooth vector field on G/L may be identified with a smooth function

(7.12)(d)  Y : G → 𝔤/𝔩,  Y (gl) = Ad(l)^{−1}Y (g) (l ∈ L, g ∈ G)

(cf. (5.7)(a)). In this identification, vector fields of type (0, 1) are those taking values in 𝔮/𝔩. Smooth functions on G/L correspond to smooth functions

f: G → ℂ, f(gl) = f(g).

The vector field Y acts on f by

(7.12)(e)  (Y · f)(g) = [ρ(Y (g)) · f](g).

That is, we differentiate f on the right by the Lie algebra element Y (g). (Of course Y (g) is only a coset of 𝔩, but that is harmless since f is invariant on the right by L. The condition on Y in (7.12)(d) forces the new function Y · f also to be right invariant by L.)

From the identification in (7.12)(d), it is not hard to deduce an identification of the smooth m-forms on G/L:

(7.13)(a)  A^{m}(G/L) ≃ Hom_{L} (⋀^{m}(𝔤/𝔩), C^{∞}(G)) .

Here L acts on the exterior algebra by Ad, and on the smooth functions by right translation. The decomposition

𝔤/𝔩 = \overline{𝔮}/𝔩 ⊕ 𝔮/𝔩

(which follows from (7.12)(b)) gives

⋀^{m}(𝔤/𝔩) = ∑_{p+q=m} ⋀^{p}(\overline{𝔮}/𝔩) ⊗ ⋀^{q}(𝔮/𝔩),

and a corresponding decomposition of the m-forms. The pieces are exactly the (p, q) forms of (7.5)(b):

(7.13)(b)  A^{p,q}(G/L) ≃ Hom_{L} (⋀^{p}(\overline{𝔮}/𝔩) ⊗ ⋀^{q}(𝔮/𝔩), C^{∞}(G)) .

Writing formulas for the operators ∂ and \overline{∂} in this setting is slightly unpleasant, because the description of vector fields in (7.12)(d) does not obviously hand us any holomorphic or antiholomorphic vector fields. We will sweep this problem under the rug for the moment, by not writing formulas yet.

A smooth equivariant vector bundle V on G/L is the same thing as a smooth representation (τ, V) of L; the correspondence is

(7.13)(c)  V ↦ (fiber of V at eL),  V ↦ G ×_{L} V.

The space of smooth sections of V may be identified with smooth functions

(7.13)(d)  f: G → V,  f(gl) = τ(l)^{−1}f(g) (l ∈ L, g ∈ G).

This description makes sense for infinite-dimensional vector bundles. What does it mean for V to be a holomorphic vector bundle? Certainly this ought to amount to imposing some additional structure on the representation (τ, V) of L. Here is the appropriate definition, taken from [TW71].

Definition 7.14. Suppose L is a Lie group with complexified Lie algebra 𝔩. Assume that 𝔮 is a complex Lie algebra containing 𝔩, and the adjoint action of L extends to

Ad: L → Aut(𝔮),

with differential the Lie bracket of 𝔩 on 𝔮. A (𝔮, L)-representation is a complete locally convex vector space V, endowed with a smooth representation τ of L and a continuous Lie algebra action (written just with a dot). These are required to satisfy

(1) The 𝔮 action extends the differential of τ: if Y ∈ 𝔩 and v ∈ V, then

dτ(Y)v = Y · v.

(2) For l ∈ L, Z ∈ 𝔮, and v ∈ V, we have

τ(l)(Z · v) = (Ad(l)Z) · (τ(l)v).

For l ∈ L_{0}, condition (2) is a consequence of condition (1).

Condition (2) can also be formulated as requiring that the action map

𝔮 × V → V, (Z, v) → Z · v

is L-equivariant. This entire definition is formally very close to that of a (𝔤, K)-module in Definition 4.14, except that we have no finiteness assumption on the L representation.

Proposition 7.15. Suppose G/L is a homogeneous space for Lie groups, and that 𝔮 defines an invariant complex structure (cf. (7.12)). Then passage to the fiber at eL defines a bijective correspondence from G-equivariant holomorphic vector bundles V on G/L, to (𝔮, L)-representations (τ, V) (Definition 7.14). Suppose \overline{U} is an open subset of G/L, and U its inverse image in G. Then holomorphic sections of V on \overline{U} correspond to smooth functions

f: U → V

satisfying the transformation law

f(gl) = τ(l)^{−1}f(g) (l ∈ L, g ∈ U)

and the differential equations

(ρ(Z)f)(g) = Z · (f(g)) (Z ∈ 𝔮, g ∈ U).

Here ρ is the right regular representation of the Lie algebra on smooth functions.

To be more honest and precise: this result is certainly true for finitedimensional bundles (where it is proved in [TW71]). I have not thought carefully about the appropriate abstract definition of infinite-dimensional holomorphic vector bundles; but that definition needs to be arranged so that Proposition 7.15 is true.

The transformation law in Proposition 7.15 is just what describes a smooth section of V (cf. (7.13)(d)). For Z ∈ 𝔩, the differential equation is a consequence of the transformation law. The differential equations for other elements of 𝔮 are the Cauchy-Riemann equations.

Lie algebra cohomology was invented for the purpose of studying de Rham cohomology of homogeneous spaces. It is therefore not entirely surprising that Dolbeault cohomology (which we described in (7.5) as built from de Rham cohomology)) is also related to Lie algebra cohomology. To state the result, we need one more definition.

Definition 7.16. Suppose V is a (𝔮, L)-representation (Definition 7.14). The complex defining Lie algebra cohomology is

C^{m}(𝔮; V) = Hom (⋀^{m} 𝔮, V ) .

The differential is

dω(Z_{0}, …, Z_{m}) = ∑_{i=0}^{m}(−1)^{i}Z_{i} · ω(Z_{0}, …, Ẑ_{i}, …, Z_{m})
+ ∑_{i<j}(−1)^{i+j}ω(Z_{0}, …, Ẑ_{i}, …, Ẑ_{j}, …, Z_{m})

The Lie algebra cohomology of 𝔮 with coefficients in V is by definition

H^{m}(𝔮; V) = (kernel of d on C^{m}(𝔮; V))/(image of d from C^{m−1}(𝔮; V)).

We now consider the subspace

C^{m}(𝔮, L; V) = Hom_{L} (⋀^{m} 𝔮/𝔩, V ) .

We are imposing two conditions: that ω vanish on the ideal generated by 𝔩 in the exterior algebra, and that the linear map ω respect the action of L (by Ad on the domain and τ on the range). The differential d respects the second condition; and in the presence of the second condition, it respects the first as well. We can therefore define the relative Lie algebra cohomology of 𝔮 with coefficients in V as

H^{m}(𝔮, L; V)=(kernel of d on C^{m}(𝔮, L; V))/(image of d from C^{m−1}(𝔮, L; V)).

This cohomology is most often considered in the case when L is compact. One reason is that when L is not compact, taking L invariants (as in Hom_{L} in the definition of the relative complex) is not an exact functor, and should really only be considered along with its derived functors. This difficulty will come back to haunt us in section 9, but for now we ignore it.

Here now is Kostant’s description of Dolbeault cohomology for equivariant bundles.

Proposition 7.17. (Kostant [Kos61], (6.3.5); see also [Won99], section 2). Suppose G/L is a homogeneous space for Lie groups, and that 𝔮 defines an invariant complex structure (cf.(7.12)). Suppose (τ, V) is a (𝔮, L)-representation (Definition 7.14), and V the corresponding G-equivariant holomorphic vector bundle on G/L (Proposition 7.15). We regard C^{∞}(G, V) as a (𝔮, L)- representation by the “tensor product” of the right regular action on functions with the action on V. Explicitly, the representation τ_{r} of L is

[τ_{r}(l)f](g) = τ(l)f(gl^{−1}).

The action of Z ∈ 𝔮 is

[Z · f](g) = [ρ(Z)f](g) + Z · (f(g)).

The first term is the right regular action of 𝔤 on functions, and the second is the action of 𝔮 on V.

Then the space of smooth (0, q) forms on G/L with values in V is

A^{0,q}(G/L, V) ≃ Hom_{L} (⋀^{q}(𝔮/𝔩), C^{∞}(G, V)) = C^{q}(𝔮, L; C^{∞}(G, V)),

(Definition 7.16). This identifies the Dolbeault differential \overline{∂} with the relative Lie algebra cohomology differential d, and so

H^{0,q}(G/L, V) ≃ H^{q}(𝔮, L; C^{∞}(G, V)).

To talk about (p, q) Dolbeault cohomology, we can use the fact mentioned before Theorem 7.11. This involves the bundle Ω^{p} of holomorphic p-forms on X; so we need to understand Ω^{p} in the case of X = G/L. This is an equivariant vector bundle, so it corresponds to a certain representation of L: the pth exterior power of the holomorphic cotangent space (T_{eL}^{1,0})^{∗}. According to (7.12)(c), this is

⋀^{p}(\overline{𝔮}/𝔩).

To specify the holomorphic structure, we need a representation of 𝔮 on this space, extending the adjoint action of 𝔩. This we can get from the natural isomorphism

\overline{𝔮}/𝔩 ≃ 𝔤/𝔮,

which is a consequence of (7.12)(b). That is, in the correspondence of Proposition 7.15,

(7.18)(a)  Ω^{p} ↔ ⋀^{p}(𝔤/𝔮)^{∗},

with the obvious structure of (𝔮, L)-representation on the right. A consequence of this fact, Proposition 7.17, and the fact before Theorem 7.11 is

(7.18)(b)  H^{p,q}(G/L, V) ≃ H^{q}(𝔮, L; C^{∞}(G, ⋀^{p}(𝔤/𝔮)^{∗} ⊗ V)).

With Dolbeault cohomology in our tool box, we can now make the idea at the beginning of this section a little more precise. Beginning with a measurable complex flag variety X = G/L and a G-equivariant holomorphic vector bundle V over X, we want to consider representations of G on Dolbeault cohomology spaces H^{p,q}(X, V). First of all, notice that G acts by translation on the forms A^{p,q}(V) (cf. (7.9)), and that this action respects \overline{∂}. It follows that we get a linear action of G on the Dolbeault cohomology. To have a representation, of course we need a topological vector space structure. The space of V-valued differential forms (for any smooth vector bundle on any smooth manifold) naturally has such a structure; in our case, the forms are described in Proposition 7.17 as a closed subspace of the (complete locally convex) space C^{∞}(G, V) tensored with a finite-dimensional space. (This shows in particular that if V is Fréchet, then so is A^{p,q}(V).) With respect to this topology, any differential operator is continuous; so in particular the \overline{∂} operator is continuous, and its kernel is a closed subspace of A^{p,q}(V). It is also clear that the action of G on A^{p,q}(V) is continuous.

We can impose on H^{p,q}(X, V) the quotient topology coming from the kernel of \overline{∂}: a subset of the cohomology is open (or closed) if and only if its preimage in the kernel of \overline{∂} is open (or closed). The action of G is clearly continuous for this quotient topology. The difficulty is that the closure of the point 0 in the quotient topology is equal to

(closure of the image of \overline{∂})/(image of \overline{∂}).

In particular, the topology is Hausdorff only if the image of \overline{∂} is closed. This difficulty is essentially the only difficulty: if W is a complete locally convex Hausdorff space and U is a closed subspace, then the quotient topology on W/U is complete and locally convex Hausdorff. (In these notes “Hausdorff” is part of the definition of “locally convex”; I have mentioned it explicitly here only for emphasis.)

Here is a summary of this discussion.

Proposition 7.19. Suppose X = G/L is a measurable complex flag variety for the real reductive group G (Definition 6.5), and that V is the holomorphic vector bundle on X attached to a (𝔮, L)-representation (τ, V). Endow the Dolbeault cohomology H^{p,q}(X, V) with the quotient topology as above, and define

H_{top}^{p,q}(X, V) = maximal Hausdorff quotient of H^{p,q}(X, V)
= kernel of \overline{∂}/closure of image of \overline{∂}.

Then H_{top}^{p,q}(X, V) carries a smooth representation of G (by translation of forms).

Serious geometers find the notion of H_{top}^{p,q} in this result to be anathema. Many of the long exact sequences (that make life worth living in sheaf theory) are lost on this quotient. Nevertheless, representation theory seems to demand this quotient. We will make use of it once more in section 8, to formulate Serre’s duality theorem for Dolbeault cohomology.

In the end our examples will offer no conclusive evidence about the value of the notion of H_{top}^{p,q}(X, V). We will recall next a theorem of Hon Wai Wong which says that in all of the cases we will consider, the operator \overline{∂} has closed range.

The first definition is analogous to Definition 5.6.

Definition 7.20. Suppose G is a real reductive group, 𝔮 is a very nice parabolic subalgebra of the complexified Lie algebra 𝔤, and L is a Levi factor of for 𝔮 (Definition 6.5). A (𝔮, L) representation (τ, V) (Definition 7.14) is said to be admissible if the representation τ of L is admissible (Definition 4.12). In this case the Harish-Chandra module of V is the (𝔮, L ∩ K)-module V^{L∩K} of L ∩ K-finite vectors in V.

Because 𝔮 = 𝔩⊕𝔲, with 𝔲 an L-stable ideal in 𝔮, every admissible representation (τ, V) of L extends canonically to an admissible (𝔮, L) representation, by making 𝔲 act by zero. If (τ, V) is irreducible for L, then this is the only possible extension. But if the representation of L is reducible, then other extensions exist, and even arise in practice.

Theorem 7.21. (Wong [Won99], Theorem 2.4). In the setting of Definition 7.20, assume that the admissible representation V is the maximal globalization of the underlying (𝔮, L ∩ K) module. Let V be the G-equivariant holomorphic vector bundle on X = G/L attached to V (Proposition 7.15). Then the \overline{∂} operator for Dolbeault cohomology has closed range, so that each of the spaces H^{p,q}(X, V) carries a smooth representation of G. Each of these representations is admissible, and is the maximal globalization of its underlying Harish-Chandra module.

Wong goes on to explain how these Harish-Chandra modules are constructed from V^{L∩K}, by a process called “cohomological parabolic induction.” We will say only a little about this.

This theorem should be compared to Proposition 5.11, to which it bears some formal resemblance. In detail it is unfortunately much weaker. With real parabolic induction, using any globalization on L led to a globalization of the same Harish-Chandra module on G. In the present setting that statement may be true, but Wong’s methods seem not to prove it.

Another difference is that in section 5 we were able to get many different globalizations just by varying the kinds of functions we used. The situation here is quite different. It is perfectly possible to consider (for example) the Dolbeault complex with generalized function coefficients instead of smooth functions. But the resulting Dolbeault cohomology turns out to be exactly the same. (This is certainly true if the vector bundle V is finite-dimensional, and it should be possible to prove a version for infinite-dimensional bundles as well.)

One goal of this section was to find a reasonable extension of the Borel-Weil Theorem to noncompact reductive groups. Theorem 2.3 suggested that one might look at bundles that are “antidominant” in some sense; but Propositions 7.2 and 7.4 suggested that antidominant is not such a good choice for noncompact G. I will dispense with further illuminating examples, and instead pass directly to the definition we want.

Suppose therefore that

(7.22)(a)  𝔮 = 𝔩 + 𝔲

is a very nice parabolic subalgebra (Definition 6.5), and L is a real Levi factor for 𝔮. Write

(7.22)(b)  X = G/L,  dim_{ℂ} X = dim_{ℂ}(𝔲) = n

(7.22)(c)  Z = K/L ∩ K,  dim_{ℂ} Z = dim_{ℂ}(𝔲 ∩ 𝔨) = s

Because of (7.18)(b), we are going to need

(7.22)(d)  2ρ(𝔲) = representation of L on ⋀^{n}(𝔤/𝔮)^{∗}

This is a one-dimensional character of L, so its differential (which we also write as 2ρ(𝔲)) is a linear functional on the Lie algebra 𝔩:

(7.22)(e)  2ρ(𝔲): 𝔩 → ℂ,  2ρ(𝔲)(Y) = trace of ad(Y) on (𝔤/𝔮)^{∗}

This linear functional is of course divisible by two, so we can define ρ(𝔲) ∈ 𝔩^{∗}, even though the group character 2ρ(𝔲) may have no square root. Any G-invariant symmetric nondegenerate bilinear form on 𝔤 provides an Lequivariant identification

(7.22)(f)  (𝔤/𝔮)^{∗} ≃ 𝔲,

which allows for some simplifications in the formulas for 2ρ(𝔲).

Let (τ^{L∩K}, V^{L∩K}) be an irreducible Harish-Chandra module for L, and (τ^{ω}, V^{ω}) its maximal globalization. Regard V^{ω} as (𝔮, L)-representation by making 𝔲 act by zero. Let

(7.22)(g)  V^{ω} = G ×_{L} V

be the associated holomorphic vector bundle on X (Proposition 7.15).

We want to write a condition on τ, more or less analogous to “antidominant” in Theorem 2.3, that will force Dolbeault cohomology with coefficients in V^{ω} to be well-behaved. For this purpose, a little bit of structure theory in the enveloping algebra is needed. Put

(7.23)(a)  ℨ(𝔤) = center of U(𝔤)

The group G acts by algebra automorphisms on ℨ(𝔤); the G_{0} action is trivial, so G/G_{0} is a finite group of automorphisms. We need the fixed point algebra

(7.23)(b)  ℨ^{G}(𝔤) = {u ∈ U(𝔤) | Ad(g)u = u, all g ∈ G} ⊂ ℨ(𝔤).

(The first algebra appeared already in the definition of admissible representations in Definition 4.12.) These algebras are described by the Harish-Chandra isomorphism. For that, fix a Cartan subalgebra

(7.23)(c)  𝔥 ⊂ 𝔩

The Weyl group of 𝔥 in 𝔤 (generated by root reflections) is written

(7.23)(d)  W = W (𝔤, 𝔥).

The Harish-Chandra isomorphism is

(7.23)(e)  ξ^{𝔤}: ℨ(𝔤)→^{∼}S(𝔥)^{W (𝔤,𝔥)}.

The disconnectedness of G provides a slightly larger group

(7.23)(e)  W^{G}(𝔤, 𝔥) ⊂ Aut(𝔤),

still acting as automorphisms of the root system. (We omit the definition in general. In the special case that every component of G has an element normalizing 𝔥, then W^{G} is generated by W (𝔤, 𝔥) and the automorphisms coming from Ad(G).) The group W^{G} contains W (𝔤, 𝔥) as a normal subgroup, and there is a natural surjective homomorphism

(7.23)(f)  G/G_{0} ↠ W^{G}(𝔤, 𝔥)/W (𝔤, 𝔥).

Now G/G_{0} acts on ℨ(𝔤), and W^{G}/W acts on S(𝔥)^{W}. These two actions are compatible via the Harish-Chandra isomorphism of (7.23)(e) and (7.23)(f). In particular, we get an isomorphism

(7.23)(g)  ξ^{G}: ℨ^{G}(𝔤)→^{∼}S(𝔥)^{W^{G}(𝔤,𝔥)}.

It follows from (7.23) that there is a bijection

(7.24)(a) (algebra homomorphisms ℨ^{G}(𝔤) → ℂ) ⟷ 𝔥^{∗}/W^{G}(𝔤, 𝔥).

The connection with representation theory is this. On any irreducible admissible representation (π, U) of G, the algebra ℨ^{G}(𝔤) must act by scalars. Consequently there is an element

(7.24)(b)  λ = λ(π) ∈ 𝔥^{∗}

(defined up to the action of W^{G}) with the property that

(7.24)(c)  π(z) = ξ^{G}(z)(λ)  (z ∈ ℨ^{G}(𝔤).

We call λ(π) the infinitesimal character of π.

The notion of dominance that we need for the representation τ will be defined in terms of the infinitesimal character of τ. To put the result in context, here is a basic fact about how the Dolbeault cohomology construction of Theorem 7.21 affects infinitesimal characters.

Proposition 7.25. In the setting of Theorem 7.21, assume that the (𝔮, L)- representation (τ, V) has infinitesimal character λ_{L}(τ) ∈ 𝔥^{∗}. Write n = dim_{ℂ} X, and ρ(𝔲) ∈ 𝔥^{∗} for the restriction of the linear functional in (7.22)(e). Then each G-representation H^{0,q}(X, V) has infinitesimal character λ_{L} −ρ(𝔲), and each G representation H^{n,q}(X, V) has infinitesimal character λ_{L} + ρ(𝔲).

The second assertion (about (n, q)-cohomology) is an immediate consequence of the first and (7.18)(b): tensoring a representation of L with ⋀^{n}(𝔤/𝔮)^{∗} adds 2ρ(𝔲) to its infinitesimal character. The first assertion is a version of the Casselman-Osborne theorem relating the action of ℨ^{G}(𝔤) to cohomology. (One can use the description (7.18)(b) of Dolbeault cohomology. The action of ℨ^{G}(𝔤) in that picture is by differentiation on the left on functions on G. Because we are considering central elements, this is equal to differentiation on the right, which is where the (𝔮, L)-cohomology is computed. We omit the elementary details.)

Of course the weight λ_{L} in Proposition 7.25 is defined only up to W^{L}(𝔩, 𝔥).

Definition 7.26. Suppose 𝔮 = 𝔩 + 𝔲 is a Levi decomposition of a parabolic subalgebra in the complex reductive Lie algebra 𝔤, and 𝔥 ⊂ 𝔩 is a Cartan subalgebra. A weight λ ∈ 𝔥^{∗} is called weakly dominant with respect to 𝔲 if for every coroot α^{∨} corresponding to a root of 𝔥 in 𝔲, ⟨α^{∨}, λ⟩ is not a strictly negative real number. That is,

⟨α^{∨}, λ⟩ ≥ 0  or  ⟨α^{∨}, λ⟩ is not real.

We say that λ is strictly dominant if (still for every such coroot)

⟨α^{∨}, λ⟩ > 0  or  ⟨α^{∨}, λ⟩ is not real.

The set of coroots α^{∨} for roots of 𝔥 in 𝔲 is permuted by W^{L}(𝔩, 𝔥), so these condtions depend only on the W^{L}(𝔩, 𝔥)-orbit of λ.

The terminology here is far from standard. One common variant is to require only that ⟨α^{∨}, λ⟩ never be a negative integer. That kind of hypothesis is not sufficient for the assertions about unitarity in Theorem 7.27.

Theorem 7.27. In the setting of Proposition 7.25, assume also that λ_{L}+ρ(𝔲) is weakly dominant for 𝔲 (Definition 7.26). Recall that Z = K/L ∩ K ⊂ X is a compact complex subvariety, and set s = dim_{ℂ}(Z).

(1) H^{n,q}(X, V) = 0 unless q = s.

(2) If L = L_{max} (Definition 6.5) and V is an irreducible representation of L, then H^{n,s}(X, V) is irreducible or zero.

(3) If the Harish-Chandra module of V admits an invariant Hermitian form, then the Harish-Chandra module of H^{n,s}(X, V) admits an invariant Hermitian form.

(4) If the Harish-Chandra module of V is unitary, then the Harish-Chandra module of H^{n,s}(X, V) is unitary.

Suppose now that λ_{L} + ρ(𝔲) is strictly dominant for 𝔲.

(5) If L = L_{max}, then the representation V of L is irreducible if and only if H^{n,s}(X, V) is irreducible or zero.

(6) The Harish-Chandra module of V admits an invariant Hermitian form if and only if the Harish-Chandra module of H^{n,s}(X, V) admits an invariant Hermitian form.

(7) The Harish-Chandra module of V is unitary if and only if the Harish-Chandra module of H^{n,s}(X, V) is unitary.

This summarizes some of the main results of [Vog84], translated into the language of Dolbeault cohomology using [Won99].

Theorem 7.27 is in many respects a valuable analogue of the Borel-Weil theorem for noncompact groups. One annoying feature is that the statement does not contain the Borel-Weil theorem as a special case. If G is compact, then Theorem 7.27 concerns top degree cohomology and dominant V, whereas Theorem 2.3 concerns degree zero cohomology and antidominant V. In order to round out the motivation appropriately, here is an alternate version of Theorem 2.3 addressing this incompatibility.

Theorem 7.28. (Borel-Weil, Harish-Chandra; see [HC56], [Ser59]). Suppose K is a compact connected Lie group with maximal torus T; use the notation of (2.1) and (2.2) above, and put n = dim_{ℂ}(K/T).

(1) The infinitesimal character of the representation (μ, ℂ_{μ}) of T is given by the differential of dμ ∈ 𝔱^{∗} of μ.

(2) The weight dμ + ρ is strictly dominant (Definition 7.26) if and only if μ is dominant in the sense of (2.2).

(3) The top degree Dolbeault cohomology H^{n,n}(K/T, L_{μ}) is non-zero if and only if μ is dominant. In that case, the Dolbeault cohomology space is an irreducible representation of K.

(4) This correspondence defines a bijection from dominant characters of T onto K̂.

Here we say that μ is dominant if and only if the inverse character −μ is antidominant (cf. (2.2)); that is, if and only if

⟨μ, α^{∨}⟩ ≥ 0

for every simple root α of T in K.

8 Compact supports and minimal globalizations

Theorem 7.27 provides a large family of group representations with unitary Harish-Chandra modules. It is entirely natural to look for something like a pre-Hilbert space structure on these group representations, that might be completed to a unitary group representation. Theorem 7.21 guarantees that each representation provided by Theorem 7.27 is the maximal globalization of its Harish-Chandra module. As explained in the introduction, we will see in Theorem 9.16 that maximal globalizations never admit G-invariant pre-Hilbert space structures (unless they are finite-dimensional). We need something analogous to Theorem 7.21 that produces instead minimal globalizations. Because of Definition 4.23, this means that we need to identify the dual of the topological vector space H^{p,q}(X, V). Let us first examine this in the setting of Definition 7.10, with V a holomorphic vector bundle on the complex manifold X. The definition involves the topological vector spaces A^{p,q}(V) of smooth (p, q)-forms on X with values in V (cf. (7.9)), and the Dolbeault operators

(8.1)(a)  A^{p,q−1}(V) →^{\overline{∂}} A^{p,q}(V) →^{\overline{∂}} A^{p,q+1}(V).

The first point is to identify the topological duals of these three spaces. The space C_{c}^{−∞}(W) of compactly supported distribution sections of a vector bundle W is by definition the topological dual of the space C^{∞}(W ⊗ D), with D the bundle of densities on the manifold. Because our manifold X is complex, it is orientable; so the bundle of densities is just the bundle of top degree differential forms on X. Top degree forms are (n, n)-forms (cf. (7.5)(b)), and it follows easily that

(8.1)(b)  A^{p,q}(V)^{∗} ≃ A_{c}^{(n−p,n−q),−∞}(V^{∗}).

Any continuous linear map T: E → F between topological vector spaces has a transpose ^{t}T: F^{∗} → E^{∗}. The Dolbeault operators in (8.1)(a) therefore give rise to transposes (8.1)(c)

A_{c}^{(n−p,n−q−1),−∞}(V^{∗}) →^{^{t}\overline{∂}} A_{c}^{(n−p,n−q),−∞}(V^{∗}) →^{^{t}\overline{∂}} A_{c}^{(n−p,n−q+1),−∞}(V^{∗}).

Calculating in coordinates shows that (up to a sign depending on p and q, which according to [Ser55], page 19 is (−1)^{p+q+1}) this transpose map “is” just the \overline{∂} operator for the Dolbeault complex for V^{∗}, applied to compactly supported distribution sections. (The way this calculation is done is to regard compactly supported smooth forms A_{c}^{(n−p,n−q+1),∞}(V^{∗}) as linear functionals on A^{p,q}(V), by pairing the V and V^{∗} and integrating the resulting (compactly supported smooth) (n, n)-form over X. Comparing the effects of \overline{∂} and ^{t}\overline{∂} on A_{c}^{(n−p,n−q+1),∞}(V^{∗}) now amounts to integrating by parts.)

Using this new complex, we can formulate an analogue of Definition 7.10.

Definition 8.2. Suppose X is a complex manifold of complex dimension n, and V is a holomorphic vector bundle on X. The compactly supported (p, q)- Dolbeault cohomology of X with coefficients in V is by definition

H_{c}^{p,q}(X, V) =

(kernel of \overline{∂} on A_{c}^{(p,q),−∞}(V))/(image of \overline{∂} from A_{c}^{(p,q−1),−∞}(V)).

At least if V is finite-dimensional, this is the Čech cohomology with compact supports of X with coefficients in the sheaf O_{Ω_{p}⊗V} of holomorphic p-forms with values in V. Just as in Proposition 7.19, there is a natural quotient topology on this cohomology, and we can define

H_{c,top}^{p,q}(X, V) = maximal Hausdorff quotient of H_{c}^{p,q}(X, V)
= kernel of \overline{∂}/closure of image of \overline{∂}.

In order to discuss transposes and duality, we need to recall a little about topologies on the dual E^{∗} of a complete locally convex space E. Details may be found for example in [Tre67], Chapter 19. For any subset B ⊂ E, and any ϵ > 0, we can define

(8.3)(a)  W_{ϵ}(B) = {λ ∈ E^{∗} | sup_{e∈B} |λ(e)| ≤ ϵ} ⊂ E^{∗}.

This is a subset of E^{∗} containing 0. The topologies we want on E^{∗} are defined by requiring certain of these subsets to be open. The weak topology on E^{∗} is defined to have neighborhood basis at the origin consisting of the sets W_{ϵ}(B) with B ⊂ E finite. (Another way to say this is that the weak topology is the coarsest one making all the evaluation maps λ ↦ λ(e) continuous.) We write E_{wk}^{∗} for E^{∗} endowed with the weak topology. (Treves writes E_{σ}^{∗}, and Bourbaki writes E_{s}^{∗}; more precisely, each uses a prime instead of a star for the continuous dual.) The topology of compact convergence on E^{∗} is defined to have neighborhood basis at the origin consisting of the sets W_{ϵ}(B) with B ⊂ E compact. We write E_{cpt}^{∗} for this topological space; Treves writes E_{c}^{∗}. The strong topology is defined to have neighborhood basis at the origin consisting of the sets W_{ϵ}(B) with B ⊂ E bounded. (Recall that B is bounded if for every neighborhood U of 0 in E, there is a scalar r ∈ ℝ so that B ⊂ rU.) We write E_{str}^{∗} for this topological space; Treves and Bourbaki write E_{b}^{∗}. Because a finite set is automatically compact, and a compact set is automatically bounded, it is clear that the topologies

(8.3)(b)  weak, compact convergence, strong

are listed in increasing strength; that is, each has more open sets than the preceding ones. For any of these three topologies on dual spaces, the transpose of a continuous linear map is continuous ([Tre67], Corollary to Proposition 19.5). If E is a Banach space, then the usual Banach space structure on E^{∗} defines the strong topology.

For most of the questions we will consider, statements about the strong topology on E^{∗} are the strongest and most interesting. Here is an example. We can consider the double dual space (E^{∗})^{∗}; what this is depends on the chosen topology on E^{∗}. Strengthening the topology on E^{∗} allows more continuous linear functionals, so

(8.4)(a)  (E_{wk}^{∗})^{∗} ⊂ (E_{cpt}^{∗})^{∗} ⊂ (E_{str}^{∗})^{∗}.

Each of these spaces clearly includes E (the evaluation maps at an element of E being continuous on E^{∗} in all of our topologies). In fact

(8.4)(b)  (E_{wk}^{∗})^{∗} = E

([Tre67], Proposition 35.1); this equality is a statement about sets, not topologies. Asking for similar statements for the other two topologies on E^{∗} asks for more; the most that one can ask is

Definition 8.5 . (see [Tre67], Definition 36.2). The (complete locally convex) topological vector space E is called reflexive if the natural inclusion

E ↪ (E_{str}^{∗})^{∗}

is an isomorphism of topological vector spaces.

For us, reflexivity will arise in the following way.

Definition 8.6. (see [Tre67], Definition 34.2). The (complete locally convex) topological vector space E is called a Montel space if every closed and bounded subset B ⊂ E is compact.

Proposition 8.7. ([Tre67], Corollary to Proposition 36.9, and Corollary 3 to Proposition 50.2). A Montel space is reflexive. A complete nuclear space is Montel. In particular, the analytic, smooth, distribution, and hyperfunction globalizations of any finite-length Harish-Chandra module (cf. section 4) are all reflexive.

Topological vector spaces that we define as dual spaces, like distribution spaces and the maximal globalization, will usually be endowed with the strong topology.

We are interested in the dual space of Dolbeault cohomology, which is a quotient of subspaces of a simple space of forms. We therefore need to know how to compute dual spaces of subspaces and quotients of topological vector spaces.

Proposition 8.8. Suppose E is a complete locally convex topological vector space, and M ⊂ E is a closed subspace. Endow M with the subspace topology, and E/M with the quotient topology (whose open sets are the images of the open sets in E.) Write i: M → E for the inclusion, q: E → E/M for the quotient map, and M^{⊥} ⊂ E^{∗} for the subspace of linear functionals vanishing on M.

(1) Every continuous linear functional λ_{M} on M (endowed with the subspace topology) extends to a continuous linear functional λ on E. That is, the transpose map

ti: E^{∗} → M^{∗}

is surjective, with kernel equal to M^{⊥}.

(2) Suppose that E is reflexive. Then the vector space isomorphism

ti: E^{∗}/M^{⊥} →^{∼} M^{∗}

is a homeomorphism from the quotient of the strong topology on E^{∗} to the strong topology on M^{∗}.

(3) If E/M is endowed with the quotient topology, then the continuous linear functionals are precisely those on E that vanish on M. That is, the transpose map

tq: (E/M)^{∗} → E^{∗}

is injective, with image equal to M^{⊥}.

(4) Suppose that E and M^{⊥} are reflexive. Then the vector space isomorphism

tq: (E/M)^{∗} →^{∼} M^{⊥}

is a homeomorphism from the strong topology on (E/M)^{∗} onto the subspace topology on M^{⊥} induced by the strong topology on E^{∗}.

The first assertion is the Hahn-Banach Theorem (see for example [Tre67], Chapter 18). The second may be found in [Bou87], Corollary to Theorem 1 in section IV.2.2. The third is more or less obvious. For the fourth, applying the second assertion to M^{⊥} ⊂ E^{∗} gives a homeomorphism

(M^{⊥})_{str}^{∗} ≃ E/M.

Now take duals of both sides, and use the reflexivity of M^{⊥}.

Finally, we need a few general remarks about transpose maps (to be applied to \overline{∂}). So suppose that

(8.9)(a)  T: E → F

is a continuous linear map of complete locally convex topological vector spaces, and

(8.9)(b)  ^{t}T: F^{∗} → E^{∗}

is its transpose. The kernel of T is a closed subspace of E, so the quotient E/ ker T is a complete locally convex space in the quotient topology. The image of T is a subspace of F, but not necessarily closed; its subspace topology is locally convex, and the completion of im T may be identified with its closure in F. We have a continuous bijection

(8.9)(c)  E/ ker T → im T,

but this need not be a homeomorphism. We now have almost obvious identifications

(8.9)(d)  ker^{t}T = linear functionals on F vanishing on im T

= (F/\overline{im T})^{∗} ⊂ F^{∗}

(8.9)(e)  im^{t}T = linear functionals on E vanishing on ker T,

and extending continuously from im T to F

= (im T)^{∗} = (\overline{im T})^{∗} ⊂ E^{∗}

(8.9)(f)  \overline{im^{t}T} = linear functionals on E vanishing on ker T
= (E/ ker T)^{∗} ⊂ E^{∗}

The question of when these vector space isomorphisms respect topologies is addressed by Proposition 8.8.

Lemma 8.10. In the setting of (8.9)(a), assume that the map (8.9)(c) is a homeomorphism. Then ^{t}T has closed range.

This is immediate from the descriptions in (8.9)(e) and (8.9)(f), together with the Hahn-Banach theorem. A famous theorem of Banach gives a sufficient condition for (8.9)(c) to be a homeomorphism:

Theorem 8.11. ([Tre67], Theorem 17.1). In the setting of (8.9)(a), assume that E and F are Fréchet spaces. Then (8.9)(c) is a homeomorphism if and only if T has closed range.

We can now say something about duals of cohomology spaces.

Proposition 8.12. Suppose that

E →^{T} F →^{S} G

is a complex of continuous linear maps of complete locally convex topological vector spaces, so that S ∘ T = 0. Define

H = ker S/ im T,

endowed with the quotient topology. This may be non-Hausdorff, and we define the maximal Hausdorff quotient

H_{top} = ker S/\overline{im T} ,

a complete locally convex topological vector space. Define the transpose complex

G^{∗} →^{^{t}S} F^{∗} →^{^{t}T} E^{∗}

with cohomology

tH = ker^{t}T/ im^{t}S

and maximal Hausdorff quotient

tH_{top} = ker^{t}T/\overline{im^{t}S}.

(1) There is a continuous linear bijection

F^{∗}/\overline{im^{t}S} → (ker S)^{∗}.

This is a homeomorphism if F is reflexive.

(2) The map in (1) restricts to a continuous linear bijection

tH_{top} = ker^{t}T/\overline{im^{t}S} → (im T)^{⊥} ⊂ (ker S)^{∗}.

This is a homeomorphism if F is reflexive.

(3) There is a continuous linear bijection

H^{∗} = (H_{top})^{∗} → (im T)^{⊥} ⊂ (ker S)^{∗}.

This is a homeomorphism if ker S and (im T)^{⊥} are both reflexive.

(4) There is a linear bijection

H^{∗} = (H_{top})^{∗} → ^{t}H_{top}.

This is a homeomorphism if F, its subspace ker S, and (im T)^{⊥} ⊂ (ker S)^{∗} are all reflexive.

(5) Assume that E, F, and G are nuclear Fréchet spaces, and that

H ≃ H_{top}, ^{t}H ≃ ^{t}H_{top}

are Hausdorff. The linear isomorphism

H^{∗} = (H_{top})^{∗} ≃ ^{t}H_{top}

of (4) is a homeomorphism.

(6) Assume that E, F, and G are nuclear Fréchet spaces, and that T and S have closed range. Then ^{t}T and ^{t}S also have closed range, so the cohomology spaces

H ≃ H_{top}, ^{t}H ≃ ^{t}H_{top}

are Hausdorff. The linear isomorphism

H^{∗} ≃ ^{t}H

of (4) is a homeomorphism.

Proof. Parts (1)–(3) are more or less immediate from Proposition 8.8, in light of (8.9). Part (4) simply combines (2) and (3). For (5), we need to know the reflexivity of the three spaces mentioned in (4). A subspace of a nuclear space is nuclear, and therefore reflexive (Proposition 8.7). This shows that F and ker S are reflexive. The dual of a nuclear Fréchet space is nuclear ([Tre67], Proposition 50.6), so (ker S)^{∗} is nuclear; so its closed subspace (im T)^{⊥} is nuclear, and therefore reflexive. For (6), the assertion about closed range follows from Banach’s Theorem 8.11, and the fact that the cohomology is Hausdorff follows at once.

From these generalities in hand, we get immediately a description of the topological dual of Dolbeault cohomology.

Theorem 8.13. (Serre [Ser55], Théorème 2). Suppose X is a complex manifold of dimension n, and V is a smooth holomorphic nuclear Fréchet vector bundle on X. Write V^{∗} for the topological dual bundle. Write H^{p,q}(X, V) for the (p, q) Dolbeault cohomology of X with coefficients in V, endowed with the (possibly non-Hausdorff) topological vector space structure defined before Proposition 7.19, and H_{top}^{p,q}(X, V) for its maximal Hausdorff quotient. Similarly define H_{c}^{p,q}(X, V^{∗}), the Dolbeault cohomology with compact support (and generalized function coefficients), and its maximal Hausdorff quotient H_{c,top}^{p,q}(X, V^{∗}) as in Definition 8.2. Then there is a natural topological isomorphism

H_{top}^{p,q}(X, V)^{∗} ≃ H_{c,top}^{n−p,n−q}(X, V^{∗}).

If the Dolbeault cohomology operators for V have closed range, then the same is true for the Dolbeault operators on compactly supported V^{∗}-valued forms with generalized function coefficients, and

H^{p,q}(X, V)^{∗} ≃ H_{c}^{n−p,n−q}(X, V^{∗}).

The main point is that the space of smooth sections of a smooth nuclear Fréchet bundle is a nuclear Fréchet space; it is easy to imitate [Ser55], section 8, to define a countable collection of seminorms giving the topology. With this fact in hand, Theorem 8.13 is a special case of Proposition 8.12, (4)–(6) (together with (8.1) and Definition 8.2).

Corollary 8.14. (cf. Bratten [Bra97], Theorem on page 285). In the setting of Definition 7.20, suppose X is the complex manifold G/L, and assume that the admissible representation V is the minimal globalization of the underlying (𝔮, L ∩ K)-module. Let A_{c}^{p,q}(X, V) be the Dolbeault complex for V with generalized function coefficients of compact support (cf. (8.1)(c)). Then the \overline{∂} operator has closed range, so that each of the corresponding coholomogy spaces H_{c}^{p,q}(X, V) carries a smooth representation of G (on the dual of a nuclear Fréchet space). Each of these representations of G is admissible, and is the minimal globalization of its underlying Harish-Chandra module.

This is immediate from Wong’s Theorem 7.21, Serre’s Theorem 8.13, and the duality relationship between minimal and maximal globalizations (Definitions 4.21 and 4.23). The theorem proved by Bratten is slightly different: he defines a “sheaf of germs of holomorphic sections” A(X, V), and proves a parallel result for the sheaf cohomology with compact support on X with coefficients in A(X, V). When V is finite-dimensional, the two results are exactly the same, since it is easy to check that Dolbeault cohomology (with compactly supported generalized function coefficients) computes sheaf cohomology in that case.

For infinite-dimensional V, comparing Corollary 8.14 with Bratten’s results in [Bra97] is more difficult. In these notes I have avoided many subtleties by speaking only about the Dolbeault complex, and not about sheaf cohomology. Part of the point of page 317 of Bratten’s paper is that I have in the past (for example in Conjecture 6.11 of [Vog87]) glossed over the difficulty of connecting sheaf and Dolbeault cohomology for infinite-dimensional bundles.

In the same way, we can translate Theorem 7.27 into this setting. For context, we should remark that Proposition 7.25 (computing infinitesimal characters of Dolbeault cohomology representations) applies equally to Dolbeault cohomology with compact support. The weight λ_{L} − ρ(𝔲) appearing in the next corollary is therefore the infinitesimal character of the representation H_{c}^{0,r}(X, V).

Corollary 8.15. In the setting of Definition 7.20, recall that Z = K/L ∩ K is an s-dimensional compact complex submanifold of the n-dimensional complex manifold X = G/L. Write r = n − s for the codimension of Z in X. Assume that V is an admissible (𝔮, L)-module of infinitesimal character λ_{L} ∈ 𝔥^{∗} (cf. (7.24)), and that V is the minimal globalization of the underlying (𝔮, L ∩ K)-module. Assume that λ_{L} − ρ(𝔲) is weakly antidominant for 𝔲; that is, that −λ_{L} + ρ(𝔲) is weakly dominant. Then

(1) H_{c}^{0,q}(X, V) = 0 unless q = r.

(2) If L = L_{max} (Definition 6.5) and V is an irreducible representation of L, then H_{c}^{0,r}(X, V) is irreducible or zero.

(3) If the Harish-Chandra module of V admits an invariant Hermitian form, then the Harish-Chandra module of H_{c}^{0,r}(X, V) admits an invariant Hermitian form.

(4) If the Harish-Chandra module of V is unitary, then the Harish-Chandra module of H_{c}^{0,r}(X, V) is unitary.

Suppose now that λ_{L} − ρ(𝔲) is strictly antidominant for 𝔲.

(5) If L = L_{max}, then the representation V of L is irreducible if and only if H_{c}^{0,r}(X, V) is irreducible or zero.

(6) The Harish-Chandra module of V admits an invariant Hermitian form if and only if the Harish-Chandra module of H_{c}^{0,r}(X, V) admits an invariant Hermitian form.

(7) The Harish-Chandra module of V is unitary if and only if the Harish-Chandra module of H_{c}^{0,r}(X, V) is unitary.

These statements follow immediately from Theorem 7.27 and Theorem 8.13.

We will see in section 9 that the Hermitian forms of Corollary 8.15(6) automatically extend continuously to H_{c}^{0,r}(X, V).

To conclude this section, notice that in the setting of the Borel-Weil Theorem (Theorem 2.3), we have X = Z = K/T, so r = 0; Theorem 2.3 is therefore “compatible” with Corollary 8.15.

9 Invariant bilinear forms and maps between representations

In Theorem 7.21 and Corollary 8.14, we have identified many representations with spaces related to smooth functions and distributions on manifolds. In this section, we will use these realizations to describe Hermitian forms on the representations. This is a three-step process. First, we will see (in Definition 9.6) how to

(9.1)(a) understand a Hermitian form on one representation as a special kind of linear map between two representations.

Describing Hermitian forms therefore becomes a special case of describing linear maps. For the representations we are considering, this amount to describing linear maps between function spaces. The second step (Theorem 9.8) is to

(9.1)(b) understand spaces of linear maps between function spaces as topological tensor products of function spaces.

The third step (which we will deal with more or less case by case) is to

(9.1)(c) understand tensor products of function spaces as function spaces on a product.

The second and third steps are closely connected to the Schwartz kernel theorem for distributions, and rely on the theory of nuclear spaces that Grothendieck developed to explain and generalize Schwartz’s theorem.

Before embarking on the technical details, we record the elementary ideas that we will be trying to generalize. So suppose for a moment that A and B are finite sets, say with n elements and m elements respectively. Define

(9.2)(a)  V_{A} = {complex-valued functions on A} ≃ ℂ^{n},

(9.2)(b)  V_{A}^{∗} = {complex-valued measures on A} ≃ ℂ^{n},

and similarly for B. The space V_{A}^{∗} is naturally identified with the dual space of V_{A} (as the notation indicates), by

λ(f) = ∫_{A} f dλ = ∑_{a∈A} f(a)λ(a);

in the second formula, the measure λ has been identified with the linear combination of delta functions (unit masses at points of A) ∑ λ(a)δ_{a}. For motivating the ideas above, we are meant to be thinking of V_{A} as the space of smooth functions on the manifold A, and of V_{A}^{∗} as distributions on A. In this setting, a version of (9.1)(b) is

(9.2)(c)  Hom_{ℂ}(V_{A}^{∗}, V_{B}) ≃ V_{A} ⊗ V_{B}.

The natural map from right to left is

f ⊗ g → T_{f⊗g}, T_{f⊗g}(λ) = λ(f)g.

A version of (9.1)(c) is

(9.2)(d)  V_{A} ⊗ V_{B} ≃ V_{A×B}.

The composite map

(9.2)(e)  V_{A×B} →^{∼} Hom_{ℂ}(V_{A}^{∗}, V_{B})

is

h → K_{h}, [K_{h}(λ)](b) = ∫_{A} h(x, b) dλ(x).

The operator K_{h} is a kernel operator, and (9.2)(e) is an example of the Schwartz kernel theorem.

One lesson that can be extracted even from this very simple example is that some of the easiest linear maps to understand are those going from spaces of distributions to spaces of functions. By rearranging the example slightly, we could also have found a nice description of the linear maps from a space of functions to a space of distributions.

As a second kind of warming up, here are two versions of the Schwartz kernel theorem that we will be imitating in the steps (9.1)(b) and (9.1)(c) above. In order to state these theorems, we will follow Schwartz and write D′(M) for the space of distributions on the smooth manifold M with arbitrary support; that is, the continuous dual of C_{c}^{∞}(M). (Elsewhere we have written this as C^{−∞}(M, D), with D the bundle of smooth densities on M.)

Theorem 9.3. (Schwartz kernel theorem; see [Tre67], Theorem 51.7). Suppose X and Y are smooth manifolds. Then the space L(C_{c}^{∞}(Y), D′(X)) (of continuous linear maps from compactly supported smooth functions on Y to distributions on X) may be identified with D′(X ×Y). The identification sends a distribution h on X × Y to the kernel operator

K_{h}: C_{0}^{∞}(Y) → D′(X), [K_{h}(ϕ)](ψ) = h(ψ ⊗ ϕ).

Here on the left we are describing the distribution K_{h}(ϕ) by evaluating it on a test function ψ ∈ C_{c}^{∞}(X). On the right, we regard ψ ⊗ ϕ as a test function on X × Y (to which the distribution h may be applied) by

(ψ ⊗ ϕ)(x, y) = ψ(x)ϕ(y).

Formally, the kernel operator in the theorem may be written

K_{h}(ϕ)(x) = ∫_{Y} h(x, y)ϕ(y).

This equation makes sense as written if h = H(x, y)dx dy, with dx and dy smooth measures on X and Y, and H a continuous function on X × Y. In this case

K_{h}(ϕ) = f(x) dx, f(x) = ∫_{Y} H(x, y)ϕ(y) dy.

Again following Schwartz, write E′(M) for the space of distributions with compact support (what we have written elsewhere as C_{c}^{−∞}(M, D).) For us a useful variant of the kernel theorem will be

Theorem 9.4. ([Tre67], page 533). Suppose X and Y are smooth manifolds. Then the space L(E′(Y), C_{c}^{∞}(X)) (of continuous linear maps from compactly supported distributions on Y to smooth functions on X) may be identified with C^{∞}(X × Y). The identification sends h ∈ C^{∞}(X × Y) to the kernel operator

K_{h}: E′(Y) → C^{∞}(X), [K_{h}(λ)](x) = λ(h(x, ·)).

We begin now with the machinery of linear maps and invariant Hermitian forms.

Definition 9.5. Suppose E and F are complete locally convex topological vector spaces. Write L(E, F) for the vector space of continuous linear maps from E to F. There are a number of important topologies on L(E, F), but (by virtue of omitting proofs) we will manage with only one: the strong topology of uniform convergence on bounded subsets of E (cf. [Tre67], page 337). (The definition is a straightforward generalization of the case F = ℂ described in (8.3) above.) Write L_{str}(E, F) for the topological vector space of linear maps with this topology. This is a locally convex space, and it is complete if E is bornological; this holds in particular if E is Fréchet or the dual of a nuclear Fréchet space.

Definition 9.6. Suppose E is a complete locally convex topological vector space. The Hermitian dual E^{h} of E consists of the continuous conjugate-linear functionals on E:

E^{h} = \{λ: E → ℂ, λ(av + bw) = \overline{a}λ(v) + \overline{b}λ(w) (a, b ∈ ℂ, v, w ∈ E)\}.

These are the complex conjugates of the continuous linear functionals on E, so there is a conjugate-linear identification E^{∗} ≃ E^{h}. We use this identification to topologize E^{h} (cf. (8.3)); most often we will be interested in the strong topology E_{str}^{h}. In particular, we use the strong topology to define the double Hermitian dual, and find a natural continuous linear embedding

E ↪ (E^{h})^{h},

which is a topological isomorphism exactly when E is reflexive.

Any continuous linear map T: E → F has a Hermitian transpose

T^{h}: F^{h} → E^{h}, T^{h}(λ)(e) = λ(T e).

The map T → T^{h} is conjugate-linear. In case S ∈ L(E, F^{h}), we will also write

S^{h} ∈ L(F, E^{h})

for the restriction of the Hermitian transpose to F ⊂ (F^{h})^{h}.

A Hermitian pairing between E and F is a separately continuous map

⟨, ⟩: E × F → ℂ

that is linear in the first variable and conjugate linear in the second. It is immediate that such pairings are naturally in bijection with L(E, F^{h}). The correspondence is

⟨, ⟩_{T} ↔ T: E → F^{h}, T (e)(f) = ⟨e, f⟩_{T} .

In case E = F, we say that the pairing is a Hermitian form on E if in addition

⟨e, f⟩ = \overline{⟨f, e⟩}.

In terms of the corresponding linear map T ∈ L(E, E^{h}), the condition is T = T^{h}. (Here we restrict T^{h} to E ⊂ (E^{h})^{h}.) The Hermitian form is said to be positive definite if

⟨e, e⟩ > 0, all non-zero e ∈ E.

Of course one can speak about bilinear pairings between E and F, which correspond to L(E, F^{∗}).

For tensor products we will make only a few general remarks, referring for details to [Tre67].

Definition 9.7. Suppose E and F are complete locally convex topological vector spaces. A “topological tensor product” of E and F is defined by imposing on the algebraic tensor product E⊗F a locally convex topology, and completing with respect to that topology. We will be concerned only with the projective tensor product. If p is a seminorm on E and q a seminorm on F, then we can define a seminorm p ⊗ q on E ⊗ F by

p ⊗ q(x) = inf ; x=∑ e_{i}⊗f_{i} ∑_{i} p(e_{i})q(f_{i}) (x ∈ E ⊗ F).

The projective topology on E ⊗ F is that defined by the family of seminorms p ⊗ q, where p and q vary over seminorms defining the topologies of E and F. The projective tensor product of E and F is the completion in this topology; it is written

E ˆ⊗_{π}F.

A characteristic property of this topology is that for any complete locally convex topological vector space G, L(E ˆ⊗_{π}F, G) may be identified with G-valued jointly continuous bilinear forms on E × F.

Here is Grothendieck’s general solution to the problem posed as (9.1)(b) above.

Theorem 9.8. ([Tre67], Proposition 50.5). Suppose E and F are complete locally convex topological vector spaces. Assume that

(1) E is barreled ([Tre67], page 346).

(2) E^{∗} is nuclear and complete.

(Both of these conditions are automatic if E is nuclear Fréchet or the dual of a nuclear Fréchet space.) Then the natural isomorphism

E^{∗} ⊗ F ≃ finite rank continuous linear maps from E to F

extends to a topological isomorphism

E^{∗} ⊗_{π} F ≃ L_{str}(E, F).

To translate this into representation-theoretic language, we need a lemma.

Lemma 9.9. Suppose X_{1} and X_{2} are Harish-Chandra modules of finite length for reductive groups G_{1} and G_{2}.

(1) X_{1} ⊗ X_{2} is a Harish-Chandra module of finite length for G_{1} × G_{2}.

(2) The minimal globalization of X_{1} ⊗ X_{2} is the projective tensor product of the minimal globalizations of X_{1} and X_{2}:

X_{1}^{ω} ⊗_{π} X_{2}^{ω} ≃ (X_{1} ⊗ X_{2})^{ω}.

(3) The smooth globalization of X_{1} ⊗ X_{2} is the projective tensor product of the smooth globalizations of X_{1} and X_{2}:

X_{1}^{∞} ⊗_{π} X_{2}^{∞} ≃ (X_{1} ⊗ X_{2})^{∞}.

(4) The distribution globalization of X_{1} ⊗ X_{2} is the projective tensor product of the distribution globalizations of X_{1} and X_{2}:

X_{1}^{−∞} ⊗_{π} X_{2}^{−∞} ≃ (X_{1} ⊗ X_{2})^{−∞}.

(5) The maximal globalization of X_{1} ⊗ X_{2} is the projective tensor product of the maximal globalizations of X_{1} and X_{2}:

X_{1}^{−ω} ⊗_{π} X_{2}^{−ω} ≃ (X_{1} ⊗ X_{2})^{−ω}.

Proof. The assertion in (1) is elementary. For the rest, fix Hilbert space globalizations X_{i}^{Hilb}, with orthonormal bases {e_{i}^{m}} of K_{i}-finite vectors. Then X_{1}^{Hilb} ⊗_{π} X_{2}^{Hilb} is a Hilbert space globalization of X_{1} ⊗ X_{2}, with orthonormal basis {e_{1}^{m} ⊗ e_{2}^{n}}. (The projective tensor product of two Hilbert spaces is topologically the same as the Hilbert space tensor product.) Now all of the canonical globalizations in sight are sequence spaces. For example,

X_{1}^{∞} = {∑ a_{m}e_{1}^{m} | a_{m} ∈ ℂ, |a_{m} · m^{k}| ≤ C_{k}, all k ≥ 0}

(cf. Theorem 4.18). The assertion in (3) amounts to the statement that the projective tensor product of the space of rapidly decreasing sequences on ℕ with itself is the space of rapidly decreasing sequences on ℕ×ℕ. This is an easy exercise (using the seminorms implicit in the definition of rapidly decreasing; compare [Tre67], Theorem 51.5). The remaining cases can be treated in exactly the same way.

Here is an abstract representation-theoretic version of the Schwartz kernel theorem.

Corollary 9.10. Suppose X_{1} and X_{2} are Harish-Chandra modules of finite length for G. Write X_{1}^{dual} for the K-finite dual Harish-Chandra module, X_{1}^{∞} for its smooth globalization, and so on as in section 4.

(1) There is a natural identification

Hom_{K × K-finite}(X_{1}, X_{2}) ≃ X_{1}^{dual} ⊗ X_{2}.

This is a Harish-Chandra module of finite length for G × G.

(2) There is a natural identification (as representations of G × G)

L_{str}(X_{1}^{ω}, X_{2}^{−ω}) ≃ (X_{1}^{dual})^{−ω} ⊗_{π} X_{2}^{−ω} ≃ (X_{1}^{dual} ⊗ X_{2})^{−ω}.

That is, the space of continuous linear maps from the minimal globalization of X_{1} to the maximal globalization of X_{2} may be identified with the maximal globalization of a Harish-Chandra module for G × G.

(3) There is a natural identification (as representations of G × G)

L_{str}(X_{1}^{∞}, X_{2}^{−∞}) ≃ (X_{1}^{dual})^{−∞} ⊗_{π} X_{2}^{−∞} ≃ (X_{1}^{dual} ⊗ X_{2})^{−∞}.

That is, the space of continuous linear maps from the smooth globalization of X_{1} to the distribution globalization of X_{2} may be identified with the distribution globalization of a Harish-Chandra module for G × G.

Proof. The assertions in (1) are elementary. The first isomorphism in (2) is Theorem 9.8 (bearing in mind the fact from Definition 4.23 that (X^{ω})^{∗} = (X^{dual})^{−ω}). The second is Lemma 9.9(2). Part (3) is identical.

We want to express this corollary more geometrically in the presence of geometric realizations of the representations X_{i}. As a warmup, we consider the situation of Proposition 5.11.

Corollary 9.11. Suppose P_{1} and P_{2} are parabolic subgroups of the reductive groups G_{1} and G_{2} (Definition 5.1), and that E_{i} is an admissible Harish-Chandra module for P_{i} (Definition 5.6). Write

X_{i} = (Ind_{P_{i}}^{G_{i}})^{K_{i}}(E_{i})

for the induced Harish-Chandra module for G_{i} as in (5.12), and describe their various canonical globalizations as in (5.15).

(1) The space L_{str}(X_{1}^{ω}, X_{2}^{−ω}) of maps from the minimal globalization to the maximal one may be identified with

(Ind_{P_{1}×P_{2}}^{G_{1}×G_{2}})^{−ω}(L_{str}(E_{1}^{ω}, E_{2}^{−ω})),

the space of hyperfunction sections of the bundle on (G_{1} × G_{2})/(P_{1} × P_{2}) induced by the corresponding space of linear maps between representations of P_{i}.

(2) The space L_{str}(X_{1}^{∞}, X_{2}^{−∞}) of maps from the smooth globalization to the distribution one may be identified with

(Ind_{P_{1}×P_{2}}^{G_{1}×G_{2}})^{−∞}(L_{str}(E_{1}^{∞}, E_{2}^{−∞})),

the space of distribution sections of the bundle on (G_{1} × G_{2})/(P_{1} × P_{2}) induced by the corresponding space of linear maps between representations of P_{i}.

The second observation was first made by Bruhat, who used it to begin the analysis of reducibility of induced representations. Here is the idea.

Corollary 9.12. (Bruhat [Bru56], Théorème 6;1). In the setting of Corollary 9.11, assume that G_{1} = G_{2} = G. The the space of G-intertwining operators from X_{1}^{∞} to X_{2}^{−∞} may be identified with the space of G_{Δ}-invariant generalized function sections of the bundle on (G × G)/(P_{1} × P_{2}) induced by L_{str}(E_{1}^{∞}, E_{2}^{−∞}). This space can in turn be identified with the space of continuous linear maps

Hom_{P_{1}}(E_{1}^{∞}, X_{2}^{−∞}),

or with

Hom_{P_{2}}(X_{1}^{∞}, E_{2}^{−∞}).

The first displayed formula (which is a version of Frobenius reciprocity) identifies intertwining operators with distributions on G/P_{2} having a certain transformation property under P_{1} on acting on the left. Bruhat proceeds to analyze such distributions using the (finite) decomposition of G/P_{1} into P_{2} orbits; equivalently, using the (finite) decomposition of (G×G)/(P_{1} ×P_{2}) into G_{Δ} orbits.

Here is the corresponding result for Dolbeault cohomology.

Corollary 9.13. Suppose 𝔮_{1} and 𝔮_{2} are very nice parabolic subalgebras for the reductive groups G_{1} and G_{2}, with Levi factors L_{1} and L_{2} (Definition 6.5), and that E_{i} is an admissible (𝔮_{i}, L_{i} ∩ K_{i})-module (Definition 7.20). Set

n_{i} = dim_{ℂ} Y_{i} = G_{i}/L_{i}.

Write E_{1}^{ω} for the minimal globalization of E_{1}, and E_{2}^{−ω} for the maximal globalization of E_{2}. These define holomorphic vector bundles

E_{1}^{ω} → Y_{1} = G_{1}/L_{1}, E_{2}^{−ω} → Y_{2} = G_{2}/L_{2}.

Define

X_{1}^{p,ω} = H_{c}^{0,n_{1}−p}(Y_{1}, E_{1}^{ω})

(Definition 8.2), the compactly supported Dolbeault cohomology of Y_{1} with coefficients in E_{1}^{ω}. This is an admissible representation of G_{1}, the minimal globalization of the underlying Harish-Chandra module X_{1}^{p,K_{1}} (Corollary 8.14). Similarly, define

X_{2}^{q,−ω} = H^{n_{2},q}(Y_{2}, E_{2}^{−ω})

(Definition 8.2), the Dolbeault cohomology of Y_{2} with coefficients in E_{2}^{−ω}. This is an admissible representation of G_{2}, the maximal globalization of the underlying Harish-Chandra module X_{2}^{q,K_{2}} (Theorem 7.21).

(1) The space of continuous linear maps

E_{12}^{−ω} = L_{str}(E_{1}^{ω}, E_{2}^{−ω})

is an admissible (𝔮_{1} ×𝔮_{2}, L_{1} ×L_{2})-representation (Definition 7.20), and is the maximal globalization of its underlying Harish-Chandra module. Write

E_{12}^{−ω} → Y_{1} × Y_{2}

for the corresponding holomorphic bundle.

(2) There is a natural identification

L_{str}(X_{1}^{·,ω}, X_{2}^{·,−ω}) ≃ H^{n_{1}+n_{2},·}(Y_{1} × Y_{2}, E_{12}^{−ω}).

Here in each case the dot · indicates that one should sum over the possible indices in question. More precisely,

∑_{p+q=m} L_{str}(X_{1}^{p,ω}, X_{2}^{q,−ω}) ≃ H^{n_{1}+n_{2},m}(Y_{1} × Y_{2}, E_{12}^{−ω}).

Proof. The cohomology on Y_{1} × Y_{2} is computed using a complex of forms

A^{n_{1}+n_{2},m}(Y_{1} × Y_{2}, E_{12}^{−ω}).

The fibers of E_{12}^{−ω} are tensor products

(E_{1}^{dual})^{−ω} ⊗ E_{2}^{−ω}

(Corollary 9.10(2)). Using this fact, the group-equivariant description of forms in Proposition 7.17, and standard ideas about tensor products of function spaces (cf. [Tre67], Theorem 51.6), one can prove that

A^{n_{1}+n_{2},m}(Y_{1} × Y_{2}, E_{12}^{−ω}) ≃ ∑_{p+q=m} A^{n_{1},p}(Y_{1}, E_{1}^{dual,−ω}) ⊗_{π} A^{n_{2},q}(Y_{2}, E_{2}^{−ω})

That is, the complex for Dolbeault cohomology on Y_{1} × Y_{2} is the projective tensor product of the complexes for Y_{1} and Y_{2}. Now one needs a Künneth formula for tensor products of nice complexes. (Recall that we know from Wong’s Theorem 7.21 that the \overline{∂} operators are topological homomorphisms, and all the spaces here are nuclear Fréchet.) We leave this step as an exercise for the reader.

Corollary 9.14. In the setting of Corollary 9.13, assume that G_{1} = G_{2} = G. Then the space of G-intertwining operators

∑_{p+q=m} Hom_{G}(X_{1}^{p,ω}, X_{2}^{q,−ω})

may be identified with the space of G_{Δ}-invariant Dolbeault cohomology classes in H^{n_{1}+n_{2},m}(Y_{1} × Y_{2}, E_{12}^{−ω}).

Because the coefficient bundle E_{12}^{−ω} is a tensor product, the Dolbealt cohomology on Y_{1} × Y_{2} has a natural quadruple grading; that is, each term of the bidegree has a bidegree, reflecting the degrees on Y_{1} and Y_{2}. We could therefore write

(9.15)  Hom_{G}(X_{1}^{p,ω}, X_{2}^{q,−ω}) ≃ H^{(n_{1},n_{2}),(p,q)}(Y_{1} × Y_{2}, E_{12}^{−ω}).

Just as in the setting of Corollary 9.12, the group G_{Δ} acts on Y_{1} ×Y_{2} with finitely many orbits. Everything about the analysis of this setting is slightly more complicated than in Corollary 9.12; even the Frobenius reciprocity isomorphisms described there are replaced by spectral sequences. Nevertheless one should be able to find some reasonable and interesting statements. We leave this task to the reader (with some suggestions in section 10).

It is now a simple matter to apply this result to the description of invariant Hermitian forms on Dolbeault cohomology representations. We begin with some general facts about Hermitian forms on representations.

Theorem 9.16. Suppose X^{K} is a Harish-Chandra module of finite length for G. Write X^{dual,K} for the K-finite dual Harish-Chandra module, and X^{herm,K} for the K-finite Hermitian dual; this is the same real vector space as X^{dual,K}, with the conjugate complex structure. Write X^{ω} and (X^{herm})^{ω} for the minimal globalizations, and so on as in section 4.

(1) The algebraic Hermitian dual of X is isomorphic to (X^{herm})^{−K}. Accordingly there is a natural identification

Hom(X, (X^{herm})^{−K}) ≃ (Hermitian pairings on X).

The conjugate linear automorphism of order two given by Hermitian transpose corresponds to interchanging variables and taking complex conjugate on Hermitian pairings. Hermitian forms on X correspond to the fixed points of this automorphism. We have

Hom_{𝔤,K}(X, (X^{herm})^{−K}) ≃ (invariant Hermitian pairings on X).

Any linear map on the left must take values in (X^{herm})^{K}.

(2) There is a natural identification of the continuous Hermitian dual

(X^{ω})^{h} = (X^{herm})^{−ω}.

Accordingly there is a natural identification

L_{str}(X^{ω}, (X^{herm})^{−ω}) ≃ (continuous Hermitian pairings on X^{ω}.)

This restricts to

Hom_{G,cont}(X^{ω}, (X^{herm})^{−ω}) ≃ (invariant Hermitian pairings on X^{ω}).

Any linear map on the left must take values in (X^{herm})^{ω}.

(3) There is a natural identification of the continuous Hermitian dual

(X^{∞})^{h} = (X^{herm})^{−∞}.

Accordingly there is a natural identification

L_{str}(X^{∞}, (X^{herm})^{−∞}) ≃ (continuous Hermitian pairings on X^{∞}).

This restricts to

Hom_{G,cont}(X^{∞}, (X^{herm})^{−∞}) ≃ (invariant Hermitian pairings on X^{∞}).

Any linear map on the left must take values in (X^{herm})^{∞}.

(4) Restriction of linear transformations defines isomorphisms of the (finitedimensional) spaces

Hom_{G,cont}(X^{∞}, (X^{herm})^{−∞}) ≃ Hom_{G,cont}(X^{ω}, (X^{herm})^{−ω})
≃ Hom_{𝔤,K}(X, (X^{herm})^{−K}).

(5) Any (𝔤, K)-invariant Hermitian form on the admissible Harish-Chandra module X^{K} extends continuously to the minimal and smooth globalizations X^{ω} and X^{∞}.

(6) Assume that X^{K} is irreducible. Then X^{K} admits a non-zero invariant Hermitian form if and only if X^{K} is equivalent to the (irreducible) Harish-Chandra module (X^{herm})^{K}. Such a form has a unique continuous extension to X^{ω} and to X^{∞}; it has no continous extension to X^{−∞} or to X^{−ω} unless X^{K} is finite-dimensional.

In all cases “continuous” Hermitian pairing means “separately continuous.” It turns out that the separately continuous forms here are automatically continuous; see for example [Tre67], Theorem 41.1.)

Proof. All the assertions in (1) are easy. The first assertion in (2) is essentially Definition 4.23 (with some complex conjugations inserted). The second then follows from the remarks in Definition 9.7. The third isomorphism is an obvious consequence. The final assertion in (2) is a special case of the “functoriality of minimal globalization” established in [Sch85]. Part (3) is proved in exactly the same way, using the Casselman-Wallach results. For part (4), we can change −∞ to ∞, −ω to ω, and −K to K by parts (1), (2), and (3).

Then these isomorphisms are again “functoriality of globalization.” Part (5) restates (4) using the facts in (2) and (3). For part (6), the irreducibility of (X^{herm})^{K} is elementary, so the assertion about forms on X^{K} amounts to (1) and Schur’s lemma. The existence of extensions to X^{ω} and to X^{∞} is (5).

For the non-existence of extensions to (say) X^{−∞}, one can prove exactly as in (3) that invariant Hermitian pairings on X^{−∞} correspond to continuous G-equivariant linear maps

Hom_{G,cont}(X^{−∞}, (X^{herm})^{∞}).

A G-map of admissible group representations must restrict to a (𝔤, K)-map of the underlying Harish-Chandra modules; and in our setting that map (if it is non-zero) has to be an isomorphism. From section 4, it follows that a non-zero map T must restrict to an isomorphism

T^{∞}: X^{∞} → (X^{herm})^{∞}.

The sequence space descriptions of the globalizations in section 4 show that such an isomorphism cannot extend continuously to X^{−∞}: a sequence (x_{μ}) would necessarily (by continuity) map to the sequence (T^{∞}(x_{μ})). This sequence is rapidly decreasing if and only if (x_{μ}) is rapidly decreasing (by the Casselman-Wallach uniqueness theorem for smooth globalization). If X^{K} is infinite-dimensional, then X^{−∞} must include slowly increasing sequences (x_{μ}) that are not rapidly decreasing, so T (x_{μ}) cannot be defined. The argument for X^{−ω} is identical.

We turn finally to Hermitian forms on Dolbeault cohomology representations. So suppose 𝔮 = 𝔩+𝔲 is a very nice parabolic subalgebra for the reductive group G, with Levi factor L. Put

(9.17)(a)  Y = G/L,  n = dim_{ℂ} Y.

Suppose E is an admissible (𝔮, L∩K)-module (Definition 7.20), with minimal globalization E^{ω}. Define

(9.17)(b)  E^{herm} = L ∩ K-finite Hermitian dual of E,

(cf. Definition 9.6); this is naturally an admissible (\overline{𝔮}, L∩K)-module. By Theorem 9.16(2), the maximal globalization of E^{herm} is precisely the (continuous) Hermitian dual of E^{ω}:

(9.17)(c)  (E^{herm})^{−ω} ≃ (E^{ω})^{h}.

Define

(9.17)(d)  F = Hom_{L ∩ K × L ∩ K-finite}(E, E^{herm}) ≃ E^{dual} ⊗ E^{h},

a space of Hermitian forms on E (cf. Definition 9.6 and Corollary 9.10(1)). This is an admissible (𝔮×\overline{𝔮}, L∩K ×L∩K)-module. Its maximal globalization is

(9.17)(e)  F^{−ω} = L_{str}(E^{ω}, (E^{ω})^{h}) ≃ (E^{dual})^{−ω} ⊗_{π} (E^{h})^{−ω},

the space of (separately continuous) Hermitian pairings on the minimal globalization E^{ω} (cf. Theorem 9.16(2) and Corollary 9.10(2)). The space F^{−ω} carries a conjugate-linear involution that we will write as bar. On Hermitian pairings τ, it is defined by

(9.17)(f)  \overline{τ}(e, f) = \overline{τ(f, e)}.

On linear maps, it is Hermitian transpose (Definition 9.6). In the tensor product (the last isomorphism of (9.17)(e)) it simply interchanges the factors. (This makes sense because E^{h} is the same real vector space as E^{dual}, with the opposite complex structure. With respect to the 𝔮 × \overline{𝔮} action, we have

(9.17)(g)  \overline{(X, Y) · τ} = (\overline{Y} , \overline{X}) · \overline{τ}  (X ∈ 𝔮, Y ∈ \overline{𝔮}).

There is a similar formula for the L × L representation.

Corollary 9.18. In the setting of (9.17), define G representations

X^{p,ω} = H_{c}^{0,n−p}(Y, E^{ω}),

which are minimal globalizations of the underlying Harish-Chandra modules X^{p}. Write Y^{op} for G/L with the opposite complex structure (defined by \overline{𝔮} instead of 𝔮).

(1) The Hermitian dual of X^{p,ω} is

(X^{p,h})^{−ω} ≃ H^{n,p}(Y^{op}, (E^{h})^{−ω}).

(2) The space of separately continuous Hermitian pairings on X^{p,ω} is

H^{(n,n),(p,p)}(Y × Y^{op}, F^{−ω});

here the coefficient bundle is induced by the representation F^{−ω}of (9.17)(e).

(3) The space of G-invariant Hermitian forms on X^{p,ω} may be identified with the space of G_{Δ}-invariant real Dolbeault cohomology classes in

H^{(n,n),(p,p)}(Y × Y^{op}, F^{−ω}).

Almost all of this is a formal consequence of Corollary 9.14, Theorem 9.16, and the definitions. One point that requires comment is the reference to “real” cohomology classes in (3). Suppose X is any complex manifold, and X^{op} the opposite complex manifold: this is the same as X as a smooth manifold, and the holomorphic vector fields on one are the antiholomorphic vector fields on the other. Complex conjugation carries (p, q)-forms on X to (p, q) forms on X^{op}, and respects \overline{∂}. Therefore complex conjugation defines a conjugate-linear isomorphism of order two

H^{p,q}(X) ≃ H^{p,q}(X^{op}).

Beginning with this idea, and the automorphism bar of F^{−ω}, one finds a conjugate linear isomorphism of order two

H^{(a,b),(c,d)}(Y × Y^{op}, F^{−ω}) → H^{(b,a),(d,c)}(Y × Y^{op}, F^{−ω}).

(The terms like (a, b) in the bidegree are transposed when we use the isomorphism

(Y × Y^{op})^{op} ≃ Y × Y^{op},

which interchanges the factors.) A “real” cohomology class is one fixed by this isomorphism.

What do the identifications of Hermitian pairings in Corollary 9.18 look like? An element

(9.19)(a)  v ∈ X^{p,ω} = H_{c}^{0,n−p}(Y, E^{ω})

is represented by a compactly supported (0, n−p)-form ṽ on Y, with values in the bundle E^{ω}. (I will write ṽ as if it were a smooth function, even though it actually has generalized function coefficients.) We can identify ṽ as a function

(9.19)(b)  G → Hom (⋀^{n−p} \overline{𝔲}, E^{ω}) ,

satisfying a transformation law on the right under L (cf. Proposition 7.17). If w̃ is another such representative, then ṽ ⊗ \overline{w̃} is a function

(9.19)(c)  G × G → Hom (⋀^{n−p} \overline{𝔲} ⊗ ⋀^{n−p} 𝔲, E^{ω} ⊗ \overline{E^{ω}}) .

Suppose now that τ is a cohomology class as in Corollary 9.18(2). A representative τ̃ may be identified with a smooth map

(9.19)(d)  G × G → Hom (⋀^{p} \overline{𝔲} ⊗ ⋀^{p} 𝔲, F^{−ω}) ,

with F^{−ω} the space of continuous Hermitian pairings on E^{ω}. Consequently the formal product τ̃ ∧ (ṽ ⊗ \overline{w̃}) is a (2n, 2n) form on Y × Y^{op} taking values in

(9.19)(e)  F^{−ω} ⊗ E^{ω} ⊗ \overline{E^{ω}}

At each point of G, we can apply the form value to the two vector values:

(9.19)(f)  F^{−ω} ⊗ E^{ω} ⊗ \overline{E^{ω}} → ℂ,  ϕ ⊗ e ⊗ \overline{f} ↦ ϕ(e, f).

This defines a complex-valued (2n, 2n)-form that we might sensibly denote τ̃(ṽ, w̃). This form is compactly supported because v and w are. It has generalized function coefficients, meaning that it is defined as an element of the dual space of smooth functions on Y × Y^{op}. We may therefore integrate it (that is, pair it with the function 1) and define

(9.19)(g)  ⟨v, w⟩_{τ} = ∫_{Y ×Y^{op}} τ̃(ṽ, w̃)

This is the identification in Corollary 9.18(2).

Here is a construction of unitary representations.

Corollary 9.20. In the setting of Definition 7.20, recall that Z = K/L ∩ K is an s-dimensional compact complex submanifold of the n-dimensional complex manifold X = G/L. Write r = n − s for the codimension of Z in X. Assume that V^{(2)} is an irreducible unitary representation of L of infinitesimal character λ_{L} ∈ 𝔥^{∗} (cf. (7.24)). Write V^{ω} for the subspace of analytic vectors in V^{(2)}. Regard V^{ω} as a (𝔮, L)-module by making 𝔲 act by zero, and let V^{ω} be the corresponding holomorphic bundle on X. Assume that λ_{L} − ρ(𝔲) is weakly antidominant for 𝔲; that is, that −λ_{L} + ρ(𝔲) is weakly dominant. Then

(1) H_{c}^{0,q}(X, V^{ω}) = 0 unless q = r.

(2) If L = L_{max} (Definition 6.5), then H_{c}^{0,r}(X, V) is irreducible or zero.

(3) The representation H_{c}^{0,r}(X, V^{ω}) admits a natural continuous positive definite invariant Hermitian form. Completing H_{c}^{0,r}(X, V^{ω}) with respect to this form defines a unitary representation of G.

Suppose in addition that λ_{L} − ρ(𝔲) is strictly antidominant for 𝔲. Then H_{c}^{0,r}(X, V^{ω}) is not zero.

This result is immediate from Corollary 9.18 and Corollary 8.15.

10 Open questions

My original goal in these notes was to write down (explicitly and geometrically) the pre-unitary structures provided by Corollary 9.20. According to Corollary 9.18, this amounts to

Question 10.1. In the setting of Corollary 9.20, write F^{ω} for the space of continuous Hermitian pairings on the analytic vectors V^{ω} for the unitary representation V^{(2)} of L. Regard F^{ω} as a smooth L × L representation, and write F^{ω} for the corresponding holomorphic bundle on Y × Y^{op}. Corollaries 9.18 and 9.20 provide a distinguished G_{Δ}-invariant Dolbeault cohomology class in

H^{(n,n),(s,s)}(Y × Y^{op}, F^{ω}),

with s the complex dimension of K/L ∩ K. This class is non-zero if λ_{L} − ρ(𝔲) is strictly anti-dominant for 𝔲. The problem is to give a simple geometric description of this class; perhaps to write down a representative (2n, 2s) form. The space of forms F^{ω} contains a distinguished line corresponding to the invariant form on V^{ω}. The difficulty is that this form is only L_{Δ}-invariant, so the line does not define a one-dimensional subbundle of F^{ω} (except along the diagonal in Y × Y^{op}).

In the case of Verma modules, construction of the Shapovalov form depends entirely on understanding the universal mapping property of Verma modules. In our setting, Question 10.1 should be related to questions of Frobenius reciprocity for Dolbeault cohomology representations, and these are in any case of interest in their own right.

Question 10.2. Suppose E is an admissible Harish-Chandra module for L, with maximal globalization E^{−ω} and minimal globalization E^{ω}. Regard these representations of L as (𝔮, L)-modules, by making 𝔲 act by zero. If X is any smooth admissible representation of G, we would like to calculate

Hom_{G}(X, H^{n,p}(Y, E^{ω})).

This should be related to (in fact equal to if L is compact)

Hom_{L}(H^{n−p}(𝔲, X), E^{−ω}).

We will offer a more precise statement in Conjecture 10.3. What appears in the second formula is the cohomology of the Lie algebra 𝔲 with coefficients in X. This is at least formally a representation of L; it is not clear how to define a nice topology.

Similarly, we would like to calculate

Hom_{G}(H_{c}^{0,q}(Y, E^{ω}), X).

This should be related to (equal to if L is compact)

Hom_{L}(E^{ω}, H^{q}(𝔲, X)).

Conjecture 10.3. (cf. [KV95], Theorem 5.120). Suppose X^{K} is an admissible Harish-Chandra module for G, with canonical globalizations X^{g} (for g = ω, g = ∞, and so on). Suppose 𝔮 = 𝔩 + 𝔲 is a very nice parabolic subalgebra of 𝔤 with Levi factor L (Definition 6.5). It is known that the Lie algebra cohomology H^{p}(𝔲, X^{K}) is an admissible Harish-Chandra module for L. Here are the conjectures.

(1) The 𝔲-cohomology complexes

Hom (⋀^{p} 𝔲, X^{g})

have the closed range property, so that the cohomology spaces inherit nice locally convex topologies. That L acts continuously on these cohomology spaces is easy.

(2) The representations of L on these cohomology spaces are canonical glob-

alizations:

H^{p}(𝔲, X^{g}) ≃ [H^{p}(𝔲, X^{K})]^{g}

for g = ω, g = ∞, and so on.

(3) In the setting of Question 10.2, there are two first quadrant spectral sequences with E_{2} terms

Ext_{L}^{r}(H^{n−t}(𝔲, X^{ω}), E^{−ω})

and

Ext_{G}^{a}(X^{ω}, H^{n,b}(Y, E^{−ω}))

with a common abutment.

(4) There are two first quadrant spectral sequences with E_{2} terms

Ext_{L}^{r}(E^{ω}, H^{t}(𝔲, X^{−ω}))

and

Ext_{G}^{a}(H_{c}^{0,b}(Y, E^{ω}), X^{−ω}).

Statement (1) makes sense with X^{g} replaced by any smooth globalization of X^{K}; one should ask only that the cohomology be some smooth globalization of the right Harish-Chandra module. I have not thought carefully about this, but I know no reason for it to fail. The specific version in (1) here ought to be fairly easy to prove, however.

Similarly, statements (3) and (4) should probably be true with X^{ω} and X^{−ω} replaced by any smooth globalization of X^{K}. The specific versions here are those most closely related to the construction of forms in Corollary 9.20.

In the case of the minimal globalization (the case g = ∞), generalizations of statements (1) and (2) of these conjectures have been established by Tim Bratten in [Bra98]; he considers arbitrary parabolic subalgebras endowed with real θ-stable Levi subalgebras. The generalization of (1) for the maximal globalization follows easily by a duality argument.

Bratten has pointed out that the generalization of (2) cannot extend to the case of real parabolic subalgebras and distribution or maximal globalizations. Here is one reason. Suppose that P = LU is a real parabolic subgroup of G, and that X^{K} is an irreducible Harish-Chandra module with maximal globalization X^{−ω}. Harish-Chandra’s subquotient theorem guarantees that H^{0}(𝔲, X^{−ω}) ≠ 0. (Simply embed the dual representation, which is a minimal globalization, in a space of analytic sections of a bundle on G/P. Then evaluation of sections at the identity coset eP defines a U-invariant continuous linear functional on the dual representation; that it, a U-invariant vector in X^{−ω}. The same argument applies to the distribution globalization.) But H^{0}(𝔲, X^{K}) is equal to zero in almost all cases, contradicting the analogue of (2).

The spaces Ext_{L} and Ext_{G} are in the category of continuous representations of G. In order to interpret Conjecture 10.3, it would be helpful to have

Conjecture 10.4. Suppose X^{K} and Y^{K} are admissible Harish-Chandra modules for G, with canonical globalizations X^{g} and Y^{g}, for g = ω, g = ∞, and so on. Then the standard complex

Hom_{K} (⋀^{p}(𝔤/𝔨, L(X^{ω}, Y^{−ω}))

has the closed range property. Its cohomology is isomorphic to Ext_{𝔤,K}(X^{K},Y^{K}). The same result holds with ω replaced by ∞.

Results at least very close to this may be found in [BW80], Chapter 9; I have not checked whether this statement follows from their results.

References

[Bou87] N. Bourbaki. Topological vector spaces. Chapters 1–5. Springer-Verlag, Berlin-Heidelberg-New York, 1987.

[Bra97] T. Bratten. Realizing representations on generalized flag manifolds. Compositio math., 106:283–319, 1997.

[Bra98] T. Bratten. A comparison theorem for Lie algebra homology groups. Pacific J. Math., 182:23–36, 1998.

[Bru56] F. Bruhat. Sur les représentations induites des groupes de Lie. Bull. Soc. Math. France, 84:97–205, 1956.

[BW80] A. Borel and N. Wallach. Continuous Cohomology, Discrete Subgroups, and Representations of Reductive Groups. Princeton University Press, Princeton, New Jersey, 1980.

[Cas89] W. Casselman. Canonical extensions of Harish-Chandra modules to representations of G. Can. J. Math., 41:385–438, 1989.

[HC56] Harish-Chandra. Representations of semisimple Lie groups. V. Amer. J. Math., 78:1–41, 1956.

[Hel94] S. Helgason. Geometric Analysis on Symmetric Spaces, volume 39 of Mathematical Surveys and Monographs. American Mathematical Society, Providence, Rhode Island, 1994.

[How88] R. Howe. The oscillator semigroup. In The Mathematical Heritage of Hermann Weyl, volume 48 of Proceedings of Symposia in Pure Mathematics, pages 61–132. American Mathematical Society, Providence, Rhode Island, 1988.

[How89] R. Howe. Transcending classical invariant theory. J. Amer. Math. Soc., 2:535–552, 1989.

[Kna86] A. Knapp. Representation Theory of Semisimple Groups: An Overview Based on Examples. Princeton University Press, Princeton, New Jersey, 1986.

[Kos61] B. Kostant. Lie algebra cohomology and the generalized Borel-Weil theorem. Ann. of Math., 74:329–387, 1961.

[KV95] A. Knapp and D. Vogan. Cohomological Induction and Unitary Representations. Princeton University Press, Princeton, New Jersey, 1995.

[KZ77] A. Knapp and G. Zuckerman. Classification theorems for representations of semisimple Lie groups. In Non-commutative Harmonic Analysis, volume 587 of Lecture Notes in Mathematics, pages 138–159. Springer-Verlag, Berlin-Heidelberg-New York, 1977.

[Sch85] W. Schmid. Boundary value problems for group invariant differential equations. In The mathematical heritage of Élie Cartan, pages 311–321. Astérisque, 1985.

[Ser55] J. P. Serre. Un théorème de dualité. Comment. Math. Helv., 29:9–26, 1955.

[Ser59] J. P. Serre. Représentations linéaires et espaces homogènes kählériens des groupes de Lie compacts. In Séminaire Bourbaki, 6ième année: 1953/54. Secrétariat mathématique, Paris, 1959.

[Tre67] F. Treves. Topological vector spaces, distributions, and kernels. Academic Press, New York, 1967.

[TW71] J. Tirao and J. Wolf. Homogeneous holomorphic vector bundles. Indiana Univ. Math. J., 20:15–31, 1970/1971.

[Vog84] D. Vogan. Unitarizability of certain series of representations. Ann. of Math., 120:141–187, 1984.

[Vog87] D. Vogan. Unitary Representations of Reductive Lie Groups. Annals of Mathematics Studies. Princeton University Press, Princeton, New Jersey, 1987.

[Wei64] A. Weil. Sur certaines groupes d’opérateurs unitaires. Acta Math., 111:143– 211, 1964.

[Wol69] J. Wolf. The action of a real semisimple group on a complex flag manifold. i. orbit structure and holomorphic arc components. Bull. Amer. Math. Soc., 75:1121–1237, 1969.

[Won99] H. Wong. Cohomological induction in various categories and the maximal globalization conjecture. Duke Math. J., 96:1–27, 1999.