noblackboxes The Orbit Method and Unitary Representations for Reductive Lie Groups

Based on lectures at the European School of Group Theory Sandbjerg, Denmark August 1994 David A. Vogan, Jr. * Department of Mathematics Massachusetts Institute of Technology Cambridge, Massachusetts 02139

Introduction. Suppose G is a real reductive algebraic group. A unitary representation of G is a Hilbert space endowed with a continuous action of G by unitary operators. It is irreducible if the Hilbert space is non-zero, but cannot be written as a direct sum in a non-trivial G-invariant way. The orbit method pioneered by Kirillov and Kostant seeks to construct irreducible unitary representations by analogy with quantization procedures in mechanics.

A classical physical system may sometimes be modelled by a configuration space M, a manifold whose points represent the possible positions of the bodies in the system. The corresponding phase space is the cotangent bundle T^{∗}M, whose points represent the possible states (positions and momenta) of the bodies in the system. In this setting a classical observable is a function on T^{∗}M. The quantum mechanical analogue of this system is based on the Hilbert space L^{2}(M) of square-integrable half densities on M. A state of the quantum system is a unit vector v in L^{2}(M); |v|^{2} is a probability density on M, usually thought of as describing the probability of observing the quantum mechanical state in a given classical configuration. A quantum-mechanical observable is an operator T on L^{2}(M); the scalar product ⟨T v, v⟩ is the expected value of the observable on the state v. Some connection between classical and quantum observables is provided by a symbol calculus; an interesting quantum observable is often a differential operator on M, and the corresponding classical observable is related to the symbol of the differential operator (a function on T^{∗}M).

The states of some more complicated classical mechanical systems may be represented as points of a symplectic manifold X. Here X corresponds to the cotangent bundle T^{∗}M in the previous example, but there is no longer an underlying configuration manifold M. Classical observables are still functions on X. A quantization of this classical system is a Hilbert space H(X) endowed with an algebra A(X) of operators. Here A(X) is to be some non-commutative analogue of the algebra of functions on X. It is not easy to say in general what H(X) ought to be, except in some special classes of examples like the one in the previous paragraph.

This relationship between classical and quantum mechanics suggests a classical analogue of an irreducible unitary representation of G: a homogeneous space for G, endowed with an invariant symplectic structure. Such homogeneous spaces turn out to be very close to coadjoint orbits: orbits of G on the dual of its Lie algebra. The orbit method in representation theory seeks to attach to a coadjoint orbit X (regarded as the phase space of a classical mechanical system admitting G as a group of symmetries) a Hilbert space H(X) (the state space for the quantized system). If this can be done in sufficiently natural way, then the action of G on X by symplectomorphisms will give rise to an action of G on H(X) by unitary operators; that is, to a unitary representation π(X) of G.

In the case of reductive groups, the orbit method is fairly well understood for semisimple orbits X: that is, one can construct in a natural way a unitary representation π(X). The purpose of these notes is to outline this construction. Following an idea of Dixmier, we will emphasize not so much the Hilbert space H(X) as the algebra of operators A(X). There are several advantages to this approach. First, the relationship between X and A(X) is somewhat more elementary and direct than that between X and H(X). Second, A(X) depends only on the complexification of G. Because complex reductive groups are more or less combinatorial in nature, the operator algebras are necessarily uncomplicated.

Here is an outline of the notes. In section 1 we recall the definition of real reductive groups, and some of their structure theory. Section 2 recalls some general facts about representation theory of Lie groups and enveloping algebras. Section 3 outlines ideas of Dixmier about ideals in enveloping algebras. At the end of section 3 there is a moderately precise formulation of the problem that the orbit method seeks to solve.

Some of the basic notions from the method of coadjoint orbits are summarized in section 4. Sections 5 and 6 describe the classification of coadjoint orbits for reductive groups. The orbits are all built from three

* Supported in part by NSF grant DMS-9402994. special classes: hyperbolic, elliptic, and nilpotent. The semisimple orbits are those built only from hyperbolic and elliptic pieces. It turns out (although we will not explain why) that the construction of representations attached to coadjoint orbits can more or less be reduced to these three classes. The first two have been treated completely (as we will explain), but the third is understood only in special cases.

In section 7 we construct the representations attached to hyperbolic coadjoint orbits. Roughly speaking, these are principal series and “degenerate series” representations induced from one-dimensional characters of parabolic subgroups.

The operator algebras that we construct here are all differential operator algebras. Section 8 is an elementary discussion of such operators, arranged in a form that fits well with the orbit method. The actual construction of some algebras A(X) appears in section 9.

Finally, section 10 contains a (very incomplete) outline of Zuckerman’s construction of representations associated to elliptic coadjoint orbits. A complete account will appear in [Knapp-Vogan].

Some other expositions of related material include [Vogan87], [Guichardet], and [Vogan88].

During the summer of 1994 I lectured on this material at the Nankai Institute of Mathematics; at the Instituto de Matemáticas at UNAM; at CIMAT in Guanajuato; and finally at the European School of Group Theory. The comments of all of those audiences have been a great help to me in preparing these notes. Henrik Schlichtkrull offered many improvements and corrections to the manuscript. For the flaws that remain, I must of course assume the responsibility.

1. Reductive groups.

A real reductive group is an abstract mathematical object of which there are relatively few examples. For this reason the general theory is informed and guided by the examples to a remarkable extent. This is evident even in one of the simplest definitions of a real reductive group, which is based on some familiar properties of matrices.

Write G = GL(n, ℝ) for the group of invertible n × n matrices with real entries, the general linear group over ℝ. If g ∈ G, define

θg = ^{t}g^{−1},  (1.1)(a)

the inverse of the transpose of g. Since transpose and inversion are both anti-automorphisms of G (meaning that ^{t}(gh) = ^{t}h^{t}g, for example), the map θ is an automorphism. Since transpose and inversion have order 2 and commute with each other, θ has order 2. We call θ the Cartan involution of G. Write K = G^{θ} for the subgroup of fixed points of θ. This is the group O(n) of n × n real orthogonal matrices, the orthogonal group. It is compact.

Write 𝔤_{0} = 𝔤𝔩(n, ℝ) for the Lie algebra of G, consisting of all n × n matrices with real entries. The differential of θ at the identity is an involutive linear automorphism of 𝔤_{0}, also written θ. We have

θX = −^{t}X  (1.1)(b)

for X ∈ 𝔤_{0}. (Using the same letter for the differential of θ is an abuse of notation, since G ⊂ 𝔤_{0}, but no confusion should result.) The +1-eigenspace of θ is

𝔨_{0} = n × n skew-symmetric real matrices;  (1.1)(c)

it is the Lie algebra of K. The −1-eigenspace of θ is

𝔭_{0} = n × n symmetric real matrices.  (1.1)(d)

We have

𝔤_{0} = 𝔨_{0} ⊕ 𝔭_{0},  (1.1)(e)

the Cartan decomposition of 𝔤𝔩(n, ℝ).

There is a corresponding decomposition for G. For that, we need a lemma. Recall that a real matrix g is called positive definite symmetric if the bilinear form on ℝ^{n} defined by B_{g}(v, w) = ⟨gv, w⟩ is positive definite and symmetric. (Here we have written ⟨, ⟩ for the usual inner product on ℝ^{n}.)

Lemma 1.2 a) An n × n real matrix X is symmetric if and only if there is an orthogonal basis of ℝ^{n} consisting of

eigenvectors of X.

b) An n × n real matrix g is positive definite symmetric if and only if there is an orthogonal basis of ℝ^{n} consisting of eigenvectors of g with strictly positive eigenvalues.

c) The exponential map is an analytic diffeomorphism

exp : 𝔭_{0} → P

from symmetric matrices to positive definite symmetric matrices. The inverse

log : P → 𝔭_{0}

is also analytic.

Sketch of proof. Part (a) is standard linear algebra. For (b), suppose first that g ∈ P. The symmetry of the form B_{g} makes g symmetric. The existence of an orthonormal basis of eigenvectors follows from (a). If v is a non-zero eigenvector with eigenvalue λ, then the assumed positivity of B_{g} gives

0 < B_{g}(v, v)/⟨v, v⟩ = (gv, v)/⟨v, v⟩ = (λv, v)/⟨v, v⟩ = λ.

So the eigenvalues are positive. The converse assertion is elementary. For (c), it follows from (a) and (b) that exp is a one-to-one map of 𝔭_{0} onto P. The analyticity of log is slightly more subtle; it may be proved for example by calculating the Jacobian of exp. Q.E.D.

Proposition 1.3 (Polar or Cartan decomposition for GL(n, ℝ).) Suppose G = GL(n, ℝ), K = O(n), and 𝔭_{0} is the space of n × n symmetric matrices. Then the map

K × 𝔭_{0} → G (k, X) ↦ k exp(X)

is an analytic diffeomorphism of K × 𝔭_{0} onto G. The inverse is given by

g ↦ (g exp(−1/2 log((θg)^{−1}g)), 1/2 log((θg)^{−1}g)).

Proof. Suppose g ∈ GL(n, ℝ). Define p = (θg)^{−1}g = (^{t}g)g. The bilinear form attached to p is
B_{p}(v, w) = ⟨pv, w⟩ = ⟨^{t}ggv, w⟩ = ⟨gv, gw⟩,

which is symmetric and positive definite. So p is positive definite symmetric. By Lemma 1.2, there is a unique symmetric matrix X with p = exp(2X). Define k = g exp(−X). Then

tkk = exp(−X)^{t}gg exp(−X) = exp(−X) exp(2X) exp(−X) = 1,

so k belongs to O(n). By construction g = k exp(X), which proves the surjectivity of the polar decomposition map. The construction also proves the formula for the inverse. Q.E.D.

The polar decomposition allows one to study many structural problems about GL(n, ℝ) in two steps: one involving the compact group K = O(n), and one involving the vector space 𝔭_{0} (often regarded as a representation of K). Elie Cartan discovered that all real reductive groups share a similar property. It is so fundamental that it may be taken as the definition of the class.

Definition 1.4. A subgroup G ⊂ GL(n, ℝ) is called a linear real reductive group if

1) G is closed;

2) if X is a symmetric matrix, then X ∈ Lie(G) if and only if exp(X) ∈ G; and

3) G is preserved by the Cartan involution θ of GL(n, ℝ). That is, a matrix g belongs to G if and only if tg belongs to G.

Suppose G is such a group. The restriction of θ to G (still denoted θ) is called the Cartan involution of G. Write

𝔤_{0} = Lie(G) ⊂ 𝔤𝔩(n, ℝ)

for the Lie algebra of G; necessarily it is preserved by the Cartan involution θ of 𝔤𝔩(n, ℝ). Accordingly we can write

𝔤_{0} = 𝔨_{0} ⊕ 𝔭_{0}

for the decomposition into +1 and −1 eigenspaces (the skew-symmetric and symmetric matrices in 𝔤_{0}). Put

K = G^{θ} = G ∩ O(n),

a compact subgroup of G (by condition (1)); its Lie algebra is 𝔨_{0}.

Proposition 1.5 (Cartan decomposition for linear real reductive groups). In the setting of Definition 1.4, the map

K × 𝔭_{0} → G, (k, X) ↦ k exp(X)

is an analytic diffeomorphism of K × 𝔭_{0} onto G.

Proof. Suppose g ∈ G. Write g = k exp(X) for the polar decomposition of g in GL(n, ℝ). Then exp(2X) = ^{t}gg, which belongs to G by condition (3) of Definition 1.4. By condition (2) of the definition, X belongs to 𝔭_{0}. It follows that k = g exp(−X) belongs to G ∩ O(n) = K. This proves the surjectivity of the Cartan decomposition map. The remaining assertions follow from Proposition 1.3. Q.E.D.

Many of the most interesting real reductive groups are non-linear; that is, they do not appear as subgroups of GL(n, ℝ). The following definition is broad enough for us.

Definition 1.6. A real reductive group is a Lie group G endowed with a continuous homomorphism π : G → GL(n, ℝ), subject to the following conditions:

1) π(G) = \overline{G} is a linear real reductive group (Definition 1.4); and

2) the kernel of π is finite.

Suppose G is such a group. We use the differential of π to identify the Lie algebra 𝔤_{0} of G with \overline{𝔤}_{0} = Lie(\overline{G}) ⊂ 𝔤𝔩(n, ℝ). Define 𝔨_{0}, 𝔭_{0}, and \overline{K} as in Definition 1.4, and put

K = π^{−1}(\overline{K}).

By hypothesis (2), K is a compact subgroup of G with Lie algebra 𝔨_{0}.

Proposition 1.7 (Cartan decomposition for real reductive groups). In the setting of Definition 1.6, the map

K × 𝔭_{0} → G, (k, X) ↦ k exp(X)

is an analytic diffeomorphism of K × 𝔭_{0} onto G. The map θ : G → G defined by

θ(k exp(X)) = k exp(−X)

is an involutory automorphism (the Cartan involution) with G^{θ} = K.

Proof. Given g ∈ G, write

π(g) = \overline{k} exp(\overline{X}),

with \overline{X} ∈ \overline{𝔭}_{0}. The identification of \overline{𝔭}_{0} with 𝔭_{0} provides an element X ∈ 𝔭_{0}. The element k = g exp(−X) satisfies π(k) = \overline{k}, and so belongs to K. This proves the surjectivity of the decomposition, and the analyticity of the inverse maps. Analyticity of the map to G follows from the analyticity of the exponential map and of multiplication.

For the last assertion, θ is obviously an analytic map of order two with fixed point set K. What must be shown is that θ(gh) = θ(g)θ(h) for all g, h ∈ G. To prove this, consider the function

F (g, h) = θ(gh)θ(h)^{−1}θ(g)^{−1}

from G × G to G. We want to show that it carries G × G to the point 1. Since θ is an automorphism of \overline{G}, F takes values in the kernel of π. Since this is a finite set, F is constant on the connected components of G × G. Obviously F is trivial on K × K; but by the Cartan decomposition, this subgroup meets every connected component of G × G. Q.E.D.

The definition of reductive group has the following “hereditary” property.

Proposition 1.8. Let G = K exp(𝔭_{0}) be the Cartan decomposition of a real reductive group with Cartan involution θ. Let H be a subgroup of G, and assume that

a) H is closed;

b) if X ∈ 𝔭_{0}, then X ∈ Lie(H) if and only if exp(X) ∈ H; and

c) H is preserved by θ.

Then H is a real reductive group with Cartan involution θ|_{H}.

This is almost immediate from Definition 1.6.

In order to construct interesting examples of reductive groups, we need a way to verify condition (2) of Definition 1.4. Here is one.

Lemma 1.9. Suppose A is an n × n matrix and X is a symmetric n × n matrix. Then X commutes with A if and only if exp(X) commutes with A.

Proof. The first (respectively second) condition is equivalent to the assertion that the linear transformation A respects the decomposition of ℝ^{n} as a direct sum of eigenspaces of X (respectively exp(X)). By Lemma 1.2, these decompositions coincide. Q.E.D.

Lemma 1.10. Suppose G is a real reductive group and X ∈ 𝔭_{0}.

a) If g ∈ G, then Ad(g)X = X if and only if g commutes with exp X.

b) If Y ∈ 𝔤_{0}, then ad(X)Y = 0 if and only if Ad(exp X)Y = Y.

Proof. The two statements are very similar; we consider only the first. Write π : G → \overline{G} ⊂ GL(n, ℝ) as in Definition 1.6, and \overline{g} = π(g), \overline{X} = dπ(X). “Only if” is trivial, so suppose g commutes with exp X. Applying π, we find that \overline{g} commutes with exp \overline{X}. By Lemma 1.9, \overline{g} commutes with \overline{X}; so Ad(\overline{g})\overline{X} = \overline{X}. Consequently \overline{Ad(g)X} = Ad(\overline{g})\overline{X} = \overline{X}. Since dπ is one-to-one, it follows that Ad(g)X = X, as we wished to show. Q.E.D.

Proposition 1.11. Suppose G is a real reductive group with Cartan involution θ.

a) If S is a θ-stable subset of G, then

H = Z_{G}(S) = {g ∈ G | gsg^{−1} = s, all s ∈ S}

is a real reductive group with Cartan involution θ|_{H}.

b) If 𝔰 is a θ-stable subset of 𝔤_{0}, then

H = Z_{G}(𝔰) = {g ∈ G | Ad(g)Y = Y, all Y ∈ 𝔰}

is a real reductive group with Cartan involution θ|_{H}.

Proof. We apply Proposition 1.8. Conditions (a) and (c) are immediate, and (b) is satisfied because of Lemma 1.10. Q.E.D.

A second way to verify condition (2) is using bilinear forms.

Lemma 1.12. Suppose B is a bilinear form on ℝ^{n}, and X is a symmetric n × n matrix. Then the two conditions

1) for all v, w ∈ ℝ^{n}, B(Xv, w) + B(v, Xw) = 0, and

2) for all v, w ∈ ℝ^{n}, B(exp(X)v, exp(X)w) = B(v, w)

are equivalent.

Proof. The conditions may be checked for v and w belonging to eigenspaces of X or exp(X). Write V_{s} for the eigenspace of X with eigenvalue s ∈ ℝ. Then (1) is equivalent to 1′) Whenever s + t ≠ 0, B(V_{s}, V_{t}) = 0.

Similarly, write W_{λ} for the eigenspace of exp(X) with eigenvalue λ > 0. Then (2) is equivalent to

2′) Whenever λμ ≠ 1, B(W_{λ}, W_{μ}) = 0.

Since V_{t} = W_{exp t}, conditions (1′) and (2′) are equivalent. Q.E.D.

Any bilinear form B on ℝ^{n} may be represented by a unique n × n matrix A, by the formula

B(v, w) = ⟨Av, w⟩;  (1.13)(a)

the form on the right is the standard inner product on ℝ^{n}. The symmetry group of the form is

\begin{aligned}G(B) & = \{g ∈ GL(n, ℝ) | B(gv, gw) = B(v, w), all v, w ∈ ℝ^{n}\} \\ & = \{g ∈ GL(n, ℝ) |^{t}gAg = A\}. \end{aligned}  (1.13)(b)

We are interested in constructing real reductive groups as symmetry groups of forms. In order to apply Proposition 1.8, we need to know conditions for G(B) to be θ-stable. Here is a simple one.

Proposition 1.14. Suppose B is a bilinear form on ℝ^{n}, represented by an n×n matrix A. If A^{2} = cI is a non-zero scalar matrix, then the symmetry group G(B) is preserved by the Cartan involution θ of GL(n, ℝ). Consequently G(B) is a real reductive group; the maximal compact subgroup K(B) is the centralizer of A in O(n).

Proof. Recall that θg = ^{t}g^{−1} for g ∈ GL(n, ℝ). The condition in (1.13)(b) for g to belong to G(B) is tgAg = A. By inverting both sides, this condition is equivalent to g^{−1}A^{−1} ^{t}g^{−1} = A^{−1}. The hypothesis on A says that A^{−1} = c^{−1}A; multiplying our condition on g by c gives g^{−1}A^{t}g^{−1} = A, or ^{t}(θg)A(θg) = A; and this is the condition for θg to belong to G(B). To see that G(B) is reductive, apply Definition 1.4: condition (1) is clear, (2) is Lemma 1.12, and we have just established (3). Q.E.D.

Example 1.15. Suppose p and q are non-negative integers, and n = p + q. The standard quadratic form of signature (p, q) on ℝ^{n} is the form

B_{p,q}(v, w) = v_{1}w_{1} + ⋯ + v_{p}w_{p} − v_{p+1}w_{p+1} − ⋯ − v_{n}w_{n}.

The corresponding matrix A_{p,q} is diagonal with p entries equal to 1 and q equal to −1. Obviously A_{p,q}^{2} = I, so the symmetry group of B_{p,q} is a linear real reductive group. It is called O(p, q), the real orthogonal group of signature (p, q). The maximal compact subgroup is O(p) × O(q).

For a second example, the standard symplectic form on ℝ^{2n} is

ω(v, w) = ∑_{i=1}^{n} v_{i}w_{n+i} − v_{n+i}w_{i}.

The corresponding matrix is often called J; it is

J = ( \begin{matrix}0 & −I_{n} \\ I_{n} & 0 \end{matrix} ) .

This is precisely the matrix of multiplication by i in a certain identification of ℝ^{2n} with ℂ^{n}. A first consequence is that J^{2} = −I, so the symmetry group of ω is a linear real reductive group. It is called Sp(2n, ℝ), the real symplectic group. The maximal compact subgroup K(ω) is the centralizer of J in O(2n). The centralizer of J consists precisely of the linear transformations that are complex linear when ℝ^{2n} is identified with ℂ^{n}. We may therefore identify K(ω) with the unitary group U(n).

A third construction of real reductive groups is by changing the base field. Let 𝔽 denote one of the three fields ℝ, ℂ, or ℍ, and put d = dim_{ℝ} 𝔽. The standard basis of 𝔽 (namely {1}, or {1, i}, or {1, i, j, k}) provides an identification of the right vector space 𝔽^{n} with ℝ^{nd}. In particular, each element z ∈ 𝔽 defines by right multiplication a linear transformation ρ(z) ∈ End(ℝ^{nd}). The ℝ-bilinear form on 𝔽^{n} defined by

⟨v, w⟩ = Re (∑_{p=1}^{n} v_{p}\overline{w_{p}})

is just the standard inner product on ℝ^{nd}; here bar denotes the standard anti-automorphism of 𝔽 (acting by +1 on the first basis element of 𝔽 ands by −1 on the others). Obviously

\begin{aligned}⟨ρ(z)v, w⟩ & = ⟨vz, w⟩ = Re (∑_{p=1}^{n} v_{p}z\overline{w_{p}}) \\ & = Re (∑_{p=1}^{n} v_{p}\overline{(w_{p}\overline{z})}) = ⟨v, w\overline{z}⟩ = ⟨v, ρ(\overline{z})w⟩ \end{aligned}  (1.16)(a)

So

ρ(\overline{z}) = ^{t}ρ(z),  (1.16)(b)

and

ρ(\overline{z}^{−1}) = θ(ρ(z))  (1.16)(c)

for z ∈ 𝔽 non-zero.

Now the algebra 𝔤𝔩(n, 𝔽) of n × n matrices over 𝔽 may be identified with the algebra of 𝔽-linear transformations of 𝔽^{n}, and so with the ℝ-linear transformations of ℝ^{nd} commuting with right multiplications by 𝔽; that is, with the centralizer of ρ(𝔽) (or ρ(𝔽^{×})) in 𝔤𝔩(nd, ℝ). In particular,

GL(n, 𝔽) = centralizer of ρ(𝔽^{×}) in GL(nd, ℝ);  (1.17)

this is a linear real reductive group by (1.16)(c) and Proposition 1.11. More generally, we have

Proposition 1.18. Suppose 𝔽 is a field of dimension d over ℝ, and G is a linear real reductive group in GL(nd, ℝ). Then G ∩ GL(n, 𝔽) (cf. (1.17)) is a linear real reductive group.

This follows from Proposition 1.8, Lemma 1.9, and (1.16)(c).

Example 1.19. Suppose p and q are non-negative integers, and n = p + q. The standard Hermitian form of signature (p, q) on ℂ^{n} is the form

H_{p,q}(v, w) = v_{1}\overline{w_{1}} + ⋯ + v_{p}\overline{w_{p}} − v_{p+1}\overline{w_{p+1}} − ⋯ − v_{n}\overline{w_{n}}.

The group of complex-linear transformations preserving this form is called U(p, q), the unitary group of signature (p, q). If we identify ℂ^{n} with ℝ^{2n}, then the real part of H_{p,q} is the quadratic form B_{2p,2q} of Example 1.15. It is easy to check that a complex-linear transformation preserving the real part of H_{p,q} must preserve the entire form; so it follows that

U(p, q) = O(2p, 2q) ∩ GL(n, ℂ).

By Proposition 1.18, this is a linear real reductive group. The proof shows that its maximal compact subgroup is

(O(2p) × O(2q)) ∩ GL(n, ℂ) = U(p) × U(q).

2. Representations and operator algebras.

Suppose G is a topological group. A unitary representation of G is a pair (π, H), with H a complex Hilbert space and π : G → U(H) a homomorphism into the group of unitary operators on H. These are the invertible operators preserving the inner product: the assumption is

⟨π(g)v, π(g)w⟩ = ⟨v, w⟩  (v, w ∈ H, g ∈ G)  (2.1)(a)

Often it is convenient to formulate this condition as

⟨π(g)v, w⟩ = ⟨v, π(g^{−1})w⟩  (v, w ∈ H, g ∈ G)  (2.1)(b)

or simply as π(g)^{∗} = π(g^{−1}). We assume also that π is weakly continuous; that is, that the map

G × H → H  (g, v) ↦ π(g)v  (2.1)(c)

is continuous. An invariant subspace of H is a closed subspace H_{0} ⊂ H that is preserved by all the operators π(g). In this case the restricted operators define a unitary representation (π_{0}, H_{0}) of G. The orthogonal complement H_{1} of H_{0} is a second invariant subspace, and H = H_{0} ⊕ H_{1}; we write π = π_{0} ⊕ π_{1} accordingly. We say that π is irreducible if H ≠ 0, and the only invariant subspaces of H are 0 and H.

Suppose (π_{1}, H_{1}) and (π_{2}, H_{2}) are unitary representations of G. An intertwining operator from π_{1} to π_{2} is a continuous linear map T from H_{1} to H_{2} with the property that

T π_{1}(g) = π_{2}(g)T, (g ∈ G).

The space of all intertwining operators is written Hom_{G}(H_{1}, H_{2}). Forming the adjoint T^{∗} defines a conjugatelinear isomorphism

Hom_{G}(H_{1}, H_{2}) ≃ Hom_{G}(H_{2}, H_{1}).

If some intertwining operator is a unitary isomorphism, then we say that π_{1} and π_{2} are equivalent. We write Ĝ for the set of equivalence classes of irreducible unitary representations of G.

The fundamental problem of abstract harmonic analysis is this: to decompose into irreducible representations an arbitrary unitary representation of G. This problem has a fairly good abstract answer for a large class of groups including the real reductive Lie groups. The case of compact groups is recalled at (2.16) below. The abstract answer is of little help in finding explicit decompositions for particular interesting representations, however, and this remains an active area of research.

A second basic problem in abstract harmonic analysis is the determination of the set Ĝ. If this can be accomplished, a question about G may sometimes be analyzed along the following lines. First, the question is made into one about some unitary representation (π, H) of G (perhaps on square-integrable functions on a homogeneous space, for example). Next, the representation π is decomposed into irreducible representations. At the same time one tries to make the original question into a family of questions about the irreducible constituents of π. Finally, this family of questions is answered using our knowledge of all irreducible representations. Of course each step of this program is fraught with peril; but it has been carried out with some success in a variety of cases. A striking example quite close to the topic of these notes is provided by the book [Borel-Wallach], which analyzes the cohomology of cocompact discrete subgroups of real reductive Lie groups.

In order to carry out the program just sketched, one needs to understand not all irreducible unitary representations, but rather just those appearing in whatever representation π one first constructs. With this in mind, we formulate

Problem. Suppose G is a real reductive Lie group. Construct a family of irreducible unitary representations of G sufficiently large to decompose many interesting unitary representations into irreducibles.

Of course this “problem” is not at all well-defined, because of the phrase “many interesting.” We will make it more precise, and outline what is known about solving it.

In order to explain the approach we will adopt, it is helpful to begin with the case of a finite group G. The group algebra ℂ[G] is the algebra over ℂ with basis {δ_{g}| g ∈ G}, and multiplication table

δ_{g}δ_{h} = δ_{gh}  (g, h ∈ G).  (2.2)(a)

This algebra has a conjugate-linear antiautomorphism ∗, defined by

(∑_{g∈G} a_{g}δ_{g})^{∗} = ∑_{g∈G} \overline{a_{g}}δ_{g^{−1}}  (2.2)(b)

Then a unitary representation (π, H) of G defines an associative algebra homomorphism

π : ℂ[G] → End(H),  π(∑ a_{g}δ_{g}) = ∑ a_{g}π(g),  (2.2)(c)

satisfying

π(a^{∗}) = π(a)^{∗}.  (2.2)(d)

(We say that π is a ∗-homomorphism.) Conversely, any ∗-homomorphism π from ℂ[G] to End(H) arises from a unitary representation of G on H.

Proposition 2.3. Suppose G is a finite group. Then the equivalence classes of irreducible unitary representations π of G are in one-to-one correspondence with the maximal two-sided ideals I_{π} in ℂ[G]. This bijection has the following properties.

a) If (π, H_{π}) is an irreducible unitary representation, then I_{π} is the kernel of the algebra homomorphism π : ℂ[G] → End(H_{π}) of (2.2)(c).

b) Suppose I_{π} is a maximal two-sided ideal in ℂ[G]. Choose a maximal left ideal J_{π} ⊃ I_{π}, and define V_{π} = ℂ[G]/J_{π}. Then V_{π} is isomorphic to H_{π} as a module for ℂ[G].

This is well-known, and not very hard to prove. It is not often used directly to determine the irreducible representations of G, because ℂ[G] is such a complicated algebra. Nevertheless, the proposition suggests a way to approach the representation theory of a group G. First, one should find an associative algebra A(G) whose module theory is related to the representation theory of G. Next, one should study the ideals in A(G).

For a locally compact group G, a natural candidate for the algebra A(G) is the convolution algebra L^{1}(G) (with respect to a left Haar measure dg). Given a unitary representation (π, H) and f ∈ L^{1}(G), one defines π(f) = ∫_{G} f(g)π(g)dg. Then one can make an excellent correspondence between unitary representations of G and appropriate modules for L^{1}(G). These modules all extend to a certain completion of L^{1}(G), called C^{∗}(G). For type I groups G (which include the real reductive Lie groups), there is a bijection between irreducible unitary representations of G and primitive ideals in C^{∗}(G).

Now this bijection is a powerful technical tool — for example, it is at the heart of the abstract theory of decomposition into irreducible representations. But as a way to describe irreducible representations explicitly, it is (like Proposition 2.3) of little value. The algebra C^{∗}(G) is too complicated. We would like instead an algebra whose module theory is perhaps not quite so perfectly related to the unitary representation theory of G, but whose ideal theory we can hope to study directly.

Suppose now that G is a Lie group. Write

𝔤_{0} = Lie(G), 𝔤 = 𝔤_{0} ⊗_{ℝ} ℂ, U(𝔤) = universal enveloping algebra of 𝔤.  (2.4)

One of the central ideas of Lie theory is that these objects can be used to translate problems about Lie groups into linear algebra. Here is an example, along the lines of Proposition 2.3. Recall that a finite-dimensional (non-unitary) representation (π, V_{π}) of G is just a continuous homomorphism π of G into GL(V_{π}), the general linear group of some finite-dimensional complex vector space V_{π}.

Proposition 2.5. Suppose G is a connected, simply connected Lie group. Then the finite-dimensional irreducible representations π of G are in one-to-one correspondence with the two-sided maximal ideals I_{π} of finite codimension in U(𝔤). This bijection has the following properties.

a) Suppose (π, V_{π}) is a finite-dimensional irreducible representation of G. Write π : 𝔤_{0} → 𝔤𝔩(V_{π}) for the differential of π (a Lie algebra representation), and π : U(𝔤) → End(V_{π}) for its extension to U(𝔤) (a homomorphism of associative algebras). Then I_{π} is the kernel of π : U(𝔤) → End(V_{π}).

b) Suppose I_{π} is a maximal two-sided ideal of finite codimension in U(𝔤). Choose a maximal left ideal J_{π} ⊃ I_{π}, and define V_{π} = U(𝔤)/J_{π}. Then the action of 𝔤_{0} on V_{π} by left multiplication in U(𝔤) is a Lie algebra representation; the corresponding group representation of G is isomorphic to π.

Like Proposition 2.3, this is an elementary result. Because U(𝔤) is a relatively uncomplicated algebra, Proposition 2.5 is sometimes even directly useful for studying finite-dimensional representations.

We turn now to the problem of extending Proposition 2.5 to cover unitary representations. A continuous homomorphism between Lie groups is automatically smooth (and even analytic); this is used in Proposition 2.5 to guarantee the existence of the Lie algebra representation π : 𝔤_{0} → 𝔤𝔩(V_{π}). For infinite-dimensional representations the situation is more complicated.

Definition 2.6 Suppose G is a Lie group, and (π, H_{π}) is a unitary representation of G. The vector v ∈ H_{π} is called smooth (respectively analytic) if the map

G → H_{π}, g ↦ π(g)v

is smooth (respectively analytic). Write H_{π}^{∞} (respectively H_{π}^{ω}) for the set of smooth (respectively analytic) vectors in H_{π}. These are G-invariant linear subspaces of H_{π}, but they are usually not closed.

Suppose X ∈ 𝔤_{0}. The differential of π at X is the linear transformation π(X) of H_{π}^{∞} defined by

π(X)v = lim_{t→0}(1/t)(π(exp tX)v − v).

Lemma 2.7. The differential of π is a Lie algebra representation of 𝔤_{0} on H_{π}^{∞}. It therefore defines a homomorphism of associative algebras π : U(𝔤) → End(H_{π}^{∞}). All of the resulting operators preserve the subspace H_{π}^{ω} ⊂ H_{π}^{∞}.

It is not very difficult to show that H_{π}^{∞} is dense in H_{π}. It is also true that H_{π}^{ω} is dense in H_{π}; this much deeper result, which we will use in the proof of Proposition 2.14 below, is due to Nelson. In any case, we can now construct one of the maps of Proposition 2.5 for unitary representations.

Definition 2.8. Suppose G is a Lie group, and (π, H_{π}) is a unitary representation of G. The annihilator of π in U(𝔤) is the kernel of the homomorphism of Lemma 2.7:

Ann(π) = ker(π : U(𝔤) → End(H_{π}^{∞})).

This is a two-sided ideal in U(𝔤); it is equal to U(𝔤) if and only if H_{π} = 0.

As some assurance that this definition is well-behaved, here is an elementary lemma.

Lemma 2.9. In the setting of Definition 2.8, suppose W ⊂ H_{π}^{∞} is any subspace that is dense in H_{π}. Then

Ann(π) = Ann(W) = {u ∈ U(𝔤) | π(u)w = 0, all w ∈ W }.

So we may compute the annihilator on analytic or (when these are defined) K-finite vectors.

What properties can we expect of the annihilator of an irreducible unitary representation? Fairly simple examples show that it need not be a maximal ideal in general. Ring theory has a natural suggestion to offer.

Definition 2.10. Suppose R is a ring with unit element. A left R-module M is called simple if it is not zero, and its only submodules are 0 and M. A two-sided ideal I ⊂ R is called (left) primitive if there is a simple R-module M such that

I = Ann(M) = {r ∈ R | rM = 0}.

Theorem 2.11 (Dixmier [Dixmier]). Suppose G is a connected Lie group and (π, H_{π}) is an irreducible unitary representation of G. Then Ann(π) is a primitive ideal in U(𝔤).

The reason this is not obvious is that H_{π}^{∞} is not a simple U(𝔤) module (unless π is finite-dimensional). For reductive G, we will see in Theorem 2.20 how Harish-Chandra constructs a simple U(𝔤)-submodule of H_{π}^{∞} that is dense in H_{π}. In light of Lemma 2.9, this proves Theorem 2.11 in the reductive case. Dixmier’s proof in general involves several important ideas, so we will outline the easiest part of it.

Lemma 2.12. Suppose G is a connected Lie group, (π, H_{π}) is a unitary representation of G, and V ⊂ H_{π}^{ω} is a U(𝔤)-invariant subspace. Then the closure H_{0} of V in H_{π} is a G-invariant subspace.

Proof. For any subset S of H_{π}, define

S^{⊥} = {w ∈ H_{π} | ⟨w, s⟩ = 0, all s ∈ S}.

The closure of any subspace S may be characterized as (S^{⊥})^{⊥}, so it suffices to show that V^{⊥} is G-invariant. For this, fix w ∈ V^{⊥} and v ∈ V; we must show that the function f(g) = ⟨π(g)w, v⟩ is identically zero. Because π is unitary, f(g) = ⟨w, π(g^{−1})v⟩. Because v is assumed to be an analytic vector, the function f is analytic on G. Because G is connected, it therefore suffices to show that all derivatives of f vanish at the identity on G. A typical derivative of f at the identity is ⟨w, π(u)v⟩, with u ∈ U(𝔤). Since V is assumed to be U(𝔤)-invariant, π(u)v ∈ V. Since w ∈ V^{⊥}, the derivative vanishes, as we wished to show. Q.E.D.

(I am grateful to P. E. Paradan for showing me this elegant argument.)

Lemma 2.13. Suppose G is a connected Lie group, and (π, H_{π}) is an irreducible unitary representation. Then any non-zero U(𝔤)-invariant subspace V ⊂ H_{π}^{ω} is dense in H_{π}. Consequently Ann(V) = Ann(π).

(The last assertion uses Lemma 2.9.)

Proposition 2.14. Suppose G us a connected Lie group, and (π, H_{π}^{ω}) is an irreducible unitary representation. Then Ann(π) is a prime ideal in U(𝔤).

Proof. Recall that a two-sided ideal I in a (possibly non-commutative) ring R is called prime if whenever J_{1} and J_{2} are two-sided ideals with J_{1}J_{2} ⊂ I, then either J_{1} ⊂ I or J_{2} ⊂ I. So suppose J_{1} and J_{2} are ideals in U(𝔤) with J_{1}J_{2} ⊂ Ann(π), but J_{2} ⊄ Ann(π). Then J_{2}H_{π}^{ω} = V is a non-zero U(𝔤)-invariant subspace of H_{π}^{ω}. By Lemma 2.13, Ann(V) = Ann(π). But

J_{1}V = J_{1}J_{2}H_{π}^{ω} ⊂ Ann(π)H_{π}^{ω} = 0;

so J_{1} ⊂ Ann(π), as we wished to show. Q.E.D.

Dixmier completes the proof of Theorem 2.11 using

Theorem 2.15 ([Dixmier]) Suppose I ⊂ U(𝔤) is a prime ideal. Then I is primitive if and only if the center of the ring of fractions of U(𝔤)/I is ℂ.

We omit the details.

We now have a reasonable analogue of Proposition 2.5(a): a map from irreducible unitary representations to primitive ideals in U(𝔤). To get an analogue of (b) (a corresponding parametrization of representations) we need to specialize to real reductive groups. Let us first recall the structure of unitary representations for compact groups.

Suppose K is a compact topological group. Then every irreducible unitary representation of K is finite-dimensional. Fix a model (δ, V_{δ}) for each equivalence class in K̂. If (π, H_{π}) is an arbitrary unitary representation of K, define

H_{π}^{δ} = Hom_{K}(V_{δ}, H_{π})  (2.16)(a)

We make H_{π}^{δ} into a Hilbert space as follows. If T and S belong to H_{π}^{δ}, then S^{∗}T is a map from V_{δ} to V_{δ} commuting with the action of K. By Schur’s lemma it is a scalar operator λI; and we define ⟨T, S⟩ = λ. An equivalent formulation is

⟨T v, Sw⟩_{H_{π}} = ⟨T, S⟩_{H_{π}^{δ}}⟨v, w⟩_{V_{δ}}  (2.16)(b)

for v, w ∈ V_{δ}. Now we can form the Hilbert space tensor product H_{π}^{δ} ⊗ V_{δ}. (Because V_{δ} is finite-dimensional, it coincides with the algebraic tensor product.) There is a natural map

H_{π}^{δ} ⊗ V_{δ} → H_{π},  T ⊗ v ↦ T v;  (2.16)(c)

and (2.16)(b) guarantees that this map preserves inner products. It is therefore an isomorphism onto its image H_{π}(δ), the δ-isotypic subspace of H_{π}. This is the largest subspace of H_{π} on which K acts by a sum of copies of δ. By Schur’s lemma again, H_{π}(δ) and H_{π}(δ′) are orthogonal whenever δ and δ′ are inequivalent. Consequently

H_{π} ≃  ; ⨁_{δ∈_{K}}H_{π}(δ) ≃  ; ⨁_{δ∈_{K}}H_{π}^{δ} ⊗ V_{δ},  (2.16)(d)

the direct sums being Hilbert space direct sums.

A vector v ∈ H_{π} is called K-finite if it is contained in a finite-dimensional K-invariant subspace. We write H_{π}^{K} for the space of K-finite vectors. Using (2.16)(d), we find that

H_{π}^{K} ≃ ⨁_{δ∈_{K}} H_{π}(δ) ≃ ⨁_{δ∈_{K}} H_{π}^{δ} ⊗ V_{δ},  (2.16)(e)

the direct sums now being algebraic rather than Hilbert space. The representation π is called admissible for K if all of the spaces H_{π}(δ) are finite-dimensional; that is, if every irreducible representation of K has finite multiplicity in π.

Suppose now that G is a real reductive Lie group with maximal compact subgroup K. A unitary representation (π, H_{π}) is called admissible if it is admissible for K.

Theorem 2.17 (Harish-Chandra [Harish-Chandra53]). Every irreducible unitary representation of a real reductive Lie group G is admissible.

This is a rather difficult result, relying on a deep study of the adjoint action of K on U(𝔤).

Admissible unitary representations have an excellent algebraic description.

Theorem 2.18 (Harish-Chandra [Harish-Chandra53]). Suppose (π, H_{π}) is an admissible unitary representation of a real reductive Lie group G. Write H_{π}^{K} for the space of K-finite vectors in H_{π}.

a) H_{π}^{K} is a U(𝔤)-invariant subspace of the analytic vectors H_{π}^{ω}. In particular, H_{π}^{K} carries representations of the group K and the Lie algebra 𝔤.

b) There is a bijection between closed G-invariant subspaces of H_{π} and arbitrary (𝔤, K)-invariant subspaces of H_{π}^{K}. The correspondence from left to right sends a closed subspace W to W^{K} (the K-finite vectors in W); from right to left it sends V to \overline{V} (the closure of V in H_{π}).

To complete this circle of ideas, we will describe formally the algebraic objects arising in Theorem 2.18. (The definition is taken from [Lepowsky].)

Definition 2.19. Suppose G is a real reductive Lie group with maximal compact subgroup K. A (𝔤, K)-module is a complex vector space V endowed with representations of the Lie algebra 𝔤 and the group K, subject to the following conditions.

1) The action of K is locally finite and smooth. That is, every v ∈ V belongs to a finite-dimensional K-invariant subspace F ⊂ V, and the action of K on F is smooth.

2) The differential of the action of K (which makes sense by (1)) is equal to the action of 𝔨_{0} = Lie(K) ⊂ 𝔤.

3) For k ∈ K, v ∈ V, and X ∈ 𝔤, we have

k · (X · v) = Ad(k)(X) · (k · v).

For k in the identity component K_{0} of K, condition (3) is a consequence of (1) and (2). We may therefore omit condition (3) when K (or, equivalently, G) is connected.

A (𝔤, K)-submodule of V is a complex subspace W invariant under the representations of 𝔤 and K. (By conditions (1) and (2), invariance under K_{0} follows from invariance under 𝔤. If K is connected, a (𝔤, K)- submodule is therefore just a 𝔤-submodule.) We say that V is irreducible if it is not zero, and the only submodules are 0 and V. Finally, we say that V is unitary if it is endowed with a positive definite Hermitian form ⟨, ⟩ satisfying

⟨k · v, k · w⟩ = ⟨v, w⟩, ⟨X · v, w⟩ + ⟨v, X · w⟩ = 0

for v, w ∈ V, k ∈ K, and X ∈ 𝔤_{0}.

Theorem 2.18(a) guarantees that the space H_{π}^{K} of K-finite vectors in an admissible unitary representation is a unitary (𝔤, K)-module; it is called the Harish-Chandra module of π. Theorems 2.17 and 2.18(b) say that H_{π}^{K} is irreducible as a (𝔤, K)-module whenever π is irreducible. Harish-Chandra’s last basic result is a converse.

Theorem 2.20 (Harish-Chandra [Harish-Chandra53]). Suppose G is a real reductive Lie group. The map π ↦ H_{π}^{K} is a bijection from (equivalence classes of) irreducible unitary representations of G onto (equivalence classes of) irreducible unitary (𝔤, K)-modules.

When G is connected, we have seen that an irreducible (𝔤, K)-module is irreducible as a representation of 𝔤. This theorem may therefore be viewed as an infinite-dimensional analogue of Proposition 2.5(b). It provides a construction of irreducible unitary representations of G from certain (very special) irreducible U(𝔤)-modules.

We can now refine slightly the problem formulated before (2.2).

Problem 2.21. Suppose G is a real reductive Lie group; assume for simplicity that G is connected.

a) Construct a family of interesting primitive ideals I ⊂ U(𝔤).

b) For each primitive ideal I as in (a), construct a finite set of irreducible unitary representations π of G, satisfying Ann(π) = I (Definition 2.8).

This formulation is still imperfect, but it begins to reflect what we will actually do. The constructions in (b) will generally take place in three steps. First, we will construct some U(𝔤)-modules W with annihilator equal to I. This step is usually fairly easy; some possibilities for W will often be suggested by the construction of I in (a). Next, we will construct from each W a (𝔤, K)-module V, still annihilated by I. The principle of this construction (due to Zuckerman) is simple and elegant, but analyzing it in detail can be quite difficult. (One minor point is that the annihilator of V may be strictly larger than I.) Finally, we will apply Harish-Chandra’s Theorem 2.20 to get a unitary representation of G.

3. Primitive ideals and Dixmier algebras.

In this section we will consider more carefully part (a) of Problem 2.21: the construction of a family of primitive ideals related to unitary representations. A more detailed account may be found in [Vogan86] and [Vogan90].

The first point is that not every primitive ideal can be the annihilator of a unitary representation. Suppose for a moment that G is a connected noncompact simple Lie group. Then G has a large family of irreducible finite-dimensional representations (parametrized by a cone in a lattice of dimension equal to the rank of G). By Proposition 2.5, it follows that U(𝔤) has a large family of maximal ideals of finite codimension. On the other hand, a unitary finite-dimensional representation is a homomorphism π : G → U(n). Because G is noncompact and simple, such a homomorphism must be trivial. This proves

Lemma 3.1. Suppose G is a connected noncompact simple Lie group, and suppose I ⊂ U(𝔤) is a maximal ideal of finite codimension. Then I is the annihilator of a unitary representation if and only if I = 𝔤U(𝔤) (the augmentation ideal).

What distinguishes the augmentation ideal among all maximal ideals of finite codimension? If I is such an ideal, then the Wedderburn theorem guarantees that

U(𝔤)/I ≃ M_{n}(ℂ),  (3.2)

the algebra of n × n matrices. Since 𝔤 is semisimple, n = 1 occurs only for the augmentation ideal. So we may ask what distinguishes 1 × 1 matrices from larger ones. One answer is the absence of zero divisors.

Definition 3.3. Suppose I is a two-sided ideal in a ring R. We say that the ideal I (or the quotient ring R/I) is completely prime if whenever a and b are elements of R with ab ∈ I, then either a ∈ I or b ∈ I.

It is easy to check that a completely prime ideal is prime. (The definition of prime was included in the proof of Proposition 2.14.) Here is some further evidence of the connection between completely prime ideals and unitary representations.

Proposition 3.4 ([Vogan86], Proposition 7.12). Suppose G is a connected complex reductive Lie group, and π ∈ Ĝ is an irreducible unitary representation. Then the annihilator I_{π} ⊂ U(𝔤) (Definition 2.8) is completely prime.

The proof is very easy, requiring no structural information about π. Exactly the same result is true for G = GL(n, ℝ); but in this case the proof requires a complete and detailed knowledge of Ĝ. Any hopes of further generalization are dashed on the rocks of G = SU(2). This group has an irreducible unitary representation of each dimension n > 0; and (3.2) guarantees that the corresponding primitive ideal is completely prime only for n = 1. (More subtle examples are available for noncompact simple groups as well. Suppose π is a holomorphic discrete series representation of Sp(4, ℝ) with Harish-Chandra parameter λ = (λ_{1}, λ_{2}). This means that λ_{1} > λ_{2} > 0 are positive integers. It turns out that Ann(π) is completely prime if and only if λ_{1} − λ_{2} = 1.)

We can find a way out of this disappointment by looking carefully at the example of G = SU(2). This group acts holomorphically on the Riemann sphere ℂℙ^{1}. The n-dimensional irreducible representation π_{n} arises as the space of holomorphic sections of a certain holomorphic line bundle L_{n} → ℂℙ^{1}. Define D(ℂℙ^{1})_{n} to be the algebra of holomorphic differential operators on sections of L_{n}. This is an algebra of “twisted differential operators” on ℂℙ^{1}. (We will return to a more detailed and general discussion of twisted differential operator algebras in section 9.) The action of G on L_{n} defines a homomorphism of associative algebras

ϕ_{n} : U(𝔤) → D(ℂℙ^{1})_{n}.  (3.5)

Write I_{n} for the kernel of ϕ_{n}.

Proposition 3.6. The ideal I_{n} is a completely prime primitive ideal in U(𝔤), contained in the annihilator Ann(π_{n}).

Sketch of proof. That I_{n} is completely prime follows from the absence of zero divisors in the twisted differential operator algebra D(ℂℙ^{1})_{n}. That it is contained in Ann(π_{n}) follows from the realization of π_{n} as holomorphic sections of L_{n}. Q.E.D.

The lesson to be drawn from this example is that an interesting unitary representation π may appear naturally as a module for some completely prime quotient U(𝔤)/I, even if Ann(π) properly contains I. It is easy to modify Problem 2.21 in accordance with this lesson; we simply weaken the condition in (b) to Ann(π) ⊃ I.

There is a hint here of a second lesson as well. The realization of π_{n} on holomorphic sections of L_{n} exhibits π_{n} as a module not only for U(𝔤)/I_{n}, but also for the full differential operator algebra D(ℂℙ^{1})_{n}. In this example the homomorphisms ϕ_{n} are all surjective, so that there is no difference between U(𝔤)/I_{n} and D(ℂℙ^{1})_{n}. When we treat general reductive groups, however, we will encounter homomorphisms

ϕ_{λ} : U(𝔤) → D(X)_{λ}.  (3.7)

Here X is a “partial flag variety” for 𝔤 (a quotient of a complex reductive group G_{ℂ} by a parabolic subgroup Q_{ℂ}); λ is a character of the Lie algebra 𝔮; and D(X)_{λ} is a twisted differential operator algebra on X. In this setting the homomorphism ϕ_{λ} is usually but not always surjective. Perhaps the simplest example when ϕ_{λ} is not surjective has 𝔤 = 𝔰𝔭(4, ℂ), X = ℂℙ^{3} (the variety of lines in the natural four-dimensional representation of 𝔤), and D(X)_{λ} the algebra of differential operators on “half forms” on X. (The top exterior power of the cotangent bundle of X has a well-defined square root L → X in this example; D(X)_{λ} is the algebra of holomorphic differential operators on sections of L.) In any case, we will construct modules for D(X)_{λ}, and not just for U(𝔤)/ker ϕ_{λ}. In order to accommodate this extra structure in something like Problem 2.21, we need a definition.

Definition 3.8. Suppose G_{ℂ} is a complex reductive algebraic group with Lie algebra 𝔤. A Dixmier algebra for G_{ℂ} is a pair (A, ϕ) satisfying the following conditions.

i) A is an algebra over ℂ, equipped with an algebraic action of G_{ℂ} by algebra automorphisms Ad(g). ii) The map ϕ : U(𝔤) → A is an algebra homomorphism, respecting the adjoint actions of G_{ℂ} on U(𝔤) and

A. The differential ad of the adjoint action of G_{ℂ} on A is the difference of the left and right actions of 𝔤 defined by ϕ:

ad(X)(a) = ϕ(X)a − aϕ(X) (X ∈ 𝔤, a ∈ A).

iii) A is finitely generated as a U(𝔤) module. iv) Each irreducible G_{ℂ}-module occurs at most finitely often in the adjoint action of G_{ℂ} on A.

We say that the Dixmier algebra is completely prime if A is a completely prime algebra. This immediately implies that the kernel I of ϕ is a completely prime ideal in U(𝔤), and one can show (using condition (iv)) that I must also be primitive.

If I is a primitive ideal in U(𝔤), then U(𝔤)/I is a Dixmier algebra for any connected G_{ℂ}. In the setting of (3.7), D(X)_{λ} is a completely prime Dixmier algebra for G_{ℂ}. The adjoint action arises from the action of G_{ℂ} on X, by change of variable in the differential operators.

It is an idea of Dixmier that for a connected complex algebraic group G_{ℂ} there should be a close connection between completely prime primitive ideals in U(𝔤) and orbits of G_{ℂ} on 𝔤^{∗}. Borho, Joseph, and others found that for reductive G_{ℂ}, this “close connection” can not be a reasonable bijection. The goal of [Vogan86] was to find a geometric description of all completely prime primitive ideals. That attempt failed, as was shown by work of McGovern. Here is a weaker statement (taken from [Vogan90], Conjecture 2.3) that appears to be consistent with everything we now understand about Dixmier algebras.

Conjecture 3.9. Suppose G_{ℂ} is a connected complex reductive algebraic group. Let O ⊂ 𝔤^{∗} be a coadjoint orbit for G_{ℂ}, and let Õ → O be a connected covering on which G_{ℂ} acts compatibly. Write R(Õ) for the algebra of regular functions on Õ; this algebra carries a natural algebraic action of G_{ℂ} by algebra automorphisms. Attached to Õ there should be a completely prime Dixmier algebra (A(Õ), ϕ(Õ)), with the property that A(Õ) is isomorphic to R(Õ) as algebraic representations of G_{ℂ}. The “Dixmier correspondence” Õ ↦ (A(Õ), ϕ(Õ)) should be injective.

The requirements of the conjecture are very far from specifying A(Õ) completely; but fortunately they are most restrictive precisely in those cases when we know least about how to construct A(Õ).

The following result allows us to relate Conjecture 3.9 to primitive ideals.

Lemma 3.10. Suppose G_{ℂ} is a complex reductive algebraic group, and (A, ϕ) is a completely prime Dixmier algebra for G_{ℂ}. Then ker ϕ = I(A, ϕ) is a completely prime primitive ideal in U(𝔤).

Proof. Since A is completely prime, so is its subalgebra U(𝔤)/I(A, ϕ). So I(A, ϕ) is a completely prime ideal, and therefore prime; we need only show it is primitive. Write Z(𝔤) for the center of U(𝔤). As a consequence of Theorem 2.15, I(A, ϕ) will be primitive if and only if I(A, ϕ) ∩ Z(𝔤) is a maximal ideal in Z(𝔤); that is, if and only if

ϕ(Z(𝔤)) ⊂ ℂ.  (3.11)(a)

Now

Z(𝔤) = {z ∈ U(𝔤) | Ad(g)(z) = z, all z ∈ (G_{ℂ})_{0}}.  (3.11)(b)

Let A_{0} be the subalgebra of A on which (G_{ℂ})_{0} acts trivially. By Definition 3.8(iv), A_{0} is finite-dimensional. Since it is also a completely prime algebra over ℂ, we must have A_{0} = ℂ. Since (3.11)(b) guarantees that ϕ(Z(𝔤)) ⊂ A_{0}, (3.11)(a) follows. Q.E.D.

In the setting of Conjecture 3.9, we write

I(Õ) = ker ϕ(Õ) ⊂ U(𝔤)  (3.12)

for the completely prime primitive ideal provided by the conjecture and Lemma 3.10. The correspondence sending O to {I(Õ) | Õ a cover of O} is (conjecturally) a kind of multi-valued Dixmier correspondence from coadjoint orbits to completely prime primitive ideals.

From the point of view of primitive ideal theory, the most serious problem with Conjecture 3.9 is that this correspondence is not surjective: not every completely prime Dixmier algebra is of the form A(Õ) for a coadjoint orbit cover Õ. Even the underlying correspondence (3.12) to completely prime primitive ideals is not surjective. To understand why this is not entirely bad, we recall an example from [Joseph] and [Vogan86].

Suppose G_{ℂ} is of type G_{2}. There is exactly one coadjoint orbit O_{8} in 𝔤^{∗} of dimension 8, and it is simply connected. Joseph found a (unique) completely prime primitive ideal I(O_{8}) with the property that U(𝔤)/I(O_{8}) is isomorphic to R(O_{8}) as representations of G_{ℂ}. It is therefore reasonable to define A(O_{8}) = U(𝔤)/I(O_{8}) as the Dixmier algebra predicted by Conjecture 3.9.

Let G be a simply connected split real reductive Lie group of type G_{2}. It turns out that G has exactly one irreducible (𝔤, K)-module V with Ann(V) = I(O_{8}). This (𝔤, K)-module corresponds to an isolated unitary representation π; in the classification of Ĝ given in [Vogan94], π is the unique isolated point among the Langlands quotients of the principal series for the non-linear group. It is constructed in [Vogan94] as the restriction to G of a ladder representation of S̃O(4, 3). Certainly π is an interesting unitary representation of G, and evidence that the approach of Problem 2.21 will find it is welcome.

Joseph found a second completely prime primitive ideal I′(O_{8}) closely related to O_{8}. (Under the adjoint action of G_{ℂ}, U(𝔤)/I(O_{8}) is slightly smaller than R(O_{8}).) The Dixmier correspondence of Conjecture 3.9 (or even (3.12)) has no room for I′(O_{8}), however; this ideal is simply omitted. There is exactly one irreducible (𝔤, K)-module V′ with Ann(V′) = I′(O_{8}). But it turns out that V′ is not unitary; so from the point of view of finding unitary representations, the omission of I′(O_{8}) is harmless (or even desirable).

Encouraged by this example, we are going to refine Problem 2.21 to accommodate Dixmier algebras. Here is the setting.

Definition 3.13. Suppose G is a real reductive Lie group. Let G_{ℂ} be a connected complex reductive algebraic group with Lie algebra 𝔤 = Lie(G)_{ℂ}. We assume that the adjoint action of G on 𝔤 factors through a homomorphism

j : G → G_{ℂ}

inducing the identity map on 𝔤. (This can always be arranged by an appropriate choice of G_{ℂ} if G is of “inner type:” that is, if every automorphism Ad(g) (for g ∈ G) of 𝔤 is inner.) In this setting we say that G is of inner type G_{ℂ}.

Suppose now that (A, ϕ) is a Dixmier algebra for G_{ℂ}. An (A, K)-module is a complex vector space endowed with a module structure for the algebra A and a representation of the group K, subject to the following conditions. (Compare Definition 2.19.)

1) The action of K is locally finite and smooth.

2) The differential of the action of K (which makes sense by (1)) is equal to the action of ϕ(𝔨_{0}) ⊂ A.

3) For k ∈ K, v ∈ V, and a ∈ A, we have

k · (a · v) = [Ad(j(k))(a)] · (k · v).

(Just as in Definition 2.19, condition (3) for k ∈ K_{0} is a consequence of (1) and (2).) A Hermitian transpose on A is a conjugate-linear antiautomorphism ∗ of A of order 2:

(ab)^{∗} = b^{∗}a^{∗}, (za)^{∗} = \overline{z}a^{∗} (a, b ∈ A, z ∈ ℂ).

We assume in addition that ∗ is compatible with the usual Hermitian transpose on U(𝔤) defined by the real form 𝔤_{0}:

[ϕ(X + iY)]^{∗} = ϕ(−X + iY) (X, Y ∈ 𝔤_{0}).

Suppose finally that V is an (A, K) module and that ∗ is a Hermitian transpose on A. We say that V is unitary if it is endowed with a positive definite Hermitian form ⟨, ⟩ satisfying

⟨k · v, k · w⟩ = ⟨v, w⟩, ⟨a · v, w⟩ = ⟨v, a^{∗} · w⟩

for v, w ∈ V, k ∈ K, and a ∈ A.

The map ϕ provides a forgetful functor that makes any (unitary) (A, K)-module into a (unitary) (𝔤, K)- module. This functor sends (A, K)-modules of finite length to (𝔤, K)-modules of finite length, but it need not send irreducibles to irreducibles.

Theorem 3.14. In the setting of Definition 3.13, suppose V is a unitary (A, K)-module. Then the Hilbert space completion H(V) carries a unitary representation π(V) of G. The space H(V)^{∞} carries a natural action of A. This action preserves the space H(V)^{K} of K-finite vectors (which are automatically smooth), making H(V)^{K} an (A, K)-module. If V has finite length, then H(V)^{K} is equal to V as an (A, K)- module.

This is an immediate consequence of Harish-Chandra’s Theorem 2.20.

Here is a refinement of Problem 2.21.

Problem 3.15. Suppose G is real reductive Lie group of inner type G_{ℂ} (Definition 3.13). Let O ⊂ 𝔤^{∗} be a coadjoint orbit for G_{ℂ}, and let Õ → O be a connected covering on which G_{ℂ} acts compatibly.

a) Construct a completely prime Dixmier algebra (A(Õ), ϕ(Õ)) as in Conjecture 3.9. b) Construct a finite collection {W_{i}(Õ) | i = 1, …, r} of modules for A(Õ).

c) For each i, construct from W_{i}(Õ) an (A(Õ), K)-module V_{i}(Õ) (Definition 3.13).

d) For each i, construct a Hermitian transpose ∗ on A(Õ) (Definition 3.13) and a unitary structure ⟨, ⟩ on V_{i}(Õ) (Definition 3.13).

If all these steps are completed, then Theorem 3.14 makes the Hilbert space completion H_{i}(Õ) of V_{i}(Õ) into a unitary representation of G.

The method of coadjoint orbits suggests roughly how the modules W_{i}(Õ) should be parametrized. Write 𝔤_{0}^{∗} for the real dual of the real Lie algebra of G. It is contained in 𝔤^{∗} as a real form.

Lemma 3.16. In the setting of Problem 3.15, the intersection O(ℝ) = O ∩ 𝔤_{0}^{∗} is a finite union of orbits of G:

O(ℝ) = O_{1}(ℝ) ∪ ⋯ ∪ O_{s}(ℝ).

The covering map Õ → O induces (possibly disconnected) G-equivariant coverings

Õ_{i}(ℝ) → O_{i}(ℝ)

having the same degree as Õ.

This is elementary. What the orbit method suggests is that each of the modules in Problem 3.15 should be parametrized by one of the real orbit covers Õ_{i}(ℝ), together with some additional data. (This idea can be made precise and correct for semisimple orbits, but it requires further refinement in general.) In section 4 we will begin to study real coadjoint orbits, and to see how one might attach representations to them.

4. Structure of coadjoint orbits.

In this section we recall some general structure theory for coadjoint orbits. We work at first with an arbitrary real Lie group G, writing

𝔤_{0} = Lie(G),  𝔤_{0}^{∗} = Hom_{ℝ}(𝔤_{0}, ℝ).  (4.1)(a)

We write G_{0} for the identity component of G. Often we write elements of G as lower case Roman letters, elements of 𝔤_{0} as upper case Roman letters, and elements of 𝔤_{0}^{∗} as lower case Greek letters. The coadjoint action of G on 𝔤_{0}^{∗} is just the transpose of the adjoint action:

[Ad^{∗}(g)ξ](Y) = ξ(Ad(g^{−1})Y)  (ξ ∈ 𝔤_{0}^{∗}, g ∈ G, Y ∈ 𝔤_{0}).  (4.1)(b)

The differential of this action is written

ad^{∗} : 𝔤_{0} → End(𝔤_{0}^{∗}),  [ad^{∗}(X)ξ](Y) = ξ(ad(−X)Y) = ξ([Y, X]).  (4.1)(c)

The isotropy group for Ad^{∗} at ξ is written G_{ξ}:

G_{ξ} = {g ∈ G | Ad^{∗}(g)ξ = ξ}  (4.1)(d)

The Lie algebra of G_{ξ} is

\begin{aligned}𝔤_{ξ,0} & = \{X ∈ 𝔤_{0} | ad^{∗}(X)ξ = 0\} \\ & = \{X ∈ 𝔤_{0} | ξ([Y, X]) = 0, all Y ∈ 𝔤_{0}\}. \end{aligned}  (4.1)(e)

To each ξ ∈ 𝔤_{0}^{∗} we attach a skew-symmetric bilinear form ω_{ξ} on 𝔤_{0}, defined by

ω_{ξ}(X, Y) = ξ([X, Y]) = [ad^{∗}(Y)ξ](X) = [−ad^{∗}(X)ξ](Y)  (4.1)(f)

Lemma 4.2. With notation as in (4.1), the radical of ω_{ξ} is equal to 𝔤_{ξ,0}. Consequently ω_{ξ} descends to a non-degenerate symplectic form (still denoted ω_{ξ}) on

𝔤_{0}/𝔤_{ξ,0} ≃ T_{ξ}(G · ξ)

(the tangent space at ξ to the coadjoint orbit through ξ). As ξ′ varies over the orbit W = G · ξ, the family of ω_{ξ′} defines a closed two-form ω_{W}, and therefore a G-invariant symplectic structure on W.

Proof. That the radical of ω_{ξ} is 𝔤_{ξ,0} is clear from (4.1)(e) and (4.1)(f). It follows that ω_{W} is a two-form on W. Each element X ∈ 𝔤_{0} defines a vector field X_{W} on W. These span the tangent space T W at each point; so to prove that ω_{W} is closed, it suffices to show that dω_{W} (X_{W} , Y_{W} , Z_{W}) = 0 for all X, Y, Z ∈ 𝔤_{0}. We compute

\begin{matrix}dω_{W} (X_{W} , Y_{W} , Z_{W}) = & X_{W} · ω_{W} (Y_{W} , Z_{W}) − Y_{W} · ω_{W} (X_{W} , Z_{W}) + Z_{W} · \\ ω_{W} (X_{W} , Y_{W}) − ω_{W} ([X_{W} , Y_{W}], Z_{W}) + ω_{W} ([X_{W} , Z_{W}], Y_{W}) − ω_{W} ([Y_{W} , Z_{W}], X_{W}). \end{matrix}

Evaluating at the point ξ′ ∈ W, we get

= X_{W} · ξ′([Y, Z]) − Y_{W} · ξ′([X, Z]) + Z_{W} · ξ′([X, Y]) − ξ′([[X, Y], Z]) + ξ′([[X, Z], Y]) − ξ′([[Y, Z], X]).

In the first three terms, we are differentiating the function on W obtained by applying the variable linear functional ξ′ to a fixed element of 𝔤_{0}. This amounts to applying −ad^{∗} to ξ′. We get

= −(ad^{∗}(X)ξ′)([Y, Z]) + (ad^{∗}(Y)ξ′)([X, Z]) − (ad^{∗}(Z)ξ′)([X, Y])
− ξ′([[X, Y], Z] − [[X, Z], Y] + [[Y, Z], X])
= −ξ′([[Y, Z], X]) + ξ′([[X, Z], Y]) − ξ′([[X, Y], Z]) − ξ′([[X, Y], Z] − [[X, Z], Y] + [[Y, Z], X])
= −2ξ′([[X, Y], Z] − [[X, Z], Y] + [[Y, Z], X]).

The argument of ξ′ vanishes by the Jacobi identity, so ω_{W} is closed. The G-invariance is clear from the definition. Q.E.D.

Lemma 4.2 says that any coadjoint orbit is in a natural way a symplectic homogeneous space. The converse (that any symplectic homogeneous space is a coadjoint orbit) is not quite true, for two reasons. First, a coadjoint orbit has a slightly stronger structure (which we will describe in a moment). Second, this additional structure lifts to covering spaces.

Definition 4.3. Suppose (W, ω) is a symplectic manifold. The symplectic form provides a smooth identification of the tangent bundle of W with the cotangent bundle: to the tangent vector X ∈ T_{w}(W) we associate the cotangent vector τ(X) defined by

τ(X)(Y) = ω_{w}(Y, X) (Y ∈ T_{w}(W));

this is the contraction of −ω_{w} with X. If f is a smooth function on W, then df is a one-form (a smooth section of the cotangent bundle). We may therefore define

X_{f} = τ^{−1}(df),

a smooth vector field on W, called the Hamiltonian vector field of f. Using the action of vector fields on functions, we now define the Poisson bracket of the smooth functions f and g by

{f, g} = X_{f} · g = dg(X_{f}) = ω(X_{f}, X_{g}) = −X_{g} · f.

Proposition 4.4. The Poisson bracket defines a Lie algebra structure on C^{∞}(W). The map f ↦ X_{f} is a Lie algebra homomorphism from C^{∞}(W) to the Lie algebra of vector fields on W. Its kernel consists of the locally constant functions on W.

Suppose W is a coadjoint orbit in 𝔤_{0}^{∗}, and Y ∈ 𝔤_{0}. Write f(Y) for the smooth function on W obtained by restricting to W the linear function Y on 𝔤_{0}^{∗}. Then the corresponding Hamiltonian vector field X_{f(Y)} is equal to the vector field Y_{W} induced by the action of G on W. The map Y ↦ f(Y) is a Lie algebra homomorphism.

Proof. The assertions about general symplectic manifolds are standard (see for example [Arnold], Chapter 8, or [Abraham-Marsden], Chapter 3; both sources use slightly different sign conventions from ours). For the rest, we compute (for Y, Z ∈ 𝔤_{0})

ω_{W} (Z_{W} , Y_{W}) = f([Z, Y])  (definition of ω_{W})

= Z_{W} · f(Y) (as at the end of the proof of Lemma 4.2) = df(Y)(Z_{W}) = τ(X_{f(Y)})(Z_{W}) (definition of X_{f(Y)})

= ω_{W} (Z_{W} , X_{f(Y)}) (definition of τ).

Because ω_{W} is non-degenerate, and the vector fields Z_{W} span each tangent space, it follows that Y_{W} = X_{f(Y)}. At the same time we have shown that

{f(Z), f(Y)} = X_{f(Z)} · f(Y) = Z_{W} · f(Y) = f([Z, Y]),

proving the last assertion. Q.E.D.

Definition 4.5 (see [Kostant], section 5). A Hamiltonian G-space is a symplectic manifold W equipped with a symplectic action of G and a linear map f : 𝔤_{0} → C^{∞}(W, ℝ), with the following properties.

a) The map f is a Lie algebra homomorphism (for the Poisson bracket Lie algebra structure).

b) The Hamiltonian vector field X_{f(Y)} on W associated to Y ∈ 𝔤_{0} is equal to the vector field Y_{W} obtained by differentiating the action of G in the direction Y.

c) The map f is G-equivariant:

f(Ad(g)Y)(w) = f(Y)(g^{−1} · w).

Condition (a) is actually a consequence of (b) and (c), but we include it because of its appealing simplicity. For g in the identity component G_{0} of G, condition (c) is a consequence of (a) and (b). Condition (b) also guarantees that the action of G_{0} is symplectic; so the entire definition may be phrased more succinctly for connected G.

Suppose (W, f) is a Hamiltonian G-space. The moment map for W is the G-equivariant smooth map

μ : W → 𝔤_{0}^{∗}, μ(w)(Y) = f(Y)(w).

Proposition 4.4 implies that each coadjoint orbit is a Hamiltonian G-space (the requirement in (c) being easy to verify). Its moment map is the identity. It is also easy to see that a G-equivariant covering of a Hamiltonian G-space is again a Hamiltonian G-space. The following result is a partial converse.

Proposition 4.6 ([Kostant], Theorem 5.4.1). Suppose W is a homogeneous Hamiltonian G-space. Then the moment map is a covering of a coadjoint orbit G · ξ. Consequently W ≃ G/G_{ξ,1}, with G_{ξ,1} an open subgroup of the isotropy group G_{ξ}.

Proof. Since μ is G-equivariant, its image must be a single orbit G·ξ. Fix a point w ∈ W with μ(w) = ξ, and define G_{ξ,1} to be the isotropy group at w. Obviously this is a subgroup of G_{ξ}; we need only show it is open. This amounts to showing that the differential of μ is one-to-one. Because W is a homogeneous space for G, the tangent space T_{w} is spanned by the vector fields Y_{W} = X_{f(Y)}. We must show that Y_{W} vanishes at w if and only if Y_{G·ξ} vanishes at ξ. By the non-degeneracy of the symplectic form on W, X_{f(Y)} vanishes at w if and only if ω_{w}(X_{f(Y)}, X_{f(Z)}) = 0 for all Z ∈ 𝔤_{0}. By the definition of the Poisson bracket on W, this is the same as the vanishing of all {f(Z), f(Y)}(w). By assumption (a) in Definition 4.5 and the definition of μ, this is the same as the vanishing of all [Z, Y] (for varying Z) at ξ. By (4.1)(e), this last condition is the same as Y ∈ 𝔤_{ξ,0}; that is, Y_{G·ξ} vanishes at ξ. Q.E.D.

According to Proposition 4.6, the coverings Õ appearing in Problem 3.15 are precisely the complex homogeneous Hamiltonian G_{ℂ}-spaces. This abstract characterization may lend a little respectability to what appears to be an ad hoc setting.

We turn now to a preliminary examination of how representations are attached to coadjoint orbits.

Definition 4.7. Suppose G is a Lie group, and ξ ∈ 𝔤_{0}^{∗}. An integral orbit datum at ξ is an irreducible unitary representation (τ, H_{τ}) of the isotropy group G_{ξ}, subject to the condition

τ(exp X) = e^{iξ(X)} · Id_{H_{τ}}  (X ∈ 𝔤_{ξ,0})  (4.7)

The orbit datum (τ, H_{τ}) at ξ is equivalent to (τ′, H_{τ′}) at ξ′ if there is a g ∈ G so that Ad^{∗}(g)ξ = ξ′, and τ is equivalent to the representation h ↦ τ′(ghg^{−1}) of G_{ξ}. (The second requirement makes sense because G_{ξ′} = gG_{ξ}g^{−1}.) The orbit G · ξ is called integral if it admits an integral orbit datum.

To understand this definition, notice first that the linear functional

iξ : 𝔤_{ξ,0} → iℝ  (4.8)(a)

is automatically a Lie algebra homomorphism by (4.1)(e). If G_{ξ,0} is simply connected, we therefore get automatically a unique group homomorphism

τ_{0} : G_{ξ,0} → U(1),  τ_{0}(exp X) = e^{iξ(X)}  (4.8)(b)

on the identity component G_{ξ,0} of G_{ξ}. In general (when G_{ξ,0} need not be simply connected) the requirement in (4.8)(b) still specifies at most one τ_{0}; the problem is existence. Define

L_{ξ,0} = {X ∈ 𝔤_{ξ,0} | exp(X) = e}.  (4.8)(c)

Then τ_{0} exists if and only if

ξ(L_{ξ,0}) ⊂ 2πℤ.  (4.8)(d)

(The necessity of this condition is clear, and sufficiency is not too difficult.)

Finally, it is not difficult to show that an integral orbit datum exists if and only if τ_{0} exists; so (4.8)(d) is precisely the condition for G · ξ to be integral. When G_{ξ} is disconnected, τ is usually not unique.

There is an obvious way to construct a unitary representation of G from an integral orbit datum: by unitary induction from G_{ξ} to G. That is, we consider continuous functions

C(G/G_{ξ}, H_{τ}) = {f : G → H_{τ} | f(gh) = τ(h)^{−1}f(g) (g ∈ G, h ∈ G_{ξ})}.  (4.9)(a)

Such a function is said to be of compact support modulo G_{ξ} if there is a compact subset K of G so that f vanishes outside KG_{ξ}. We write C_{c}(G/G_{ξ}, H_{τ}) for such compactly supported functions. Suppose f_{1} and f_{2} belong to C(G/G_{ξ}, H_{τ}). Then we can define a complex-valued function on G by

⟨f_{1}, f_{2}⟩_{loc}(g) = ⟨f_{1}(g), f_{2}(g)⟩_{H_{τ}} .  (4.9)(b)

Because τ is unitary, (4.9)(a) implies that ⟨f_{1}, f_{2}⟩_{loc} is actually a function on G/G_{ξ}. If one of the f_{i} belongs to C_{c}(G/G_{ξ}, H_{τ}), then ⟨f_{1}, f_{2}⟩_{loc} is compactly supported on G/G_{ξ}. In that case we may define

⟨f_{1}, f_{2}⟩ = ∫_{G/G_{ξ}}⟨f_{1}, f_{2}⟩_{loc}(x)dx.  (4.9)(c)

Here the G-invariant measure dx on G/G_{ξ} ≃ G · ξ arises naturally from the symplectic structure; the volume form may be taken to be the top exterior power of the symplectic form. In this way we get a G-invariant positive definite Hermitian form on C_{c}(G/G_{ξ}, H_{τ}). Its Hilbert space completion is called L^{2}(G/G_{ξ}, H_{τ}). This space carries a unitary representation Ind_{G_{ξ}}^{G}(τ) of G, given on C_{c}(G/G_{ξ}, H_{τ}) by left translation.

The difficulty is that this induced representation is almost never irreducible. (An interesting exception occurs when G_{ξ} is open in G; that is, when the orbit G · ξ is discrete.) To get an irreducible representation, we would like to make a similar construction on a smaller space; that is, to impose additional conditions on the functions f in (4.9)(a). One natural idea is to extend the representation τ to a larger subgroup H ⊃ G_{ξ}. It turns out to be good to restrict attention to extensions τ_{H} still satisfying the analogue of (4.7):

τ_{H}(exp X) = e^{iξ(X)} · Id_{H_{τ}}  (X ∈ 𝔥_{0} ⊃ 𝔤_{ξ,0}).  (4.10)(a)

Such an extension of τ can exist only if iξ is a Lie algebra homomorphism from 𝔥_{0} to iℝ; that is, only if

ξ([X, Y]) = 0  (X, Y ∈ 𝔥_{0}).  (4.10)(b)

According to (4.1)(f), this requirement is equivalent to the requirement that the symplectic form ω_{ξ} vanish on 𝔥_{0}/𝔤_{ξ,0} ⊂ 𝔤_{0}/𝔤_{ξ,0} ≃ T_{ξ}(G · ξ). We digress for a moment to recall some linear algebra related to this last condition.

Suppose (V, ω) is a real symplectic vector space, and Z ⊂ V is any subspace. Define

Z^{⊥} = {v ∈ V | ω(v, z) = 0, all z ∈ Z}.  (4.11)(a)

Then

dim Z + dim Z^{⊥} = dim V.  (4.11)(b)

We say that Z is isotropic if ω|_{Z} = 0; equivalently, if Z ⊂ Z^{⊥}. By (4.11)(b), this implies that

dim V = dim Z + dim Z^{⊥} ≥ 2 dim Z  (Z isotropic).  (4.11)(c)

Dually, Z is coisotropic if Z ⊃ Z^{⊥}. This implies that

dim V = dim Z + dim Z^{⊥} ≤ 2 dim Z  (Z coisotropic).  (4.11)(d)

We say that L is Lagrangian if it is both isotropic and coisotropic. In this case

dim L = 1/2 dim V  (L Lagrangian).  (4.11)(e)

Evidently L is Lagrangian if and only if L is isotropic and dim L = 1/2 dim V. Lagrangian subspaces always exist; all are conjugate under the action of the group Sp(ω).

Next, suppose (M, ω) is a symplectic manifold. A submanifold Λ ⊂ M is Lagrangian (respectively isotropic or coisotropic) if T_{λ}Λ is a Lagrangian (respectively isotropic or coisotropic) subspace of T_{λ}M for every λ ∈ Λ.

We want to shrink the space (4.9)(a) as much as possible, to have a good chance of getting an irreducible representation. In the setting of (4.10), this means taking the subgroup H as large as possible. We know already that 𝔥_{0}/𝔤_{ξ,0} must be isotropic; so it is natural to impose the requirement that 𝔥_{0}/𝔤_{ξ,0} be Lagrangian.

Definition 4.12. Suppose (τ, H_{τ}) is an integral orbit datum at ξ. An invariant real polarization of τ is a closed subgroup H ⊃ G_{ξ}, and an extension τ_{H} of τ to H, satisfying

a) τ_{H}(exp X) = e^{iξ(X)} · Id_{H_{τ}} (X ∈ 𝔥_{0});

b) H is generated by G_{ξ} and H_{0}; and

c) dim H/G_{ξ} = 1/2 dim G/G_{ξ}.

We will show in section 7 how to construct invariant real polarizations in one large class of examples; but let us consider briefly how one might look for them in general. According to (4.10), the existence of τ_{H} forces 𝔥_{0}/𝔤_{ξ,0} to be an isotropic subspace of 𝔤_{0}/𝔤_{ξ,0} = T_{ξ}(G · ξ). Then (c) makes it a Lagrangian subspace. To construct a polarization, we therefore need first of all a Lagrangian subspace of the symplectic vector space T_{ξ}(G · ξ). There will be many such subspaces. Each is of the form 𝔥_{0}/𝔤_{ξ,0}, with 𝔥_{0} a subspace of 𝔤_{0} containing 𝔤_{ξ,0}. We will show that each Lagrangian subspace gives rise to at most one invariant real polarization.

In order to correspond to a polarization, 𝔥_{0} must first be invariant under the adjoint action of G_{ξ}, and it must be a Lie algebra. When these two conditions are satisfied, there is a Lie subgroup H of G with Lie algebra 𝔥_{0}, satisfying (b) and (c) of Definition 4.12. The requirement that H be closed is not automatically satisfied, and is an additional restriction on 𝔥_{0}. Because of (b), condition (a) (together with the requirement that τ_{H} extend τ) determines τ_{H} uniquely; but there is a simple topological obstruction to its existence, which further constrains 𝔥_{0}.

Because of these requirements, it may easily happen that no invariant real polarizations exist. When they do exist, there are often many. The only natural notion of “equivalence” of polarizations is conjugation under the group G_{ξ}. But we have seen that each polarization is determined by a G_{ξ}-invariant Lagrangian subspace; so each equivalence class consists of a single element. For this reason the problem of relating different polarizations is a difficult one, and we will ignore it entirely. (In the special setting of section 7 we will find one distinguished polarization; others exist, however.)

In the setting of Definition 4.12, we have seen that the tangent space T_{ξ}(H · ξ) is a Lagrangian subspace of T_{ξ}(G · ξ). Since the symplectic structure is G-invariant, it follows that H · ξ ≃ H/G_{ξ} is a Lagrangian submanifold of G · ξ. A similar argument proves

Lemma 4.13. In the setting of Definition 4.12, consider the natural projection

π : G · ξ → G/H, π(g · ξ) = gH.

Then the fibers of π are connected Lagrangian submanifolds of G · ξ.

Let us consider now the construction of a unitary representation of G from an invariant polarization (H, τ_{H}) at ξ. Certainly we can define a space of continuous functions

C(G/H, H_{τ}) = {f : G → H_{τ} | f(gh) = τ_{H}(h)^{−1}f(g)  (g ∈ G, h ∈ H)}.  (4.14)(a)

For future reference, notice that we can rewrite at least the smooth functions C^{∞}(G/H, H_{τ}) as follows. The Lie algebra 𝔤_{0} acts on smooth functions by differentiation on the right:

(ρ(X)f)(g) = \frac{d}{dt}f(g exp(tX))|_{t=0}.  (4.14)(b)

This action makes equally good sense on smooth functions with values in H_{τ}. Then

\begin{aligned}C^{∞}(G/H, H_{τ}) & = \{f ∈ C^{∞}(G/G_{ξ}, H_{τ}) | ρ(X)f = −iξ(X)f (X ∈ 𝔥_{0})\} \\ & = \{f ∈ C^{∞}(G, H_{τ}) | f(gh) = τ(h)^{−1}f(g), ρ(X)f = −iξ(X)f (g ∈ G, h ∈ G_{ξ}, X ∈ 𝔥_{0})\}. \end{aligned}

(4.14)(c)

Write C_{c}(G/H, H_{τ}) for the subspace of functions of compact support modulo H. Just as in (4.9)(b), we can define

⟨f_{1}, f_{2}⟩_{loc}(g) = ⟨f_{1}(g), f_{2}(g)⟩_{H_{τ}} ,  (4.14)(d)

a complex-valued function on G/H. However we cannot imitate (4.9)(c) with an integral over G/H: even in very simple examples, this space may not admit a G-invariant measure. To circumvent this problem, we recall the notion of half-density.

Definition 4.15. Suppose V is a finite-dimensional real vector space and t is a real number. A t-density on V is a symbol c|dx|^{t}, with c ∈ ℝ and dx a Lebesgue measure on V. We identify c|dx|^{t} with c′|dx′|^{t} if dx′ = jdx and c = c′j^{t}. The t-densities on V form a one-dimensional real vector space D_{t}(V). For t = 1, they are just the multiples of Lebesgue measure on V. We have natural isomorphisms

D_{t}(V) ⊗ D_{s}(V) ≃ D_{t+s}(V).  (4.15)(a)

Suppose that M is a smooth manifold. Define a real line bundle D_{t} on M by D_{t}(m) = D_{t}(T_{m}M). We call D_{t} the t-density bundle on M; a section of D_{t} is called a t-density on M. If t = 1, sections of D_{1} may be identified with densities on M. In particular, if δ is a compactly supported continuous section of D_{1}, there is a natural integral

∫_{M} δ(m) ∈ ℝ  (δ ∈ C_{c}(M, D_{1})).  (4.15)(b)

There are also natural isomorphisms

D_{t} ⊗ D_{s} ≃ D_{t+s}  (4.15)(c)

as line bundles on M.

We return now to the setting of Definition 4.12, and the problem of construction of a unitary representation from τ_{H}. Write D_{t} = D_{t}(𝔤_{0}/𝔥_{0}). The adjoint action of H gives rise to a representation ϕ of H on 𝔤_{0}/𝔥_{0}, and so to an action χ_{t} of H on D_{t}. (It is easy to check that χ_{t}(h) = | det ϕ(h)|^{−t}.) Consider the space of continuous functions

C(G/H, H_{τ} ⊗ D_{1/2}) = {f : G → H_{τ} ⊗ D_{1/2} | f(gh) = (τ_{H} ⊗ χ_{1/2})(h^{−1})f(g)}  (4.16)(a)

(What we are doing is twisting the bundle on G/H defined by τ_{H} by the half-density bundle.) The inner product on H_{τ} and (4.15)(a) provide a sesquilinear pairing

\begin{matrix}⟨, ⟩_{1/2} : H_{τ} ⊗ D_{1/2} & × H_{τ} ⊗ D_{1/2} → D_{1}^{ℂ} \\ ⟨v ⊗ δ, v′ ⊗ δ′⟩_{1/2} & = ⟨v, v′⟩ · δ ⊗ δ′ \end{matrix}  (4.16)(b)

Given f_{1} and f_{2} in C(G/H, H_{τ} ⊗ D_{1/2}), we therefore get

⟨f_{1}, f_{2}⟩_{loc} = ⟨f_{1}(·), f_{2}(·)⟩_{1/2} ∈ C(G/H, D_{1}).  (4.16)(c)

This is a density on G/H, compactly supported if f_{1} or f_{2} is. By integrating densities, we therefore get a positive definite G-invariant quadratic form on C_{c}(G/H, H_{τ} ⊗ D_{1/2}). Its Hilbert space completion L^{2}(G/H, H_{τ} ⊗ D_{1/2}) carries a unitary representation of G, called Ind_{H}^{G}(τ_{H}). This is often a reasonable unitary representation to attach to the orbit datum (ξ, τ). The most serious shortcoming of the construction is that invariant real polarizations often do not exist.

Example 4.17. Suppose G = SL(2, ℝ), so that 𝔤_{0} consists of two by two real matrices of trace zero. We consider some examples of elements ξ ∈ 𝔤_{0}^{∗}.

1) ξ ( \begin{matrix}a & b \\ c & −a \end{matrix} ) = νa, 0 ≠ ν ∈ ℝ. The element ξ is hyperbolic (Lemma 5.7 below). Its isotropy group is

G_{ξ} = \{( \begin{matrix}x & 0 \\ 0 & x^{−1} \end{matrix} ) | x ∈ ℝ^{×}\} .

There are exactly two orbit data at ξ: both are one-dimensional unitary characters, and

τ_{+} ( \begin{matrix}x & 0 \\ 0 & x^{−1} \end{matrix} ) = |x|^{iν}, τ_{−} ( \begin{matrix}x & 0 \\ 0 & x^{−1} \end{matrix} ) = |x|^{iν} · sgn(x).

Each has an invariant real polarization by the subgroup

H = \{( \begin{matrix}x & y \\ 0 & x^{−1} \end{matrix} ) | x ∈ ℝ^{×}, y ∈ ℝ\};

the characters τ_{±,H} are trivial on elements ( \begin{matrix}1 & y \\ 0 & 1 \end{matrix} ). The unitarily induced representations of (4.16)

are always irreducible.

2) ξ ( \begin{matrix}a & b \\ c & −a \end{matrix} ) = c. The element ξ is nilpotent (Lemma 5.5 below). The corresponding symplectic form is

ω_{ξ} (( \begin{matrix}a & b \\ c & −a \end{matrix} ) , ( \begin{matrix}a′ & b′ \\ c′ & −a′ \end{matrix} )) = 2(ca′ − ac′),

and the isotropy group is

G_{ξ} = \{( \begin{matrix}ϵ & t \\ 0 & ϵ \end{matrix} ) | ϵ = ±1\} .

Again the orbit data are two one-dimensional unitary characters τ_{±}, defined by

τ_{+} ( \begin{matrix}ϵ & t \\ 0 & ϵ \end{matrix} ) = 1, τ_{−} ( \begin{matrix}ϵ & t \\ 0 & ϵ \end{matrix} ) = ϵ.

Each has an invariant real polarization by the subgroup H of (1). In this case Ind_{H}^{G}(τ_{+,H}) is irreducible, but Ind_{H}^{G}(τ_{−,H}) is a direct sum of two irreducible components.

3) ξ ( \begin{matrix}a & b \\ c & −a \end{matrix} ) = t/2(b − c), 0 ≠ t ∈ ℝ. This element is elliptic (Lemma 5.8 below). Its isotropy group is

G_{ξ} = \{( \begin{matrix}cos θ & sin θ \\ − sin θ & cos θ \end{matrix} ) | θ ∈ ℝ\} .

Write Z for the matrix ( \begin{matrix}0 & 1 \\ −1 & 0 \end{matrix} ), so that the element of G_{ξ} above is exp θZ. Since ξ(Z) = t, the

requirement (4.7) for an integral orbit datum is

τ_{t}(exp θZ) = e^{itθ}.

This is a well-defined character of G_{ξ} if and only if t ∈ ℤ; so we conclude that ξ is integral if and only if t ∈ ℤ. The subgroup G_{ξ} is maximal in G, so there is no invariant real polarization of τ_{t}.

5. (Co)adjoint orbits for reductive groups.

We turn now to a more detailed study of coadjoint orbits in the case of real reductive groups. We single out three special classes, called hyperbolic, elliptic, and nilpotent. A hyperbolic coadjoint orbit is isomorphic to an affine bundle over a real flag variety. We will eventually attach representations to such orbits by real analysis on the real flag variety. An elliptic coadjoint orbit carries an invariant (indefinite) Kähler structure; it is isomorphic to an open orbit in a complex flag variety. We will attach representations to elliptic orbits by complex analysis techniques. A nilpotent coadjoint orbit has (in general) neither real nor complex structure of these kinds; so we do not know how to attach representations to it.

In the remainder of this section we will define the three special classes of orbits, and show how to realize a general coadjoint orbit as combination of them.

So suppose G is a real reductive Lie group, with Cartan involution θ, maximal compact subgroup K, and Lie algebra 𝔤_{0}. Definition 1.6 allows us to identify 𝔤_{0} with a Lie subalgebra of 𝔤𝔩(n, ℝ) closed under transpose. On 𝔤𝔩(n, ℝ) we can define the trace form

⟨X, Y ⟩ = tr XY.  (5.1)(a)

This is a symmetric bilinear form, invariant by the adjoint action of GL(n, ℝ):

⟨Ad(g)X, Ad(g)Y ⟩ = ⟨X, Y ⟩  (5.1)(b)

for X, Y ∈ 𝔤𝔩(n, ℝ) and g ∈ GL(n, ℝ). By direct computation one finds that ⟨, ⟩ is positive definite on symmetric matrices and negative definite on skew-symmetric matrices, and that it makes these two subspaces orthogonal to each other. One immediate consequence is that ⟨, ⟩ is non-degenerate. We also deduce

Lemma 5.2. Suppose G is a real reductive Lie group with π : G → GL(n, ℝ) as in Definition 1.6. Then the restriction to 𝔤_{0} of the trace form on 𝔤𝔩(n, ℝ) (cf. (5.1)) has the following properties:

a) ⟨Ad(g)X, Ad(g)Y ⟩ = ⟨X, Y ⟩ X, Y ∈ 𝔤_{0}, g ∈ G;

b) ⟨, ⟩ is positive definite on 𝔭_{0}, negative definite on 𝔨_{0}, and has ⟨𝔭_{0}, 𝔨_{0}⟩ = 0 (notation as in Definition 1.4); and

c) ⟨, ⟩ is nondegenerate on 𝔤_{0}.

Consequently ⟨, ⟩ defines an isomorphism 𝔤_{0} ≃ 𝔤_{0}^{∗} carrying the adjoint action of G on 𝔤_{0} to the coadjoint action of G on 𝔤_{0}^{∗}.

Because of this lemma, coadjoint orbits for reductive groups are closely related to adjoint orbits; and adjoint orbits in turn are closely related to conjugacy classes of matrices. We recall some facts from linear algebra.

Lemma 5.3. Suppose X is an n × n real matrix. The following properties of X are equivalent.

a) The linear transformation of ℂ^{n} defined by X is diagonalizable.

b) The minimal polynomial of X has no repeated factors.

c) The conjugacy class

GL(n, ℝ) · X = {gXg^{−1} | g ∈ GL(n, ℝ)}

is closed in 𝔤𝔩(n, ℝ).

d) There is an X′ = gXg^{−1} in the conjugacy class of X so that X′ commutes with ^{t}X′.

When these conditions are satisfied, we say that X is semisimple.

Sketch of proof. We will sketch the proof that (b) implies (c), which is perhaps one of the least familiar parts of the argument. Let p = p_{1} ⋯ p_{r} be a factorization of the minimal polynomial of X, with p_{i} an irreducible real polynomial. Each p_{i} is either linear or quadratic with no real roots. Let d_{i} be the dimension of ker p_{i}(X). We have

ℝ^{n} = ⨁_{i=1}^{r} ker p_{i}(X),  (5.4)(a)

so ∑ d_{i} = n. Also

GL(n, ℝ) · X = {Y ∈ 𝔤𝔩(n, ℝ) | dim ker p_{i}(Y) ≥ d_{i}, all i}.  (5.4)(b)

(Obviously GL(n, ℝ) · X is contained in the right side of (5.4)(b). For the other containment, notice that for any matrix Z, the sum ∑_{i=1}^{r} ker p_{i}(Z) must be direct, as the p_{i} are distinct irreducible polynomials. For Y in the right side of (5.4)(b), it follows by dimension counting that

ℝ^{n} = ⨁_{i=1}^{r} ker p_{i}(Y),  (5.4)(c)

and that dim ker p_{i}(Y) = d_{i}. This implies easily that Y is conjugate to X.) To complete the proof of (c), notice that the right side of (5.4)(b) is closed in 𝔤𝔩(n, ℝ); this follows from the compactness of the Grassmannian manifolds of d-dimensional subspaces of ℝ^{n}. Q.E.D.

Lemma 5.5. Suppose X is an n × n real matrix. The following properties of X are equivalent.

a) X is nilpotent.

b) The characteristic polynomial of X is t^{n}.

c) The closure of the conjugacy class of X contains 0.

d) The conjugacy class of X contains a multiple rX of X, with r > 0 and r ≠ 1.

e) The conjugacy class of X contains every multiple rX of X, with r > 0.

f) There is an element A ∈ 𝔤𝔩(n, ℝ) with [A, X] = X.

When these conditions are satisfied, we say that X is nilpotent.

Proof. We first prove the ascending implications, beginning with (f). Then exp(sA)(X) exp(−sA) = exp(ad(sA))(X) = e^{s}X, which proves (e). Trivially (e) implies (d). Assume (d); that is, that gXg^{−1} = rX for some positive r ≠ 1. Possibly replacing g by g^{−1}, we may assume r < 1. Then g^{n}Xg^{−n} = r^{n}X → 0, proving (c). Assume (c). The characteristic polynomial det(tI − Z) is constant on conjugacy classes and depends continuously on Z; so the characteristic polynomial of X must coincide with that of 0, which is t^{n}. Assume (b); then X^{n} = 0 since every matrix satisfies its characteristic polynomial, so X is nilpotent.

To finish, we prove that (a) implies (f). Define

H = {r > 0 | gXg^{−1} = rX, some g ∈ GL(n, ℝ)}.

This is a subgroup of the multiplicative group of positive real numbers. If s and s′ are any two positive numbers, then sX is conjugate to s′X if and only if s and s′ belong to the same coset of H. Now every multiple of X is nilpotent, and there are only finitely many conjugacy classes of nilpotent n × n matrices. It follows that H has finite index m in ℝ^{>0}. Consequently every mth power belongs to H, so H = ℝ^{>0}. Therefore every positive multiple of X is conjugate to X. It follows that the tangent space at X to the conjugacy class of X contains X. Now the tangent space at Z to the conjugacy class of Z is [𝔤𝔩(n, ℝ), Z]; so (f) follows. Q.E.D.

Lemma 5.6 (Jordan decomposition). Suppose X is an n × n real matrix. Then there are a semisimple matrix X_{s} and a nilpotent matrix X_{n} uniquely characterized by the following two properties:

a) X = X_{s} + X_{n}; and

b) [X_{s}, X_{n}] = 0.

In addition, we have

c) X_{s} and X_{n} may be expressed as polynomials without constant term in X;

d) any matrix commuting with X commutes with X_{s} and X_{n}; and

e) any subspace of ℝ^{n} preserved by X is also preserved by X_{s} and X_{n}.

We turn now to an analogous decomposition of a semisimple matrix.

Lemma 5.7. Suppose X is an n × n real matrix. The following properties of X are equivalent.

a) X is diagonalizable.

b) The minimal polynomial of X is a product of distinct linear factors.

c) The conjugacy class of X contains a symmetric matrix.

When these conditions are satisfied, we say that X is hyperbolic. The proof of the lemma is very easy, and we omit it.

Lemma 5.8. Suppose X is an n × n real matrix. The following properties of X are equivalent.

a) The linear transformation of ℂ^{n} defined by X is diagonalizable with purely imaginary eigenvalues.

b) The minimal polynomial of X is a product of distinct factors of the form t^{2} + a^{2} and t.

c) The conjugacy class of X contains a skew-symmetric matrix.

When these conditions are satisfied, we say that X is elliptic. Again we omit the proof of the lemma. Here is a complement to the Jordan decomposition of Lemma 5.6.

Lemma 5.9. Suppose X is a semisimple n × n real matrix. Then there are a hyperbolic matrix X_{h} and an elliptic matrix X_{e} uniquely characterized by the following two properties:

a) X = X_{h} + X_{e}; and

b) [X_{h}, X_{e}] = 0.

In addition, we have

c) X_{h} and X_{e} may be expressed as polynomials in X;

d) any matrix commuting with X commutes with X_{h} and X_{e}; and

e) any subspace of ℝ^{n} preserved by X is also preserved by X_{h} and X_{e}.

f) If X commutes with ^{t}X, then X_{h} = 1/2(X +^{t}X) and X_{e} = 1/2(X −^{t}X).

g) Suppose that X is a derivation of an algebra structure on ℝ^{n}. Then X_{e} and X_{h} are derivations as well.

Proof. We first establish the existence of the decomposition. Because the definitions of semisimple, hyperbolic, and elliptic are all invariant under conjugation (see Lemmas 5.3, 5.7, and 5.8), we may replace X by a conjugate matrix. By Lemma 5.3(d), we may therefore assume that X commutes with ^{t}X. In this case the matrices X_{h} and X_{e} defined in (f) obviously commute and have sum X. Also X_{h} is symmetric, and therefore hyperbolic (Lemma 5.7(c)), and X_{e} is skew-symmetric, and therefore elliptic (Lemma 5.8(c)). This proves the existence, as well as (f). For the uniqueness, regard X, X_{h}, and X_{e} as linear transformations of ℂ^{n}. They commute with each other and each is diagonalizable; so they are simultaneously diagonalizable (with respect to some basis v_{1}, …, v_{n} of ℂ^{n}). Write z_{i} for the diagonal entries of X as a matrix in this basis, and a_{i} and b_{i} for those of X_{e} and X_{h}. Then z_{i} = a_{i} +b_{i}, a_{i} is real (Lemma 5.7(a)) and b_{i} is purely imaginary (Lemma 5.8(b)). Therefore a_{i} = Re z_{i}, b_{i} = \sqrt{−1}Im z_{i}. This means that X_{h} acts on each eigenspace of X by the real part of the corresponding eigenvalue. Similarly, X_{e} acts on each eigenspace of X by the imaginary part of the eigenvalue. These descriptions establish the uniqueness of X_{h} and X_{e}, and properties (d) and (e) in the Lemma follow easily. To prove (c), we need to know the existence of real polynomials p_{h} and p_{e} with the properties that

p_{h}(z_{i}) = Re z_{i}, p_{e}(z_{i}) = \sqrt{−1}Im z_{i}.

Because the non-real z_{i} occur in complex conjugate pairs, this is elementary.

Finally, assume that X is a derivation of an algebra structure ∘. After complexification, this means exactly that if v and v′ are eigenvectors of eigenvalues z and z′, then v ∘ v′ is an eigenvector of eigenvalue z + z′. The description above of X_{h} and X_{e} shows that they inherit this property from X. Q.E.D.

With these facts about matrices in hand, we turn to their generalizations for reductive groups.

Definition 5.10. Suppose G is a real reductive Lie group with π : G → GL(n, ℝ) as in Definition 1.6. Recall that the Lie algebra 𝔤_{0} of G is identified with a subalgebra of 𝔤𝔩(n, ℝ). An element X ∈ 𝔤_{0} is called semisimple (respectively nilpotent, hyperbolic, or elliptic) if X has the corresponding property as a matrix (Lemma 5.3, 5.5, 5.7, or 5.8).

In order to give more intrinsic characterizations of these properties, we will consider not the action of X on ℝ^{n}, but rather the adjoint action on 𝔤_{0}:

ad(X)(Y) = [X, Y].  (5.11)(a)

The kernel of the adjoint action is the center 𝔷(𝔤_{0}):

𝔷(𝔤_{0}) = {Z ∈ 𝔤_{0} | [Z, Y] = 0, all Y ∈ 𝔤_{0}}.  (5.11)(b)

Obviously this is preserved by θ, and so is the direct sum of its intersections with 𝔨_{0} and 𝔭_{0}:

𝔷(𝔤_{0}) = 𝔷_{𝔨}(𝔤_{0}) ⊕ 𝔷_{𝔭}(𝔤_{0}).  (5.11)(c)

The “image” of the adjoint action is the derived algebra

𝔤′_{0} = [𝔤_{0}, 𝔤_{0}].  (5.11)(d)

Because of the ad-invariance of the trace form (5.1), the derived algebra is precisely the orthogonal complement of the center for the trace form. Lemma 5.2 therefore implies

𝔤_{0} = 𝔤′_{0} ⊕ 𝔷(𝔤_{0}),  (5.11)(e)

a direct sum of θ-stable ideals. (Notice that although we used structure from matrices to prove (5.11)(e), the summands are defined in terms of the Lie algebra structure of 𝔤_{0}.) It is not difficult to show that the Lie algebra 𝔤′_{0} is semisimple, and we will apply to it some standard structural results.

Theorem 5.12. In the setting of Definition 5.10 and (5.11), suppose ρ : 𝔤′_{0} → End V is a finitedimensional representation of the semisimple Lie algebra 𝔤′_{0}, and X′ ∈ 𝔤′_{0}.

a) If ad(X′) is semisimple, then ρ(X′) is semisimple.

b) If ad(X′) is nilpotent, then ρ(X′) is nilpotent.

c) If T is a derivation of 𝔤′_{0}, then there is a unique element X_{T} ∈ 𝔤′_{0} with T = ad(X_{T}).

d) If X′ = X′_{s} + X′_{n} is the Jordan decomposition of X′ as an n × n matrix (Lemma 5.6), then X′_{s} and X′_{n} belong to 𝔤′_{0}.

e) If X′ is semisimple, then there is a g ∈ G so that Ad(g)(X′) = gX′g^{−1} commutes with θ(Ad(g)(X′)).

Parts (a) and (b) are proved in [Humphreys], Theorem 6.5; part (c) is [Humphreys], Theorem 5.3; part (d) is [Humphreys], Theorem 6.4; and (e) is essentially [Wallach], Lemma 2.3.3.

Lemma 5.13. In the setting of Definition 5.10, write X = X′ + X_{𝔷} for the decomposition according to (5.11)(e).

a) X is semisimple if and only if ad(X) is semisimple.

b) X is nilpotent if and only if X_{𝔷} = 0 and ad(X) is nilpotent. c) X is hyperbolic if and only if X_{𝔷} ∈ 𝔭_{0} and ad(X) is hyperbolic.

d) X is elliptic if and only if X_{𝔷} ∈ 𝔨_{0} and ad(X) is elliptic.

Proof. The matrices X′ and X_{𝔷} commute with each other; and ad(X) = ad(X′). The matrix X_{𝔷} is a sum of commuting symmetric and skew-symmetric matrices (cf. (5.11)(c)), and is therefore automatically semisimple.

We will analyze the bracket action ad_{𝔤𝔩(n,ℝ)}(X) of X on all n × n matrices. This space is naturally identified with ℝ^{n} ⊗ (ℝ^{n})^{∗}, and we compute

ad_{𝔤𝔩(n,ℝ)}(X)(v ⊗ w) = Xv ⊗ w − v ⊗^{t}Xw.  (5.14)

If X is semisimple with eigenvalues {z_{i}}, it follows immediately that ad_{𝔤𝔩(n,ℝ)}(X) is semisimple with eigenvalues {z_{i} − z_{j}}. The implication “only if” in (a) is immediate. For the converse, assume that ad(X) = ad(X′) is semisimple. By Theorem 5.12(a), it follows that X′ is semisimple. Therefore X = X′ +X_{𝔷} is a commuting sum of semisimple matrices, so X is semisimple.

For (b), suppose first that X is nilpotent; say X^{p} = 0. By (5.14), we see that (ad(X))^{2p} = 0; so ad(X) is nilpotent. By Theorem 5.12(b), X′ is nilpotent. The expression X = X′ + X_{𝔷} is a commuting sum of a nilpotent and a semisimple matrix, so it must be the Jordan decomposition of X. Since X is nilpotent, it follows that X_{𝔷} = 0. Conversely, assume that X_{𝔷} = 0 and that ad(X) = ad(X′) is nilpotent. By Theorem 5.12(b), X′ is nilpotent; so since X_{𝔷} = 0, X is nilpotent.

For (c) and (d), assume first that X is hyperbolic; that is, that X is diagonalizable with real eigenvalues. Then (5.14) implies that ad(X) = ad(X′) is as well; so ad(X) is hyperbolic. Similarly, X elliptic implies ad(X) elliptic.

To complete the proofs of (c) and (d), we will use a lemma. Here is the setting. Suppose X′ ∈ 𝔤′_{0} is a semisimple element. According to Lemma 5.9, the hyperbolic and elliptic parts T_{h} and T_{e} of the semisimple derivation ad(X′) of 𝔤′_{0} are also (semisimple) derivations. By Theorem 5.12(c), they are given by the adjoint action of unique elements X′_{[h]} and X′_{[e]} of 𝔤′_{0}. On the other hand, the semisimple matrix X′ has hyperbolic and elliptic parts X′_{h} and X′_{e}.

Lemma 5.15. In the setting just described, X′_{[h]} = X′_{h} and X′_{[e]} = X′_{e}. In particular, the hyperbolic and elliptic parts of X′ belong to 𝔤′_{0}.

Proof. Everything here behaves nicely with respect to conjugation by elements of G. According to Theorem 5.12(d), we may therefore assume that X′ commutes with θX′. In this case we have

X′_{h} = 1/2(X′ − θX′), X′_{e} = 1/2(X′ + θX′)

(Lemma 5.9(f)). Evidently these matrices belong to 𝔤′_{0}. We therefore have

ad(X′) = ad(X′_{h}) + ad(X′_{e}),

a commuting sum of derivations of 𝔤′_{0}. By what we have already proved, the first term is hyperbolic and the second elliptic as endomorphisms of 𝔤′_{0}. By the uniqueness in Lemma 5.9, they are the hyperbolic and elliptic parts of ad(X′). Q.E.D.

We return now to finish the proofs of (c) and (d). In general we will write

X_{𝔷} = X_{𝔷_{𝔨}} + X_{𝔷_{𝔭}}

for the decomposition of a central element according to (5.11)(c). Suppose X is hyperbolic. We have already shown that ad(X) = ad(X′) is hyperbolic, so Lemma 5.15 implies that X′ is hyperbolic. The three matrices X′, X_{𝔷_{𝔨}}, and X_{𝔷_{𝔭}} are therefore hyperbolic, elliptic, and hyperbolic respectively; and they commute with each other. It follows that X = (X′ + X_{𝔷_{𝔭}}) + (X_{𝔷_{𝔨}}) exhibits X as a commuting sum of hyperbolic and elliptic matrices. By the uniqueness in Lemma 5.9, the second term is zero; that is, X_{𝔷} belongs to 𝔭_{0}, as we wished to show. Conversely, assume that ad(X) is hyperbolic and that X_{𝔷} ∈ 𝔭_{0}. Lemma 5.15 shows that X′ is hyperbolic; so X is a sum of two commuting hyperbolic matrices, and is therefore hyperbolic. The proof of (d) is identical. Q.E.D.

Theorem 5.16. Suppose G is a real reductive Lie group as in Definition 1.6, and X ∈ 𝔤_{0}. Then there are hyperbolic, elliptic, and nilpotent elements X_{h}, X_{e}, and X_{n} of 𝔤_{0} (Definition 5.10) uniquely characterized by the following two properties:

a) X = X_{h} + X_{e} + X_{n}; and

b) [X_{h}, X_{e}] = [X_{h}, X_{n}] = [X_{e}, X_{n}] = 0.

This decomposition has the following additional properties.

c) Any element g ∈ G fixing X (that is, Ad(g)(X) = X) also fixes X_{h}, X_{e}, and X_{n}.

d) The adjoint representation of 𝔤_{0} preserves the decomposition in (a): ad(X_{h}) is hyperbolic, ad(X_{e}) is elliptic, and ad(X_{n}) is nilpotent.

This is an easy consequence of Theorem 5.12, Lemma 5.13, and Lemma 5.15; we leave the details to the reader.

Corollary 5.17. Suppose G is a real reductive Lie group as in Definition 1.6, and X ∈ 𝔤_{0} is a semisimple element.

a) There is a g ∈ G so that Ad(g)(X) commutes with θ(Ad(g)(X)).

b) Suppose [X, θX] = 0. Then the centralizer

G_{X} = {g ∈ G | Ad(g)(X) = X}

is a real reductive Lie group, with Cartan involution the restriction to G_{X} of θ. In addition

G_{X} = (G_{X_{h}})_{X_{e}} = (G_{X_{e}})_{X_{h}}.

Proof. Part (a) is an easy consequence of Theorem 5.12(e) (and (5.11)(c)). For (b), the descriptions of G_{X} as iterated centralizers follow from Theorem 5.16(c). That G_{X} is reductive then follows from Proposition 1.11(b). Q.E.D.

6. Interlude on the classification of (co)adjoint orbits.

Corollary 5.17 leads to a systematic procedure for describing all the coadjoint orbits for a real reductive Lie group. Although we will not really need it, the classification is simple and attractive. The method of coadjoint orbits suggests as well that it should bear a family resemblance to the classification of unitary representations. For these reasons, we will outline the classification here. Most of the proofs are easy; the more difficult results may be found in many places (including [Wallach], Chapter 2).

Definition 6.1. Suppose G is a real reductive Lie group. A Cartan subspace for G is a maximal abelian subalgebra 𝔞_{0} of 𝔭_{0}. Given such a subspace, we define

M = {k ∈ K | Ad(k)(X) = X, all X ∈ 𝔞_{0}}
M′ = {k ∈ K | Ad(k)(𝔞_{0}) = 𝔞_{0}}.

The groups M′ and M are compact, and M is an open normal subgroup of M′. The quotient is therefore a finite group

W (G, 𝔞_{0}) = M′/M ⊂ Aut(𝔞_{0}),

the Weyl group of 𝔞_{0} in G. (The action on 𝔞_{0} is by Ad.)

Theorem 6.2. Suppose G is a real reductive group, and 𝔞_{0} is a Cartan subspace of 𝔭_{0}. Then the inclusions

𝔞_{0} ⊂ 𝔭_{0} ⊂ 𝔤_{0}

induce bijections among the following three sets:

i) orbits of W = W (G, 𝔞_{0}) on 𝔞_{0}; ii) orbits of K on 𝔭_{0}; and iii) hyperbolic orbits of G on 𝔤_{0}.

Suppose X and Y belong to 𝔞_{0}. Then the group centralizers G_{X} and G_{Y} are equal if and only if the Weyl group centralizers W_{X} and W_{Y} are equal. In particular, there are only finitely many possibilities (up to conjugation in G) for the centralizer of a hyperbolic element.

Example 6.3. Suppose G = O(p, q) (Example 1.15) with p ≤ q. After a simple change of basis, the matrix A_{p,q} of the quadratic form B_{p,q} may be replaced by the matrix

A_{p,q}^{1} = ( \begin{matrix}0 & I_{p} & 0 \\ I_{p} & 0 & 0 \\ 0 & 0 & −I_{p−q} \end{matrix} ) .

We write B^{1}(p, q) for this new form and O^{1}(p, q) for the corresponding group. If g ∈ GL(p, ℝ), then it is easy to check that G^{1} = O^{1}(p, q) contains the matrix

λ^{1}(g) = ( \begin{matrix}g & 0 & 0 \\ 0 & ^{t}g^{−1} & 0 \\ 0 & 0 & I_{p−q} \end{matrix} ) .

The map λ^{1} provides an embedding

GL(p, ℝ) ⊂ O^{1}(p, q) ≃ O(p, q),

and all maps respect the Cartan involutions.

Now the space of diagonal matrices is a Cartan subspace for GL(p, ℝ), naturally isomorphic to ℝ^{p}. It is not difficult to check that the image 𝔞_{0}^{1} of the diagonal matrices under dλ^{1} is a Cartan subspace of 𝔤_{0}^{1}; 𝔞_{0}^{1} consists of diagonal matrices with the first p entries equal to the negatives of the next p, and the last p − q entries equal to zero. The Weyl group of this Cartan subspace acts by permuting and changing the signs of the coordinates of ℝ^{p}; it is isomorphic to the hyperoctahedral group

W (G, 𝔞_{0}^{1}) = S_{p} × (ℤ/2ℤ)^{p},

a semidirect product with the second factor normal.

By Theorem 6.2, each hyperbolic orbit has a unique representative

X = (X_{1}, …, X_{p}) ∈ ℝ^{p}, X_{1} ≥ ⋯ ≥ X_{p} ≥ 0.

We can compute the centralizer G_{X}^{1} as follows. Write p = p_{r} + ⋯ p_{1} + p_{0}, in such a way that

X_{1} = ⋯ = X_{p_{r}} > X_{p_{r}+1} = ⋯ = X_{p_{r}+···p_{1}} > X_{p_{r}+···+p_{1}+1} = 0.

Then

G_{X} ≃ GL(p_{r}, ℝ) × ⋯ × GL(p_{1}, ℝ) × O^{1}(p_{0}, p_{0} + q − p).

(There is a natural embedding of this group in G^{1}, which is easy to construct using λ^{1}.)

There is a parallel result for elliptic elements.

Definition 6.4. Suppose G is a real reductive Lie group. A Cartan subalgebra for K is a maximal abelian subalgebra 𝔱_{0} of 𝔨_{0}. Given such a subspace, we define

T = {k ∈ K | Ad(t)(X) = X, all X ∈ 𝔱_{0}}
T′ = {k ∈ K | Ad(k)(𝔱_{0}) = 𝔱_{0}}.

The group T is called a small Cartan subgroup of K. If K is connected, it is a torus. The group T′ is compact, and T is an open normal subgroup of T′. The quotient is therefore a finite group

W (G, 𝔱_{0}) = T′/T ⊂ Aut(𝔱_{0}),

the Weyl group of 𝔱_{0} in G. (The action on 𝔱_{0} is by Ad.)

Theorem 6.5. Suppose G is a real reductive group, and 𝔱_{0} is a Cartan subalgebra of 𝔨_{0}. Then the inclusions

𝔱_{0} ⊂ 𝔨_{0} ⊂ 𝔤_{0}

induce bijections among the following three sets:

i) orbits of W = W (G, 𝔱_{0}) on 𝔱_{0}; ii) orbits of K on 𝔨_{0}; and iii) elliptic orbits of G on 𝔤_{0}.

Suppose X and Y belong to 𝔱_{0}. Then the group centralizers G_{X} and G_{Y} are equal if and only if

1) the Weyl group centralizers W_{X} and W_{Y} are equal; and

2) the elements X and Y annihilate exactly the same weights of 𝔱_{0} on 𝔭_{0}. In particular, there are only finitely many possibilities (up to conjugation in G) for the centralizer of a elliptic element.

Example 6.6. Suppose G = O(p, q) (Example 1.15). Recall that K ≃ O(p) × O(q). Write p′ = [p/2] (the greatest integer in p/2) and q′ = [q/2]. Then there are obvious maps

SO(2)^{p′+q′} ≃ SO(2)^{p′} × SO(2)^{q′} ⊂ O(p) × O(q) ⊂ O(p, q).

The Lie algebra of SO(2) consists of skew symmetric 2 × 2 matrices, and so may be identified naturally with ℝ. The differential of the inclusion above therefore gives

τ : ℝ^{p′} × ℝ^{q′} → 𝔨_{0}

It is easy to check that the image of τ is a Cartan subalgebra 𝔱_{0} of 𝔨_{0}. The Weyl group acts by permutation and sign changes of the first p′ and last q′ coordinates separately:

W (G, 𝔱_{0}) = (S_{p′} × (ℤ/2ℤ)^{p′}) × (S_{q′} × (ℤ/2ℤ)^{q′}),

a product of two hyperoctahedral groups. According to Theorem 6.5, every elliptic orbit in 𝔤_{0} has a unique representative

Z = [(Z_{1}^{1}, …, Z_{p′}^{1}), (Z_{1}^{2}, …, Z_{q′}^{2})] ∈ ℝ^{p′+q′}, Z_{1}^{1} ≥ ⋯ ≥ Z_{p′}^{1} ≥ 0, Z_{1}^{2} ≥ ⋯ ≥ Z_{q′}^{2} ≥ 0.

We can compute the centralizer G_{Z} as follows. Write p′ = p′_{r} + ⋯ p′_{1} + p′_{0} and q′ = q′_{r} + ⋯ q′_{1} + q′_{0}, in such a way that

Z_{1}^{1} = ⋯ = Z_{p′_{r}}^{1} > Z_{p′_{r}+1}^{1} = ⋯ = Z_{p_{r}+···+p′_{1}}^{1} > Z_{p′_{r}+···+p′_{1}+1}^{1} = 0,
Z_{2}^{1} = ⋯ = Z_{q′_{r}}^{2} > Z_{q′_{r}+1}^{2} = ⋯ = Z_{q′_{r}+···+q′_{1}}^{2} > Z_{q′_{r}+···+q′_{1}+1}^{2} = 0,
Z_{p′_{r}+···+p′_{j}+1}^{1} = Z_{q′_{r}+···+q′_{j}+1}^{2}, (j = r + 1, r − 1, …, 1).

Then

G_{Z} ≃ U(p′_{r}, q′_{r}) × ⋯ × U(p′_{1}, q′_{1}) × O^{1}(p − 2(p′_{r} + ⋯ + p′_{1}), q − 2(q′_{r} + ⋯ + q′_{1})).

(The idea is that the element Z acts as Z_{p′_{r}+···+p′_{j+1}+1}^{1} times a complex structure on a subspace of the form

ℝ^{2p′_{j}+2q′_{j}}. The centralizer of Z therefore acquires a factor consisting of the complex-linear transformations of this subspace that preserve the quadratic form. This is the indefinite unitary group U(p′_{j}, q′_{j}) of Example 1.19.)

Here is the procedure for constructing all adjoint orbits for a reductive group G. First, choose a Cartan subspace 𝔞_{0} ⊂ 𝔭_{0}, and compute its Weyl group. For each element X_{h} ∈ 𝔞_{0}, consider the reductive group G_{X_{h}}; Theorem 6.2 guarantees that there are only finitely many such groups. (They are described for G = O(p, q) in Example 6.3.) For each such group, choose a Cartan subalgebra 𝔱_{X_{h},0} of 𝔨_{X_{h},0}, and compute its Weyl group. For each element X_{e} ∈ 𝔱_{X_{h},0}, consider the reductive group G_{X_{h},X_{e}}; Theorem 6.5 says that there are still only finitely many such groups. (When G_{X_{h}} is a product, it suffices to treat each factor separately. The case of O(p, q) is handled in Example 6.6; to treat all of the hyperbolic centralizers for O(p, q), we would need to consider also GL(n, ℝ). The conclusion is that for O(p, q), each group G_{X_{h},X_{e}} is a product of factors of the form GL(a, ℝ), GL(b, ℂ), U(c, d), and O(e, f); there is just one factor of this last form.)

To make a list of all semisimple adjoint orbits, we choose one representative X_{h} in each Weyl group orbit on 𝔞_{0}; and then (for each X_{h}) one representative X_{e} in each Weyl group orbit on 𝔱_{X_{h},0}. The resulting elements

X_{s} = X_{h} + X_{e}

provide exactly one representative of each semisimple orbit on 𝔤_{0}.

This procedure can be continued to give all adjoint orbits: we just need a way to give a representative for each of the (finitely many) nilpotent adjoint orbits for each of the (finitely many) reductive groups G_{X_{h},X_{e}}. For the classical groups this problem is not too hard, and for the exceptional groups it has been solved. We refer to [Collingwood-McGovern] for more information.

7. Hyperbolic elements, real polarizations and parabolic subgroups.

We turn now to the problem of attaching representations to orbits for a reductive group G. Recall from Lemma 5.2 that we can identify 𝔤_{0} with 𝔤_{0}^{∗} by an isomorphism X ↦ ξ_{X}:

ξ_{X}(Y) = ⟨X, Y ⟩ = tr XY.  (7.1)(a)

Lemma 5.2 allows us to identify the isotropy algebra and group for the coadjoint action at ξ_{X} (see (4.1)) as

G_{ξ_{X}} = {g ∈ G | Ad(g)X = X},  𝔤_{ξ_{X},0} = {Y ∈ 𝔤_{0} | [Y, X] = 0}.  (7.1)(b)

We may sometimes write G_{X} or 𝔤_{X,0} accordingly. The symplectic form of (4.1)(f) is

ω_{ξ_{X}} (Y, Z) = ⟨X, [Y, Z]⟩ = −⟨Z, [Y, X]⟩;  (7.1)(c)

notice that this formula is skew symmetric in all three variables X, Y, and Z. We sometimes write simply ω_{X}. We call ξ_{X} semisimple (or nilpotent, elliptic, or hyperbolic) if X is.

Proposition 7.2. In the setting of (7.1), suppose that ad(X) is diagonalizable with real eigenvalues; this happens in particular if X is hyperbolic (Lemma 5.13). Write 𝔤_{t,0} for the t-eigenspace of ad(X), so that

𝔤_{0} = ∑_{t∈ℝ} 𝔤_{t,0},  𝔤_{X,0} = 𝔤_{0,0}.  (7.2)(i)

Define

𝔭_{X} = ∑_{t≥0} 𝔤_{t,0},  𝔫_{X,0} = ∑_{t>0} 𝔤_{t,0},  N_{X} = exp 𝔫_{X,0}.  (7.2)(ii)

a) The decomposition (7.2)(i) makes 𝔤_{0} an ℝ-graded Lie algebra: [𝔤_{s,0}, 𝔤_{t,0}] ⊂ 𝔤_{s+t,0}.

b) The subspace 𝔤_{t,0} is orthogonal to 𝔤_{s,0} with respect to ω_{X} unless s = −t.

c) The adjoint action of N_{X} on X defines a diffeomorphism

γ : N_{X} → X + 𝔫_{X,0}, γ(n) = Ad(n)(X).

d) The coadjoint action of N_{X} on ξ_{X} defines a diffeomorphism

γ^{∗} : N_{X} → {λ ∈ 𝔤_{0}^{∗} | λ|_{𝔭_{X}} = ξ_{X}|_{𝔭_{X}} }, γ^{∗}(n) = Ad^{∗}(n)(ξ_{X}).

e) The group N_{X} is connected, simply connected, and nilpotent. It is normalized by G_{X}, and meets G_{X} exactly in the identity element. The semidirect product

P_{X} = G_{X}N_{X}

is a closed subgroup of G; more precisely, this is the Levi decomposition of a parabolic subgroup.

f) Suppose τ is an integral orbit datum at ξ_{X} (Definition 4.7). Then τ has a unique extension τ_{P_{X}} to P_{X}. This extension may be characterized by

τ_{P_{X}} (gn) = τ(g) (g ∈ G_{X}, n ∈ N_{X}).

The pair (τ_{P_{X}} , P_{X}) is an invariant real polarization of τ.

Proof. Part (a) is elementary. For (b), suppose Y ∈ 𝔤_{s,0} and Z ∈ 𝔤_{t,0}. Then (7.1)(c) implies that

ω_{X}(Y, Z) = s⟨Y, Z⟩ = −t⟨Y, Z⟩,

and (b) follows.

For (c), list the positive eigenvalues of ad(X) as t_{1} < t_{2} < ⋯ < t_{l}. Suppose Y_{i} and Z_{i} are elements of 𝔤_{t_{i},0}. Then we calculate

Ad(exp Y_{i})(X + ∑_{j} Z_{j}) = exp(ad(Y_{i}))(X + ∑_{j} Z_{j}).  (7.3)(a)

On the right side we are exponentiating the linear transformation ad(Y_{i}) of 𝔤_{0}. Now ad(Y_{i})(X) = −t_{i}Y_{i}, and ad(Y_{i}) carries 𝔤_{t,0} into 𝔤_{t+t_{i},0}. Consequently

ad(Y_{i})(𝔤_{t_{j},0}) ⊂ ∑_{k>i} 𝔤_{t_{k},0}.  (7.3)(b)

Inserting this information in (7.3)(a) gives

Ad(exp Y_{i})(X + ∑_{j} Z_{j}) = X + ∑_{j=1}^{i−1} Z_{j} + (Z_{i} − t_{i}Y_{i}) + ∑_{j=i+1}^{l} p_{j}(Y_{i}, Z_{1}, …, Z_{j}).  (7.3)(c)

Here p_{j}(Y_{i}, Z_{1}, …, Z_{j}) ∈ 𝔤_{t_{j},0} depends in a polynomial way on Y_{i} and linearly on the various Z_{k} (with k ≤ j).

We now consider the map from 𝔫_{X,0} to X + 𝔫_{X,0} defined by

π(Y_{1} + ⋯ + Y_{l}) = Ad(exp(Y_{l}) ⋯ exp(Y_{1}))(X).  (7.3)(d)

We calculate the adjoint action here one factor at a time, using (7.3)(c). The conclusion is

π(Y_{1} + ⋯ + Y_{l}) = X + ∑_{j=1}^{l}(−t_{j}Y_{j} + q_{j}(Y_{1}, … Y_{j−1}))  (7.3)(e)

Here q_{j}(Y_{1}, … Y_{j−1}) ∈ 𝔤_{t_{j},0} depends in a polynomial way on the various Y_{k}. This description shows that π is a diffeomorphism.

The map π is a composition of

τ : 𝔫_{X,0} → N_{X}, τ(Y_{1} + ⋯ + Y_{l}) = exp(Y_{1}) ⋯ exp(Y_{l}),

and the map γ of (c) in the proposition. An argument along similar lines to the one just given shows that τ is surjective. Whenever γ is smooth, τ is smooth and surjective, and γ ∘τ is a diffeomorphism, it follows that γ and τ are diffeomorphisms. This proves (c). Part (d) is just a reformulation of (c) using the identification of Lemma 5.2.

For (e), N_{X} is connected by definition. The proof of (c) provided a diffeomorphism τ from 𝔫_{X,0} to N_{X}, so N_{X} is simply connected. Nilpotence is immediate from (a). By (c), only the identity element of N_{X} fixes X, so G_{X} meets N_{X} exactly in the identity element. The adjoint action of G_{X} preserves each eigenspace 𝔤_{t,0} of ad(X), and so preserves 𝔫_{X,0}. It follows that G_{X} normalizes N_{X}. It follows that P_{X} is a subgroup of G. As a consequence of (c), we have

P_{X} = {g ∈ G | Ad(g)(X + 𝔫_{X,0}) = (X + 𝔫_{X,0})}.

This shows that P_{X} is closed in G. That P_{X} is parabolic means that it contains a minimal parabolic subgroup of G. This is more or less obvious from standard constructions of minimal parabolic subgoups; we omit the details. (Because we do not require G to be in the Harish-Chandra class, there is some question about exactly how a minimal parabolic subgroup should be defined. We define it so that a Levi subgroup is the centralizer of a maximal abelian subalgebra consisting of hyperbolic elements. Such a subspace is just a G-conjugate of a Cartan subspace in 𝔭_{0} (Definition 6.1).)

For (f), it follows from (e) that τ_{P_{X}} is well-defined. Because distinct eigenspaces of ad(X) are orthogonal with respect to ⟨, ⟩, the linear functional ξ_{X} is zero on 𝔤_{t,0} for t ≠ 0. In particular ξ_{X} vanishes on 𝔫_{X}; so τ_{P_{X}} satisfies condition (a) of Definition 4.12. Condition (b) of the definition follows from (e) of the proposition. By (b) of the proposition, the non-degeneracy of ω_{X} forces the pairing it defines between 𝔤_{t,0} and 𝔤_{−t,0} to be non-degenerate for t ≠ 0; so in particular these two spaces have the same dimension. Consequently

dim 𝔭_{X,0}/𝔤_{X,0} = dim 𝔫_{X,0} = (dim 𝔤_{0}/𝔤_{X,0})/2

This is condition (c) of Definition 4.12, proving that (τ_{P_{X}} , P_{X}) is an invariant real polarization. That τ_{P_{X}} is the only extension of τ is elementary; we omit the details. Q.E.D.

Although this construction of polarizations was the main goal of this section, we may as well discuss orbit data in this setting.

Proposition 7.4. In the setting of (7.1), suppose that ξ is a hyperbolic element of 𝔤_{0}^{∗}. Then the set of integral orbit data at ξ is naturally in one-to-one correspondence with the irreducible representations of the (finite) group of connected components of G_{ξ}. In particular, this set is non-empty, so the orbit G · ξ is integral.

Proof. By Lemma 5.2, we may write ξ = ξ_{X} with X a hyperbolic element of 𝔤_{0}. By Theorem 6.2, we may (after conjugating by an element of G) assume that X ∈ 𝔭_{0}. By Proposition 1.11(b), it follows that G_{ξ} is a real reductive group with Cartan involution θ|_{G_{ξ}}. Obviously the linear functional on 𝔤_{ξ,0} defined by X is just ξ|𝔤_{X,0}. What this means is that we have reduced Proposition 7.4 to the case G = G_{ξ}, which we now assume. This assumption means precisely that X belongs to

\begin{aligned}𝔞_{1,0} & = \{Z ∈ 𝔭_{0} | G_{Z} = G\} \\ & = \{Z ∈ 𝔭_{0} | ad(𝔤)(Z) = 0, Ad(K)(Z) = Z\}. \end{aligned}  (7.5)(a)

Now define

𝔭_{1,0} = orthogonal complement of 𝔞_{1,0} in 𝔭_{0}.  (7.5)(b)

It is easy to check that 𝔤_{1,0} = 𝔨_{0} + 𝔭_{1,0} is a Lie subalgebra of 𝔤_{0}. It is the orthogonal complement of 𝔞_{1,0} in 𝔤_{0}. Define

G_{1} = K · exp 𝔭_{1,0},  A_{1} = exp 𝔞_{1,0}.  (7.5)(c)

If Y ∈ 𝔭_{1,0} and Z ∈ 𝔞_{1,0}, then Y and Z commute; so exp(Y + Z) = exp(Y) exp(Z). Now it follows from the Cartan decomposition (Proposition 1.5) that G_{1} is a real reductive group, and that

G = G_{1} × A_{1},  (7.5)(d)

a direct product. Furthermore A_{1} is a vector group, isomorphic to its Lie algebra under the exponential map. The group G_{1} has compact center.

Now the linear functional ξ_{X} is given by inner product with X ∈ 𝔞_{1,0}, and so is trivial on 𝔤_{1,0}. According to Definition 4.7, an integral orbit datum at ξ is an irreducible unitary representation (τ, H_{τ}) of G, such that τ is trivial on the identity component G_{1,0}, and

τ(exp Z) = e^{i⟨Z,X⟩} · Id_{H_{τ}}

for Z ∈ 𝔞_{1,0}. Evidently such representations correspond precisely to the representations of G_{1}/G_{1,0} ≃ G/G_{0}, as we wished to show. Q.E.D.

Propositions 7.2 and 7.4, together with the construction of (4.16), provide a finite set of unitary representations of G attached to each hyperbolic coadjoint orbit. It can be shown that these representations are all irreducible. We are left with two (closely related) problems: to interpret this construction as part of a solution of Problem 3.15; and then to extend it to a wider class of coadjoint orbits. In the section 9 we will treat the first of these problems. The Dixmier algebras we need will be algebras of differential operators. In order to clarify the formal algebraic construction we will use for these algebras, we examine separately the familiar case of ℝ^{n}.

8. Taylor series and differential operators on ℝ^{n}.

Suppose f ∈ C^{∞}(ℝ^{n}). The Taylor series of f at 0 is a formal power series

∑_{α∈ℕ^{n}} c_{α}(f)x^{α},  c_{α}(f) = \frac{1}{α!} \frac{∂^{α}f}{∂x^{α}} (0) .  (8.1)(a)

(Here we use standard conventions for multi-indices: x^{α} = x_{1}^{α_{1}} ⋯ x_{n}^{α_{n}}, α! = α_{1}! ⋯ α_{n}!, and so on.) Every formal power series arises as the Taylor series of some smooth function f. We write ℂ[[x_{1}, …, x_{n}]] for the space of all formal power series. Then formation of Taylor series provides a surjective linear map

T : C^{∞}(ℝ^{n}) → ℂ[[x_{1}, …, x_{n}]],  T (f) = ∑_{α∈ℕ^{n}} c_{α}(f)x^{α}.  (8.1)(b)

What is essential about the Taylor series is the collection of complex numbers c_{α}; or, what amounts to the same thing, the values of all the derivatives of f at 0. To formalize this point of view, define

U = ℂ[∂/∂x_{1}, …, ∂/∂x_{n}]

= algebra of constant coefficent differential operators on ℝ^{n}.

(8.1)(c)

This algebra has as a basis the operators ∂^{α}/∂x^{α}. To specify a linear functional on U, we must therefore specify its value at each of these basis elements; and these values can be arbitrary. That is,

Hom_{ℂ}(U, ℂ) ≃ families of complex numbers {t_{α} | α ∈ ℕ^{n}}.  (8.1)(d)

We may therefore regard Taylor series as a surjective map

τ : C^{∞}(ℝ^{n}) → Hom_{ℂ}(U, ℂ),  τ(f)(u) = (u · f)(0).  (8.1)(e)

The connection with the formal power series in (8.1)(a) is

c_{α}(f) = \frac{1}{α!}(τ(f))(∂^{α}/∂x^{α}).  (8.1)(f)

We could also describe this as an isomorphism

Hom_{ℂ}(U, ℂ) ≃ ℂ[[x_{1}, …, x_{n}]],  μ ↦ ∑_{α} \frac{1}{α!}μ(∂^{α}/∂x^{α})x^{α}.  (8.1)(g)

We want to understand the algebra D^{an}(ℝ^{n}) of differential operators on ℝ^{n} with analytic coefficients. Such operators—indeed, even the algebra D^{form}(ℝ^{n}) of differential operators with formal power series coefficients—act on ℂ[[x_{1}, …, x_{n}]], and therefore on Taylor series. These actions are faithful. (For D^{an}, the reason is that a non-zero differential operator cannot annihilate all analytic functions.) Consequently

U ⊂ D^{an}(ℝ^{n}) ⊂ D^{form}(ℝ^{n}) ⊂ End(ℂ[[x_{1}, …, x_{n}]]) ≃ End(Hom_{ℂ}(U, ℂ)).  (8.2)

The problem is to identify the differential operators among all the linear transformations of formal power series. One often thinks of differential operators as characterized by the property of not increasing support. By considering only the action on Taylor series, we have in some sense already taken advantage of that property. What we can study on the level of Taylor series is order of vanishing. For this purpose it is helpful to write

|α| = α_{1} + ⋯ + α_{n}  (α ∈ ℕ^{n}).  (8.3)(a)

Define

U_{p} = span of monomials ∂^{α}/∂x^{α} with |α| ≤ p,  (8.3)(b)

the constant coefficient operators of order at most p. (We will sometimes be careless about this terminology, saying that any element of U_{p} is of order p even though it might belong to U_{p−1}.) These subspaces form an increasing filtration of U. Similarly we can define

D_{p}^{an}(ℝ^{n}) = \{ ∑_{|α|≤p} f_{α}(x)\frac{∂^{α}}{∂x^{α}}  (f_{α} ∈ C^{ω}(ℝ^{n}))\}  (8.3)(c)

D_{p}^{form}(ℝ^{n}) = \{ ∑_{|α|≤p} f_{α}(x)\frac{∂^{α}}{∂x^{α}}  (f_{α} ∈ ℂ[[x_{1}, …, x_{n}]])\},  (8.3)(d)

the differential operators of order at most p with analytic or formal power series coefficients. By the leading coefficients of an operator in D_{p}^{form} written as in (8.3)(d), we will mean the formal power series f_{α} with |α| = p. This means that before we can speak of leading coefficients, we must specify the order p we have in mind. If the operator happens to belong to D_{p−1}^{form}, then its leading coefficients are all zero. Now define

Hom_{ℂ}(U, ℂ)_{p} = {μ ∈ Hom_{ℂ}(U, ℂ) | μ(U_{p}) = 0}.  (8.3)(e)

Under the isomorphism (8.1)(g), this corresponds to

ℂ[[x_{1}, … x_{n}]]_{p} = { ∑ c_{α}x^{α} | c_{α} = 0, all |α| ≤ p}.  (8.3)(f)

Proposition 8.4. Suppose f ∈ C^{∞}(ℝ^{n}), and p ≥ 0. The following conditions on f are equivalent.

a) For every |α| ≤ p, (∂^{α}f/∂x^{α})(0) = 0.

b) For every u ∈ U_{p}, (u · f)(0) = 0 (cf. (8.3)(b)).

c) For every S ∈ D^{an}(ℝ^{n})_{p}, (S · f)(0) = 0 (cf. (8.3)(c)).

d) The Taylor series T (f) (cf. (8.1)(b)) belongs to ℂ[[x_{1}, … x_{n}]]_{p}.

e) The Taylor series τ(f) (cf. (8.1)(e)) belongs to Hom_{ℂ}(U, ℂ)_{p}.

This is elementary. When the conditions are satisfied, we say that f vanishes to order p at zero, and we write

f ∈ C^{∞}(ℝ^{n})_{p}.  (8.5)

Because of the proposition, we may refer to (say) Hom_{ℂ}(U, ℂ)_{p} as the space of Taylor series vanishing to order p. The first property distinguishing differential operators among all endomorphisms of Taylor series is this.

Proposition 8.6. Suppose f ∈ C^{∞}(ℝ^{n})_{p} (cf. (8.5)) and S ∈ D_{q}^{an}(ℝ^{n}) (cf. (8.3)(c)). Then S · f ∈ C^{∞}(ℝ^{n})_{p−q}. Here we write C^{∞}(ℝ^{n})_{r} = C^{∞}(ℝ^{n}) for r < 0.

Similarly, suppose μ ∈ Hom_{ℂ}(U, ℂ)_{p} is a Taylor series vanishing to order p (cf. (8.3)(b)), and S ∈ D_{q}^{form}(ℝ^{n}). Then S · f ∈ Hom_{ℂ}(U, ℂ)_{p−q} is a Taylor series vanishing to order p − q. Here we write Hom_{ℂ}(U, ℂ)_{r} = Hom_{ℂ}(U, ℂ) for r < 0.

This is an immediate consequence of the definitions and of the fact that the filtrations in (8.3) respect the algebra structures:

U_{q}U_{r} ⊂ U_{q+r},  D_{q}^{an}(ℝ^{n})D_{r}^{an}(ℝ^{n}) ⊂ D_{q+r}^{an}(ℝ^{n}),  D_{q}^{form}(ℝ^{n})D_{r}^{form}(ℝ^{n}) ⊂ D_{q+r}^{form}(ℝ^{n}).  (8.7)

With this property in mind, we say that a linear transformation A of Hom_{ℂ}(U, ℂ) is weakly of order q if

A · Hom_{ℂ}(U, ℂ)_{p} ⊂ Hom_{ℂ}(U, ℂ)_{p−q}  (8.8)

for all p ≥ 0. (The condition is non-empty only for p ≥ q).

The most obvious linear transformations weakly of order q are the differential operators of order q with formal power series coefficients. There are more, however. When n = 1, the differential operator A = x\frac{∂}{∂x} acts on formal power series by

A( ∑_{j≥0} c_{j}x^{j}) = ∑_{j≥0} jc_{j}x^{j}.

This action obviously preserves the subspaces ℂ[[x]]_{p} of (8.3)(f), so A is weakly of order 0. As a differential operator, however, A is first order. A little reflection shows that the problem is the vanishing at zero of the coefficient of the highest derivative in A. Here is a precise statement.

Lemma 8.9. Suppose

S = ∑_{|α|≤p} σ_{α}(x)\frac{∂^{α}}{∂x^{α}} (σ_{α} ∈ ℂ[[x_{1}, …, x_{n}]])

is a differential operator with formal power series coefficients of order at most p. Assume that there is a β with |β| = p and σ_{β}(0) ≠ 0. (Here σ_{β}(0) means the constant term of the formal power series σ_{β}.) Then S is not weakly of order p − 1.

Proof. The formal power series f = x^{β} vanishes to order p − 1 at 0; but (S · f)(0) = β! σ_{β}(0) is not zero. So S · f does not vanish to order 0 at 0, so S cannot be weakly of order p − 1. Q.E.D.

The lemma says that the notion of weak order for endomorphisms of Taylor series allows us to detect the order of differential operators whose leading coefficients do not vanish at zero. To continue, we need a way to construct such differential operators from arbitrary ones (with formal power series coefficients.) That is, we need a way to reduce the order of vanishing of the leading coefficients. The way to do that is to differentiate those coefficients. The next lemma provides a way to differentiate leading coefficients while still thinking of the operators as acting on Taylor series. Recall that a vector field on ℝ^{n} with formal power series coefficients is a first-order differential operator ξ that annihilates the constant function:

ξ = ∑_{i=1}^{n} f_{i}(x)\frac{∂}{∂x_{i}}  (f_{i} ∈ ℂ[[x_{1}, …, x_{n}]])  (8.10)(a)

The symbol of ξ at 0 is the vector

σ(ξ) = (f_{i}(0)) ∈ ℂ^{n}  (8.10)(b)

We fix now a set {ξ_{1}, …, ξ_{N}} of vector fields, and assume that

the symbols σ(ξ_{1}), …, σ(ξ_{N}) span ℂ^{n}.  (8.10)(c)

(For the purposes of this section we could just take the n vector fields ∂/∂x_{i}, but for the applications in section 9 the more general assumption will be useful.) As a consequence of (8.10)(c), we can draw the following conclusion. Suppose f is a non-zero formal power series. Then there is an integer r ≥ 0 so that f vanishes to order r − 1 at 0, but f does not vanish to order r. For this value of r, we can find a sequence (i_{1}, …, i_{r}) ∈ {1, …, N}^{r} so that

(ξ_{i_{1}} ⋯ ξ_{i_{r}}f)(0) ≠ 0.  (8.10)(d)

Here is the lemma that allows us to apply these ideas to differential operators.

Lemma 8.11. Suppose

S = ∑_{|α|≤p} σ_{α}(x)\frac{∂^{α}}{∂x^{α}} (σ_{α} ∈ ℂ[[x_{1}, …, x_{n}]])

is a differential operator with formal power series coefficients of order at most p; and suppose

ξ = ∑_{i=1}^{n} f_{i}(x)\frac{∂}{∂x_{i}} (f_{i} ∈ ℂ[[x_{1}, …, x_{n}]])

is a vector field. Write

T = [ξ, S] = ξ ∘ S − S ∘ ξ

for the commutator of S and ξ. Then T is a differential operator of order at most p with formal power series coefficients:

T = ∑_{|α|≤p} τ_{α}(x)\frac{∂^{α}}{∂x^{α}} (τ_{α} ∈ ℂ[[x_{1}, …, x_{n}]])

Assume that the leading coefficients of S all vanish to order k at x = 0: that is, that

σ_{α} ∈ ℂ[[x_{1}, …, x_{n}]]_{k} whenever |α| = p.

If |β| = p, then

τ_{β} − ξ · σ_{β} ∈ ℂ[[x_{1}, …, x_{n}]]_{k}.

That is, the leading coefficients of T are obtained from those of S by applying the differential operator ξ, up to terms vanishing to order k at x = 0.

This can be proved by a straightforward computation, which we omit.

Corollary 8.12. Suppose S is a differential operator of order at most p with formal power series coefficients. Assume that the leading coefficients of S vanish to order r − 1 at x = 0, but that some leading coefficient does not vanish to order r. Then we can find a sequence (i_{1}, …, i_{r}) ∈ {1, …, N}^{r} so that some leading coefficient of the iterated commutator

T = [ξ_{i_{1}}, ⋯ [ξ_{i_{r}}, S] ⋯]

has a leading coefficient that does not vanish at x = 0.

This is immediate from the lemma and (8.10)(d).

Corollary 8.12 suggests how to refine the definition of weak order q given at (8.8). In the setting of (8.10), we say that an endomorphism A of Hom_{ℂ}(U, ℂ) is of order q if for every sequence (i_{1}, …, i_{r}) ∈ {1, …, N}^{r}, the iterated commutator

[ξ_{i_{1}}, ⋯ [ξ_{i_{r}}, A] ⋯]  (8.13)

is weakly of order q. The definition appears to depend on the choice of the vector fields ξ_{i}, but Theorem 8.15 shows (subject to the assumption (8.10)(c)) that it does not.

Lemma 8.14. Suppose S is a non-zero differential operator with formal power series coefficients. Then the order of S as a differential operator is equal to its order as an endomorphism of the space Hom_{ℂ}(U, ℂ) of formal power series (cf. (8.13)).

This follows from Corollary 8.12, Lemma 8.9, and the definitions.

Here is the main theorem of this section.

Theorem 8.15. Suppose T is an endomorphism of Taylor series of order less than or equal to p (cf. (8.13)). Then T is a differential operator with formal power series coefficients of order less than or equal to p.

Sketch of proof. By an elementary calculation, we can find a differential operator D of order less than or equal to p, with formal power series coefficients, having the property that

D(x^{α}) = T (x^{α}) (|α| ≤ p).

(This amounts to solving a finite system of linear equations in the ring of formal power series. The coefficient matrix is square and upper triangular with non-zero integers on the diagonal; so the system is solvable.) After replacing T by T − D, we may therefore assume that

T (x^{α}) = 0  (|α| ≤ p).  (8.16)(a)

On the other hand, if |β| ≥ p + 1, then x^{β} vanishes to order at least p at 0, so T (x^{β}) vanishes to order at least 0. That is

T (x^{β})(0) = 0  (|β| ≥ p + 1).  (8.16)(b)

We are now trying to show that T = 0. The first problem is to deduce from the preceding formulas a statement about the effect of T on arbitrary formal power series. For that we use the natural topology on formal power series, in which the various subspaces ℂ[[x_{1}, …, x_{n}]]_{p} of series vanishing to order p form a neighborhood base at 0. This means that a sequence f_{m} of formal power series converges to f if f − f_{m} vanishes to order at least r_{m}, and r_{m} goes to infinity with m. In this topology, a formal power series ∑ c_{α}x^{α} is the limit of its partial sums ∑_{|α|≤m} c_{α}x^{α}. An operator A that is weakly of order p is continuous, and therefore

A (∑ c_{α}x^{α}) = lim_{m→∞} ∑_{|α|≤m} c_{α}A(x^{α}).

This means that any fixed coefficient of the formal power series A(∑ c_{α}x^{α}) is equal to the corresponding coefficient of ∑_{|α|≤m} c_{α}A(x^{α}) for m sufficiently large.

Bearing in mind these remarks, we find that (8.16)(a) and (b) imply

T (f)(0) = 0  (f ∈ ℂ[[x_{1}, …, x_{n}]]).  (8.16)(c)

We are trying to show that T = 0. Because of (8.10)(d), it suffices to show that if (i_{1}, …, i_{r}) ∈ {1, …, N}^{r}, then

(ξ_{i_{1}} ⋯ ξ_{i_{r}}T f)(0) = 0.  (8.16)(d)

We will prove this statement by induction on r, simultaneously with

([ξ_{i_{1}}, [⋯ [ξ_{i_{r}}, T]] ⋯]f)(0) = 0.  (8.16)(e)

The case r = 0 is precisely (8.16)(c); so suppose r ≥ 1, and that (8.16)(d) and (e) are known for r − 1. The iterated commutator in (8.16)(e) may be expanded as a sum of terms of the form ±(ξ_{I}T ξ_{J}f)(0). Here I and J are disjoint ordered sets whose union is {i_{1}, …, i_{r}}, and ξ_{I} is the composition of the vector fields ξ_{i} (with i ∈ I) in the order determined by I. Just one of these terms has |I| = r: it is exactly the term appearing in (8.16)(d). All of the other terms have |I| < r, and therefore vanish by inductive hypothesis. It follows that (for a fixed f) (8.16)(d) and (8.16)(e) are equivalent.

To prove one of these statements, it is enough (by the remarks about topology above) to take f = x^{α}. If |α| ≤ p, then (8.16)(d) follows from (8.16)(a). So suppose |α| > p; then x^{α} vanishes to order at least p at 0. By hypothesis the iterated commutator in (8.16)(e) is weakly of order at most p; so the iterated commutator applied to x^{α} vanishes to order at least 0 at 0, as required by (8.16)(e). This completes the induction and therefore the proof. Q.E.D.

9. Twisted differential operator algebras.

Problem 3.15 suggests that we should construct a representation by first constructing a Dixmier algebra (A, ϕ), and that the action of U(𝔤) on the representation should be given by ϕ and an action of A. The construction given in section 7, which may be summarized as Ind_{P_{X}}^{G} (τ_{P_{X}}), does not proceed in this way; but it can be rearranged to fit better. For the moment we retain the setting of section 7, although that will soon be modified. The Hilbert space of the induced representation is L^{2}(G/P_{X}, H_{τ} ⊗ D_{1/2}). The group G acts by left translation of sections of an equivariant bundle V_{τ} (whose fiber at eP_{X} is H_{τ} ⊗ D_{1/2}). Consequently elements of the complexified Lie algebra 𝔤 act by first order real analytic differential operators on V_{τ}; and elements of U(𝔤) act by real analytic differential operators on V_{τ}. (Notice that general elements of the Hilbert space of the representation are only L^{2} sections of V_{τ}, rather than smooth ones; so the differential operators will not really act on them. But it turns out (since G/P_{X} is compact) that the smooth vectors of the representation (Definition 2.6) are precisely the smooth sections of V_{τ}, and differential operators do act on these.)

This discussion suggests a candidate for the Dixmier algebra A: it might be the algebra D_{τ}^{an} of all real analytic differential operators on V_{τ}. We have seen that there is a natural algebra homomorphism ϕ : U(𝔤) → D_{τ}^{an}. The action of G on G/P_{X} provides an action Ad of G on D_{τ}^{an} by algebra automorphisms, and this is easily seen to be compatible with ϕ and the adjoint action on U(𝔤).

The first problem is that the action of G on D_{τ}^{an} is not algebraic in any sense. A basic property of algebraic representations is that they are locally finite: any vector is contained in a finite-dimensional G-invariant subspace. The differential operators of order zero include the multiplication operators m_{f} by analytic functions f on G/P_{X}. The adjoint action on m_{f} is by translation of the function f. If f is not locally constant, then its translates can never span a finite-dimensional representation of G. (Proving that statement is a good exercise in elementary representation theory.) One way out of this difficulty is to define D_{τ}^{alg} as the largest subspace of D_{τ}^{an} on which the adjoint action is algebraic. Since the adjoint action on U(𝔤) is algebraic, we will have ϕ : U(𝔤) → D_{τ}^{alg}. It is possible to make sense of this idea, but it will be convenient for us to adopt a slightly less direct approach.

We are now prepared to formulate working hypotheses for the section. Suppose that

H_{ℂ} ⊂ G_{ℂ} are complex connected algebraic groups.  (9.1)(a)

We write 𝔥 ⊂ 𝔤 for the corresponding Lie algebras. Fix a Lie algebra homomorphism

λ : 𝔥 → ℂ  (9.1)(b)

This amounts to a one-dimensional representation

λ : 𝔥 → End(ℂ_{λ}).  (9.1)(b′)

One particular homomorphism plays a special rôle:

δ : 𝔥 → ℂ,  δ(X) = \frac{1}{2}tr(ad(X) on 𝔤/𝔥)  (9.1)(c)

Eventually we will assume

G_{ℂ} is reductive and H_{ℂ} is a parabolic subgroup.  (9.1)(d)

These are all the assumptions and data required for the main construction. In order to motivate the construction, however, it will sometimes be helpful to assume more. For those purposes, we may sometimes assume

G is a real form of G_{ℂ}, and H = H_{ℂ} ∩ G is a real form of H_{ℂ}; and  (9.1)(M1)
iλ is the differential of a one-dimensional unitary character (Λ, ℂ_{Λ}) of H.  (9.1)(M2)

(The M in the equation labels stands for “motivation.”) We will then write 𝔥_{0} ⊂ 𝔤_{0} for the real Lie algebras. Under these assumptions the real analytic manifold G/H is a real form of the complex manifold (actually a complex algebraic variety) G_{ℂ}/H_{ℂ}. Then H acts by the adjoint action on the real vector space 𝔤_{0}/𝔥_{0} (the real tangent space at the base point to G/H); so H acts on the one-dimensional real vector space D^{1/2}(𝔤_{0}/𝔥_{0}). We get

δ is the differential of the character Δ of H on D^{1/2}(𝔤_{0}/𝔥_{0}).  (9.1)(M3)

Using this character, we can define

L_{Λ} = line bundle on G/H induced by Λ ⊗ Δ.  (9.1)(M4)

C_{0,Λ}^{∞} = space of compactly supported smooth sections of L_{Λ}.  (9.1)(M5)

In the setting of (4.16), the Hilbert space of the unitary representation Ind_{H}^{G}(Λ) is constructed as a completion of C_{0,Λ}^{∞}.

Here is the main result. It is a folk theorem, much harder to attribute correctly than to prove. The main ideas go back at least to Kirillov’s work in the 1960’s on nilpotent groups, and probably much earlier. The only difficult part of this formulation (that D_{λ} is a Dixmier algebra in the setting of (9.1)(d)) can be deduced from [Conze-Berline–Duflo]. The paper [Beilinson-Bernstein] demonstrated the particular importance of twisted differential operators on flag varieties, and [Borho-Brylinski82] and [Borho-Brylinski85] provide a thorough treatment.

Theorem 9.2. Suppose we are in the setting of (9.1)(a) and (b). Then there is a completely prime algebra D(G_{ℂ}/H_{ℂ})_{λ} = D_{λ}, called a twisted differential operator algebra. This algebra is endowed with an algebraic action Ad of G_{ℂ} by algebra automorphisms, and with an algebra homomorphism

ϕ_{λ} : U(𝔤) → D_{λ}.

The adjoint actions of G_{ℂ} on U(𝔤) and D_{λ} are compatible with ϕ_{λ}. The differential ad of the adjoint action is the difference of the left and right actions of 𝔤 defined by ϕ_{λ}:

ad(X)(T) = ϕ_{λ}(X)T − T ϕ_{λ}(X) (X ∈ 𝔤, T ∈ D_{λ})

If in addition (9.1)(d) is satisfied, then (D_{λ}, ϕ_{λ}) is a Dixmier algebra for G_{ℂ} (Definition 3.8).

Suppose that the auxiliary conditions (9.1)(M1) and (9.1)(M2) are satisfied. Then D_{λ} is isomorphic to a subalgebra of the analytic differential operators on sections of L_{Λ} (cf. (9.1)(M4) and (9.1)(M5)). The adjoint action of G arises by change of variables from the action of G on G/H by left translation; and the homomorphism ϕ_{λ} from the natural action of U(𝔤) by differential operators on C_{0,Λ}^{∞}.

Proof. The statements in the last paragraph are intended to guide the construction of D_{λ} in general. The problem we face is essentially to find a description of the differential operators on L_{Λ} that refers only to G, H, and λ. To that end, recall that the differential operators are certain endomorphisms of C_{0,Λ}^{∞}. The operators we want are actually going to have analytic coefficients. This suggests the possibility of studying them by means of Taylor series expansions. As a first step, we need to understand Taylor series for sections of L_{Λ}. Now the Taylor series of a function f at a point p can be thought of as a list of all the values at p of derivatives of f. If f is not a function but a section of a bundle, then we should apply differential operators on sections of the bundle, and the values will lie in the fiber of the bundle at p. Here is a description of Taylor series for sections of L_{Λ}.

Lemma 9.3. In the setting of (9.1)(M), there is a surjective linear map (Taylor series at eH)

τ : C_{0,Λ}^{∞} → Hom_{U(𝔥)}(U(𝔤), ℂ_{iλ+δ}).

(Here the Hom is defined using the left action of U(𝔥) on U(𝔤).) The kernel of τ consists of all sections vanishing to infinite order at eH.

Sketch of proof. The fiber of L_{∗} at eH is the one-dimensional representation ℂ_{Λ} ⊗ D^{1/2}(𝔤_{0}/𝔥_{0}) of H. Because of (9.1)(M2) and (9.1)(M3), the Lie algebra 𝔥 acts on this fiber by iλ + δ; so (after fixing a choice of a half density on 𝔤_{0}/𝔥_{0}) we may identify the fiber with ℂ_{iλ+δ}. The space C_{0,Λ}^{∞} may be identified with a certain space of smooth functions on G with values in ℂ_{iλ+δ} (transforming appropriately under H on the right, and satisfying a support condition.) The group G acts on this space by left translation, and the Lie algebra 𝔤_{0} acts by right-invariant vector fields. Explicitly,

(X · σ)(g) = \frac{d}{dt}(σ(exp(−tX)g))|_{t=0}  (σ ∈ C_{0,Λ}^{∞}, X ∈ 𝔤_{0}, g ∈ G)  (9.4)(a)

This is a Lie algebra representation, and so extends to an action of the universal enveloping algebra on C_{0,Λ}^{∞}. (This is the action mentioned at the end of Theorem 9.2.) The Taylor series map τ is defined by

τ(σ)(u) = (u · σ)(1)  (σ ∈ C_{0,Λ}^{∞}, u ∈ U(𝔤))  (9.4)(b)

If Z ∈ 𝔥_{0}, then

τ(σ)(Zu) = (Z · u · σ)(1)

= \frac{d}{dt}(u · σ)(exp(−tZ))|_{t=0}

= \frac{d}{dt}[(Λ ⊗ Δ)(exp tZ)] · (u · σ)(1)|_{t=0}

= [(iλ + δ)(Z)] · (u · σ(1)) = [(iλ + δ)(Z)] · τ(σ)(u).

(The third equality follows from the transformation property of u · σ under H on the right.) This shows that τ maps to the correct Hom space. Surjectivity of τ is equivalent (after introducing appropriate local coordinates on G/H and using the Poincare-Birkhoff-Witt Theorem) to the fact that every formal power series on ℝ^{n} is the Taylor series at 0 of a smooth function. The assertion about the kernel of τ is clear from (9.4)(b). Q.E.D.

We want to investigate the space of Taylor series more closely. Notice first that

M_{λ} = Hom_{U(𝔥)}(U(𝔤), ℂ_{iλ+δ})  (9.5)(a)

may be defined in the setting (9.1)(a–d); the additional structure of (9.1)(M) is not used. The complex vector space M_{λ} carries a U(𝔤)-module structure:

(v · μ)(u) = μ(uv)  (u, v ∈ U(𝔤), μ ∈ M_{λ}).  (9.5)(b)

Sometimes it will be convenient to write this action as an algebra homomorphism

ϕ_{λ} : U(𝔤) → End(M_{λ}),  (ϕ_{λ}(v)μ)(u) = μ(uv).  (9.5)(b′)

Because we use the right action of U(𝔤) on itself, v · μ inherits from μ the transformation property under U(𝔥) on the left, and so belongs to M_{λ}. Next, recall the standard filtration on U(𝔤): U_{n}(𝔤) is the span of all products of less than or equal to n elements of 𝔤. In terms of the realization of U(𝔤) as right-invariant differential operators on a real form G, U_{n}(𝔤) corresponds to the differential operators of order at most n. We define the order of vanishing filtration of M_{λ} by

M_{λ,n} = {μ ∈ M_{λ} | μ(U_{n}(𝔤)) = 0}  (9.5)(c)

This is a decreasing filtration of M_{λ}:

M_{λ} ⊃ M_{λ,0} ⊃ M_{λ,1} ⊃ ⋯,  ⋂_{n=0}^{∞} M_{λ,n} = 0.  (9.5)(d)

(It is often useful to define U_{−1}(𝔤) = 0, and M_{λ,−1} = M_{λ}.) The action of U(𝔤) is compatible with the filtrations in the following sense:

U_{p}(𝔤) · M_{λ,q} ⊂ M_{λ,q−p}.  (9.5)(e)

Lemma 9.6. In the setting of (9.1)(M), the Taylor series map τ of Lemma 9.3 intertwines the action of U(𝔤) on C_{0,Λ}^{∞} (cf. (9.4)(a)) and on M_{λ} (cf. (9.5)(b)). The subspace M_{λ,n} is precisely the image under τ of sections vanishing to order at least n at eH.

Here we say that a section σ vanishes to order zero at a point p if σ(p) = 0; and we say that σ vanishes to order n if (T σ)(p) = 0 for every differential operator T of order at most n. We omit the simple proof.

Now that we have a space of Taylor series, Theorem 8.15 suggests how to define differential operators (with formal power series coefficients). The distinguished vector fields used in (8.13) are provided here by the action of 𝔤.

Definition 9.7. In the setting of (9.1) and (9.5), an endomorphism T of M_{λ} is said to be weakly of order q if T (M_{λ,p}) ⊂ M_{λ,p−q} for all p ≥ 0 (cf. (8.8)). It is of order q if for every sequence (X_{1}, …, X_{r}) of elements of 𝔤, the iterated commutator [ϕ_{λ}(X_{1}), [⋯ [ϕ_{λ}(X_{r}), T]] ⋯] is weakly of order q. We write D_{λ,q}^{form} for the collection of endomorphisms of order q, and

D_{λ}^{form} = ⋃_{q} D_{λ,q}^{form}.

It is not difficult to see that D_{λ}^{form} is a filtered algebra, isomorphic to the algebra of differential operators in dim 𝔤/𝔥 variables with formal power series coefficients. (Consequently D_{λ}^{form} is completely prime.) As a consequence of (9.5)(e), the map of (9.5)(b′) restricts to a filtered algebra homomorphism

ϕ_{λ} : U(𝔤) → D_{λ}^{form}.  (9.8)(a)

We make 𝔤 act on D_{λ}^{form} by

ad(X)(T) = [ϕ_{λ}(X), T].  (9.8)(b)

Because ϕ_{λ} is a Lie algebra homomorphism, ad is a Lie algebra representation of 𝔤 (by derivations) on D_{λ}^{form}. It therefore extends to an associative algebra homomorphism

ad : U(𝔤) → End(D_{λ}^{form}).  (9.8)(c)

The algebra D_{λ}^{form} is close to the requirements of Theorem 9.2. What is missing is the action Ad of G_{ℂ} by algebra automorphisms. This should be an exponentiated form of the action ad of 𝔤. We cannot perform this exponentiation on all of D_{λ}^{form}. Using an idea of Zuckerman, we will essentially define D_{λ} to be the largest subalgebra of D_{λ}^{form} on which the exponentiated action makes sense. Here is a simple version of Zuckerman’s idea.

Definition 9.9 (see [Vogan81], Definition 6.2.4). Suppose G_{ℂ} is a connected complex algebraic group with Lie algebra 𝔤, and V is a representation of 𝔤. We will define an algebraic representation Γ^{G_{ℂ}}(V) = ΓV of G_{ℂ}. (Recall that this means that every element of ΓV is to lie in a finite-dimensional G_{ℂ}-invariant subspace, on which the action of G_{ℂ} is algebraic.) As a vector space, ΓV will be a subspace of V; and the differential of the representation of G_{ℂ} is the original action of 𝔤. To do this, let G̃_{ℂ} be the universal covering group of G_{ℂ}. There is a short exact sequence

1 → Z → G̃_{ℂ} → G_{ℂ} → 1.

Here Z is a discrete central subgroup of G̃_{ℂ}. Define

Γ̃V = {v ∈ V | dim U(𝔤)v < ∞}.

This is a 𝔤-stable subspace of V, on which the action of 𝔤 is locally finite. By the dictionary between finitedimensional Lie group and Lie algebra representations, it follows that Γ̃V carries a locally finite representation of G̃_{ℂ}, with differential given by the action of 𝔤. Set

Γ_{0}V = {v ∈ Γ̃V | z · v = v, all z ∈ Z}.

Because Z is normal in G_{ℂ}, Γ_{0}V is a G̃_{ℂ}-invariant subspace of Γ̃V; and the representation of G̃_{ℂ} on Γ_{0}V obviously factors to G_{ℂ}. Finally, define

ΓV = {v ∈ Γ_{0}V | the function g ↦ g · v from G_{ℂ} to V is algebraic}.

This makes sense because the function takes values in a finite-dimensional subspace of V. It is more or less obvious that ΓV is an invariant subspace of Γ_{0}V, and that the representation of G_{ℂ} on ΓV is algebraic.

We now define

D_{λ} = Γ^{G_{ℂ}}(D_{λ}^{form}),  (9.10)(a)

and write Ad for the representation of G_{ℂ} on D_{λ}. Because 𝔤 acts by derivations, it is straightforward to check that Γ̃D_{λ} is a subalgebra of D_{λ}^{form} on which G̃_{ℂ} acts by algebra automorphisms. It follows easily that D_{λ} is a subalgebra on which G_{ℂ} acts algebraically by algebra automorphisms. Because the adjoint action of G_{ℂ} on U(𝔤) is algebraic, the image of ϕ_{λ} (cf. (9.8)(a)) is contained in D_{λ}:

ϕ_{λ} : U(𝔤) → D_{λ}.  (9.10)(b)

The algebra D_{λ} is completely prime because it is a subalgebra of the completely prime algebra D_{λ}^{form}.

This completes the verification of the statements in the first paragraph of Theorem 9.2. That D_{λ} is a Dixmier algebra under the hypothesis (9.1)(d) — that is, that the finiteness requirements of Definition 3.8 are satisfied — is well-known, going back at least to [Conze-Berline–Duflo]. (There is an explicit verification in [Vogan90], Corollary 4.17. One can also find there a discussion of the symbol calculus for D_{λ}.) The assertions about the embedding D_{λ} in an algebra of analytic differential operators we leave to the reader; the discussion in section 8 should help to make them plausible. Q.E.D.

Example 9.11. Suppose G_{ℂ} = ℂ^{n} and H_{ℂ} is trivial, and λ = 0. Then

M_{λ} ≃ ℂ[[x_{1}, …, x_{n}]]

(see (8.1)(g) and (9.5)(a)). According to Theorem 8.15 and Definition 9.7,

D_{λ}^{form} ≃ D^{form}(ℝ^{n}),

the algebra of differential operators on ℝ^{n} with formal power series coefficients. We also have

U(𝔤) ≃ ℂ[∂/∂x_{1}, …, ∂/∂x_{n}],

and the map ϕ_{λ} from U(𝔤) to D_{λ}^{form} is the natural embedding. It is therefore easy to compute the adjoint action defined in (9.8): if T ∈ U(𝔤) is a constant coefficient differential operator, then

ad(T) (∑_{α} f_{α} \frac{∂^{α}}{∂x^{α}} ) = ∑_{α}(T f_{α})\frac{∂^{α}}{∂x^{α}} .

In the setting of Definition 9.9, it follows that Γ̃(D_{λ}^{form}) consists of those differential operators whose coefficients f_{α} have the following property: the space of all derivatives of f_{α} is finite-dimensional. (Here “derivatives” refers to applying constant coefficient differential operators.) Using linear algebra and calculus, it is not hard to show that these functions are all finite linear combinations of polynomials times exponentials:

f_{α}(x) = ∑_{ξ∈ℂ^{n}} p_{α,ξ}(x) exp(∑ ξ_{j}x_{j}),

with each p_{α,ξ} a polynomial in x. The group G_{ℂ} is simply connected, so Γ̃(D_{λ}^{form}) = Γ_{0}(D_{λ}^{form}). The adjoint action of G_{ℂ} is by (complex) translation of the coefficients:

Ad(z) (∑_{α} f_{α} \frac{∂^{α}}{∂x^{α}} ) = ∑_{α}(ρ(z)f_{α})\frac{∂^{α}}{∂x^{α}} ,

with

(ρ(z)f_{α})(x) = f_{α}(x + z) = ∑_{ξ∈ℂ^{n}} p_{α,ξ}(x + z) exp(∑ ξ_{j}(x_{j} + z_{j})).

The algebraic functions on ℂ^{n} are the polynomials. From this it follows that ΓD_{λ}^{form} consists of differential operators with polynomial coefficients:

D_{λ} ≃ ℂ[x_{1}, …, x_{n}, ∂/∂x_{1}, …, ∂/∂x_{n}].

To complete this section, we describe the coadjoint orbits to which the Dixmier algebras D_{λ} should correspond in Conjecture 3.9. Assume then that we are in the setting (9.1)(a–d). We follow section 3 of [Vogan90], which the reader may consult for more details and generalizations. Define

Σ_{λ,G_{ℂ}/H_{ℂ}} = {ξ ∈ 𝔤^{∗} | ξ|_{𝔥} = λ}.  (9.12)(a)

This is an H_{ℂ}-stable affine subspace of 𝔤^{∗}, of dimension equal to the codimension of 𝔥 in 𝔤. We may therefore use it to construct the fiber product

Σ_{λ} = G_{ℂ} ×_{H_{ℂ}} Σ_{λ,G_{ℂ}/H_{ℂ}},  (9.12)(b)

an affine bundle over the projective variety G_{ℂ}/H_{ℂ}. There is a natural “moment map”

μ_{λ} : Σ_{λ} → 𝔤^{∗},  μ_{λ}(equivalence class of (g, ξ)) = Ad^{∗}(g)(ξ)  (9.12)(c)

Obviously the image of μ_{λ} is a union of coadjoint orbits.

Proposition 9.13. In the setting of (9.1)(a–d) and (9.12), the moment map μ_{λ} is proper and generically finite; its image is the closure of a single coadjoint orbit O_{λ}. More precisely, the parabolic subgroup H_{ℂ} has an open orbit O_{λ,G_{ℂ}/H_{ℂ}} on Σ_{λ,G_{ℂ}/H_{ℂ}}. This defines an open subvariety

Õ_{λ} = G_{ℂ} ×_{H_{ℂ}} O_{λ,G_{ℂ}/H_{ℂ}}

of Σ_{λ}. The restriction of the moment map μ_{λ} to Õ_{λ} is a finite covering map onto O_{λ}.

The most difficult case of the proposition is λ = 0. In that case it is essentially due to Richardson: Σ_{λ} is the cotangent bundle of the partial flag variety G_{ℂ}/H_{ℂ}, and O is the Richardson nilpotent orbit attached to H_{ℂ}. For the general case we simply refer to [Vogan90].

It is now more or less clear that our constructions for hyperbolic orbits in section 7 fit into the framework of Problem 3.15. Here is a specific statement.

Corollary 9.14. Suppose G is a real reductive group of inner type G_{ℂ} (Definition 3.13), and that ξ ∈ 𝔤_{0}^{∗} is a hyperbolic element (Definition 5.10). Define a parabolic subgroup P = LN as in Proposition 7.2, and let P_{ℂ} ⊂ G_{ℂ} be its complexification.

a) The linear functional ξ restricts to a character λ(ξ) of the Lie algebra 𝔭. Write D_{ξ} = D_{λ(ξ)} for the twisted differential operator algebra on G_{ℂ}/Q_{ℂ} attached to λ(ξ) in (9.10).

b) Suppose τ is an integral orbit datum at ξ (Definition 4.7 and Proposition 7.4), and (π(τ), H) is the corresponding unitary representation of G (cf. (4.16)). Then the action of U(𝔤) on the smooth vectors H^{∞} extends naturally to an action of D_{ξ}.

c) In the notation of (9.12) and Proposition 9.13, Σ_{λ(ξ)} = Õ_{λ}(ξ) ≃ O_{λ} = G_{ℂ} · ξ.

The assertion in (b) follows from the last part of Theorem 9.2. That in (c) can be deduced from Proposition 7.2(d) applied to G_{ℂ}. We leave the details to the reader, along with such tasks as the construction of a Hermitian transpose on D_{ξ}.

In the setting of (9.1), the correspondence of Conjecture 3.9 should carry the orbit cover Õ_{λ} (Proposition 9.13) to the Dixmier algebra D_{λ} (cf. (9.10)). For G_{ℂ} = GL(n, ℂ) every equivariant orbit cover is of the form Õ_{λ} for some parabolic H_{ℂ} and character λ. (Actually the covers are necessarily trivial in this case.) The same orbit may arise in several different ways, but Borho has shown in [Borho] that then the various D_{λ} are all isomorphic. (To be precise, one needs in addition to [Borho] the result from [Borho-Brylinski82] that the maps ϕ_{λ} are always surjective for GL(n, ℂ).) So there is a well-defined Dixmier correspondence for GL(n, ℂ). The injectivity of the correspondence is established in [Borho-Jantzen], proving Conjecture 3.9 for GL(n, ℂ).

For most other cases, there are equivariant orbit covers not of the form Õ_{λ}. (Such covers exist exactly when G_{ℂ} has a simple factor not of type A, or G_{ℂ} has disconnected center.) In those cases the twisted differential operator algebras D_{λ} do not suffice to prove Conjecture 3.9. A discussion of what else is needed may be found in [Vogan90].

10. Elliptic orbits, complex polarizations, and admissibility.

In section 7 we gave a construction of unitary representations of a reductive group G attached to hyperbolic coadjoint orbits. This construction is very nice as far as it goes, but it does not apply to other coadjoint orbits. In this section we will introduce an analogous construction for elliptic orbits. We begin with some structure theory along the lines of Proposition 7.2.

Proposition 10.1. In the setting of (7.1), suppose that ad(X) is diagonalizable on the complexified Lie algebra 𝔤, with purely imaginary eigenvalues. (This happens in particular if X is elliptic (Lemma 5.13).) Write 𝔤^{t} for the t-eigenspace of ad(iX), so that

𝔤 = ∑_{t∈ℝ} 𝔤^{t},  𝔤_{X} = 𝔤^{0}.  (10.1)(i)

Define

𝔭_{X} = ∑_{t≥0} 𝔤^{t},  𝔫_{X} = ∑_{t>0} 𝔤^{t}.  (10.1)(ii)

a) The decomposition (10.1)(i) makes 𝔤 an ℝ-graded Lie algebra: [𝔤^{s}, 𝔤^{t}] ⊂ 𝔤^{s+t}.

b) The subspace 𝔤^{t} is orthogonal to 𝔤^{s} with respect to ω_{X} unless s = −t.

c) Complex conjugation on 𝔤 with respect to the real form 𝔤_{0} carries 𝔤^{t} onto 𝔤^{−t}. In particular,

\overline{𝔭}_{X} = ∑_{t≤0} 𝔤^{t}, \overline{𝔫}_{X} = ∑_{t<0} 𝔤^{t}

so that

𝔭_{X} + \overline{𝔭}_{X} = 𝔤, 𝔭_{X} ∩ \overline{𝔭}_{X} = 𝔤^{0}.

d) The adjoint action of G_{X} preserves each subspace 𝔤^{t}, and so preserves the subalgebra 𝔭_{X}.

e) Suppose (γ, F) is a finite-dimensional representation of G_{X}, and that dγ(X) (the differentiated representation applied to the Lie algebra element X) is a scalar operator. Then dγ extends uniquely to a representation ϕ of 𝔭_{X}. This extension satisfies ϕ|_{𝔫_{X}} = 0, and

ϕ(Ad(g)Z) = γ(g)ϕ(Z)γ(g^{−1})  (g ∈ G_{X}, Z ∈ 𝔭_{X}).  (10.1)(iii)

Proof. Parts (a) and (b) are proved just as in Proposition 7.2. (In fact iX is a hyperbolic element of 𝔤, so they may be regarded as special cases of Proposition 7.2.) For (c), use the fact that \overline{iX} = −iX and the fact that complex conjugation is a Lie algebra automorphism. Part (d) is obvious. For (e), the property of vanishing on 𝔫_{X} is easily seen to define an extension ϕ satisfying (10.1)(iii). For the uniqueness, suppose that ϕ is any representation of 𝔭_{X}. Then if Z ∈ 𝔤^{t}, ϕ(Z) must carry the λ eigenspace of ϕ(iX) into the λ + t eigenspace. In our case ϕ(iX) is assumed to act by scalars, so 𝔫_{X} must act by zero. Q.E.D.

The idea is that the complex Lie algebra 𝔭_{X} is something like a polarization at ξ_{X} (Definition 4.12). In fact 𝔭_{X}/𝔤_{X} is a complex Lagrangian subspace of the complexified tangent space T_{ξ_{X}} (G · ξ_{X})_{ℂ}. There is no subgroup of G with Lie algebra 𝔭_{X}; but we observed in (4.14)(c) that it was possible to write down an interesting representation space for G using only G_{X} and the Lie algebra of the polarization. In the setting of Proposition 10.1(e), the space in question is

\begin{aligned}Γ(G/G_{X}, 𝔭_{X}; F) = \{f ∈ C^{∞}(G, F) | & f(gh) = γ(h)^{−1}f(g), ρ(Z)f = −ϕ(Z)f \\ & (g ∈ G, h ∈ G_{X}, Z ∈ 𝔭_{X})\}. \end{aligned}  (10.2)

That is, we are beginning with sections of the smooth vector bundle on G/G_{X} defined by (γ, F) and imposing a family of first-order differential equations corresponding to elements of 𝔫_{X}. Proposition 10.1(c) suggests that these differential equations resemble the Cauchy-Riemann equations. Here is a precise statement.

Proposition 10.3. In the setting of (7.1), suppose that ad(X) is diagonalizable with purely imaginary eigenvalues. Then there is a distinguished G-invariant complex structure on the coadjoint orbit G · ξ_{X} ≃ G/G_{X}. It may be characterized by the requirement that 𝔭_{X}/𝔤_{X} (Proposition 10.1(ii)) is the antiholomorphic tangent space at the identity coset (corresponding to the point ξ_{X}).

In the setting of Proposition 10.1(e), the Lie algebra representation ϕ defines a G-invariant complex structure on the vector bundle F = G ×_{G_{X}} F over G · ξ_{X}. The space (10.2) may be identified with the space of holomorphic sections of F.

A somewhat more detailed discussion of this result may be found in [Vogan87b], Propositions 1.19 and 1.21.

Proposition 10.3 suggests that any generalization of Corollary 9.14 to elliptic coadjoint orbits will need to use complex analysis. In order to begin, we first consider exactly which (holomorphic) vector bundles on G/G_{X} are relevant. The most obvious possibility is to use the bundle defined by an integral orbit datum (γ, H_{γ}) at ξ_{X} (Definition 4.7). That is, we might require

dγ(Y) = iξ_{X}(Y) = i⟨X, Y ⟩ (Y ∈ 𝔤_{X})

But this is not precisely analogous to what we did in the hyperbolic case (cf. (4.16)). There the integral orbit datum was first twisted by the character of the isotropy group on half-densities. Explicitly (for X hyperbolic and g ∈ G_{X}) Δ_{X}(g) is the square root of the absolute value of the determinant of the adjoint action of g on (𝔤_{0}/𝔭_{X,0})^{∗}. The differential of Δ_{X} is a character δ_{X} of 𝔤_{X,0}: δ_{X}(Y) is half the trace of the adjoint action of Y on (𝔤_{0}/𝔭_{X,0})^{∗}.

For X elliptic the quotient 𝔤/𝔭_{X} is only a complex vector space, lacking a G_{X}-invariant real form. There is accordingly no natural analogue of half-densities, or of the character Δ_{X}. But δ_{X} still makes sense: we can compute traces in the complexification of a vector space, so it is consistent with the hyperbolic case to define a character of 𝔤_{X} by

δ_{X}(Y) = \frac{1}{2}tr(ad(Y) on (𝔤/𝔭_{X})^{∗})

= \frac{1}{2}tr(ad(Y) on 𝔫_{X})

(10.4)(a)

This need not be the differential of a character of G_{X}. Nevertheless, a better analogue of (4.16) is to consider bundles on G/G_{X} corresponding to irreducible unitary representations (γ, F_{γ}) of G_{X} with the property that

dγ(Y) = iξ_{X}(Y) + δ_{X}(Y).  (10.4)(b)

Such a representation γ is called an admissible orbit datum at ξ. The orbit G · ξ is called admissible if there is an admissible orbit datum at ξ. (Compare the definition of integral orbit datum in Definition 4.7. If there is a one-dimensional character Δ_{X} of G_{X} with differential δ_{X}, then tensoring with Δ_{X} defines a bijection between integral orbit data and admissible orbit data. But in general the two notions are simply different.)

The definition of admissible given here makes sense only for elliptic orbits of reductive groups. There is a more sophisticated notion, due to Duflo, that makes sense for arbitrary coadjoint orbits of arbitrary Lie groups. We refer to [Vogan87b], Definition 10.16 for Duflo’s definition; one can also find there a discussion of how it reduces to (10.4) in the elliptic case.

According to Proposition 10.3, an admissible orbit datum (γ, F_{γ}) defines a holomorphic vector bundle F_{γ} over the coadjoint orbit G · ξ_{X}. The unitary structure on the representation γ provides a G-invariant Hermitian structure on the vector bundle, and the symplectic structure on the orbit provides a canonical G-invariant measure. Accordingly there is a natural analogue of the unitary representation defined in (4.16) for a real polarization: the Hilbert space is the space L^{2}(G · ξ_{X}, 𝔭_{X}; F_{γ}) of square-integrable holomorphic sections of the vector bundle (compare (10.2) and (4.14)(c)).

Unfortunately this space is almost always zero: in fact F_{γ} usually has no non-trivial holomorphic sections at all. This is most easily seen when G is compact. Taking for example G = SU(2) and ξ_{X} any non-zero admissible element of 𝔤_{0}^{∗}, we find that G·ξ_{X} is isomorphic to ℂℙ^{1}, and that the line bundle F_{γ} is one usually denoted O(−n − 1), with n a positive integer. (Essentially −n arises from the term iξ_{X} in (10.4)(b), and −1 from δ_{X}.) The line bundle O(−n − 1) has only the zero section. What is interesting is its first cohomology:

H^{1}(G · ξ_{X}, F_{γ}) ≃ ℂ^{n},

the unique irreducible representation of G of dimension n. At first glance this suggests that we have chosen the wrong complex structure on G · ξ_{X}. Indeed everything in this section works with only trivial changes if we reverse the rôles of 𝔭_{X} and \overline{𝔭}_{X}. With the new complex structure the line bundle F_{γ} becomes O(n − 1), still with n a positive integer. This bundle does have holomorphic sections:

H^{0}(G · ξ_{X}, F_{γ}) ≃ ℂ^{n},

the n-dimensional irreducible representation.

There is a price to be paid for such a change, however. If instead we consider the group G = SL(2, ℝ) and take ξ_{X} to be a non-zero admissible elliptic element, then G · ξ_{X} is the upper half plane with the usual action of G by linear fractional transformations. The line bundle F_{γ} is holomorphically trivial, and therefore has lots of holomorphic sections (and vanishing higher cohomology). When the complex structure is chosen as in Proposition 10.3, there are even square-integrable holomorphic sections; so we get an interesting unitary representation of G (a holomorphic discrete series representation). But if the complex structure is defined instead using the opposite parabolic subalgebra, the only square-integrable section is zero.

The conclusion that we draw from these two examples is that one should use the complex structure specified in Proposition 10.3, and look for a unitary representation in some higher cohomology of G · ξ_{X} with coefficients in F_{γ}. The question of exactly which cohomology to consider is illuminated by the following vanishing theorem of Schmid and Wolf. Recall from Theorem 6.5 and (7.1) that any elliptic orbit in 𝔤_{0}^{∗} has a representative ξ_{X} with X ∈ 𝔨_{0}.

Theorem 10.5 ([Schmid-Wolf]). Suppose G is a real reductive group with maximal compact subgroup K, and X ∈ 𝔨_{0}. Put on D = G · ξ_{X} the complex structure defined by Proposition 10.3. Then Z = K · ξ_{X} is a compact complex subvariety of D; write s for its complex dimension. The variety D is (s + 1)-complete in the sense of Andreotti and Grauert. In particular, this means that Z is a compact complex subvariety of maximal dimension in D; and that if F is any coherent analytic sheaf on D, then H^{q}(D, F) = 0 for q > s.

In the first example above, Z = ℂℙ^{1} and s = 1. In the second, Z is a point and s = 0.

We can now say what representation ought to be attached to an admissible elliptic coadjoint orbit G · ξ. By Theorem 6.5, there is no loss of generality in assuming that θξ = ξ. We begin with an admissible orbit datum γ as in (10.4), and form the corresponding holomorphic vector bundle F_{γ} over the complex manifold G · ξ (Proposition 10.3). Theorem 10.5 says that K · ξ is a compact complex submanifold of dimension s. Form the cohomology group H^{s}(G · ξ, F_{γ}). There are at least two important ways to think of this space. (The isomorphism between them is Dolbeault’s theorem.) One is as a Dolbeault cohomology group, the quotient of closed (0, s) forms on G · ξ with values in F_{γ} by exact forms. This shows first of all that the cohomology group carries a natural representation of G (by translation of (0, s) forms). At the same time it emphasizes the central difficulty in putting a nice topology on the space: it is not obvious that the space of exact forms is closed. (This fact was proved in general by H. Wong in his 1992 Harvard thesis [Wong].) It follows that H^{s}(G · ξ, F_{γ}) has a natural complete locally convex Hausdorff topology.

A second way to think of H^{s}(G · ξ, F_{γ}) is as a Čech cohomology group with coefficients in the sheaf of germs of holomorphic sections of F_{γ}. From this point of view an element is represented by a family of holomorphic sections of F_{γ}, each defined over some small open set (say an intersection of s + 1 elements from a covering of G · ξ by Stein open sets). The advantage of this point of view is that every holomorphic differential operator on sections of F_{γ} clearly acts on the cohomology.

Suppose for example that G is of inner type G_{ℂ} (Definition 3.13). Then 𝔭_{ξ} (Proposition 10.1) is the Lie algebra of a parabolic subgroup P_{ξ,ℂ} of G_{ℂ}, and it is not difficult to show that

G_{ξ} = {g ∈ G | Ad(g) ∈ P_{ξ,ℂ}}.  (10.6)(a)

Consequently G · ξ ≃ G/G_{ξ} is an open submanifold of the flag variety G_{ℂ}/P_{ξ,ℂ}. Comparing (10.4) with the definitions of section 9 (particularly (9.5)(a)), we find that the twisted differential operator algebra D_{ξ} (attached to the character ξ of 𝔭_{ξ} by (9.10)) may be regarded as an algebra of holomorphic differential operators on F_{γ}. Therefore

D_{ξ} acts naturally on H^{s}(G · ξ, F_{γ}).  (10.6)(b)

The Hermitian structure on F_{γ} gives rise to a natural Hermitian transpose on differential operators on F_{γ}. This transpose preserves D_{ξ}, defining a Hermitian transpose of Dixmier algebras in the sense of Definition 3.13.

We are now getting close to having the structure required by Problem 3.15. The Dixmier algebra D_{ξ} and the group K (or even the larger group G) both act on H^{s}(G · ξ, F_{γ}). This space fails to be a (D_{ξ}, K)-module only because the action of K is not locally finite. We have already met such a problem in connection with the construction of D_{ξ}, and we adopt the same solution here: we pass to the subspace of K-finite vectors. Define

\begin{aligned}V_{ξ,γ} & = H^{s}(G · ξ, F_{γ})^{K} \\ & = \{v ∈ H^{s}(G · ξ, F_{γ}) | dim(span(K · v)) < ∞\}. \end{aligned}  (10.6)(c)

It is not difficult to show that V_{ξ,γ} is preserved by D_{ξ} and K, and that it is a (D_{ξ}, K)-module in the sense of Definition 3.13.

To complete the requirements of Problem 3.15, we need a unitary structure on V_{ξ,γ}. At this point the analytic ideas that have brought us so far seem to fail: no general construction of such a unitary structure is known. A great deal is known about special cases; two entry points to the literature are [Zierau] and [Rawnsley-Schmid-Wolf]. Briefly, one seeks a unitary inner product defined by integrating certain distinguished Dolbeault cohomology classes. In its simplest form this program cannot succeed for arbitrary admissible elliptic orbits. To see why, consider elliptic orbits for the group Sp(2n, ℝ) with G_{ξ} ≃ U(n). Such orbits are parametrized by the non-zero real numbers; the orbit G · ξ_{t} with parameter t is admissible if and only if 2t + n + 1 is an even integer. (The corresponding character γ of U(n) is det^{(2t+n+1)/2}.) The number s is equal to zero, so H^{s}(G · ξ_{t}, F_{γ}) is the space of holomorphic sections of the line bundle on Sp(2n, ℝ)/U(n) corresponding to the character γ. The unitary structure proposed in [Rawnsley-Schmid-Wolf] is just integration of holomorphic sections over the orbit. The convergence of these integrals was studied in [Harish-Chandra56]. For t > (n − 1)/2 all the K-finite sections are square-integrable; but for 0 < t ≤ (n − 1)/2 there are no non-zero square-integrable holomorphic sections. As soon as n is at least two, therefore, we find cases where the inner product does not arise by integration. (It seems likely that the integrals will converge in general for “most” admissible elliptic orbits, as they do in this example. No such convergence has yet been proved, however.)

Despite these difficulties, there is a strong positive result.

Theorem 10.7 ([Vogan84]). Suppose G is a real reductive group of inner type G_{ℂ}, and that ξ ∈ 𝔤_{0}^{∗} satisfies θξ = ξ (so that G · ξ is a typical elliptic coadjoint orbit). Assume that G · ξ is admissible, with (γ, F_{γ}) an admissible orbit datum (cf. (10.4)). Define a (D_{ξ}, K) module V_{ξ,γ} as in (10.6). Then V_{ξ,γ} carries a natural unitary structure (Definition 3.13). In particular, it is the Harish-Chandra module of a unitary representation π(ξ, γ) of G.

This result provides unitary representations attached to elliptic coadjoint orbits. The representation V_{ξ,γ} turns out to be irreducible as a (D_{ξ}, K)-module but not necessarily as a (𝔤, K)-module; so the unitary representation π(ξ, γ) of G may be reducible.

We will now outline part of the proof of this theorem. The main point is to give (following Zuckerman) an algebraic construction analogous to the geometric one in (10.6). Roughly speaking, we replace holomorphic functions by formal power series. In the algebraic setting an invariant Hermitian form can be constructed without much difficulty, and (with a little more difficulty) the positivity of the form can be proved. This is all that is required to attach a unitary representation to the orbit G · ξ (by Theorem 3.14). To complete the proof of Theorem 10.7 as stated, one must also identify the algebraic construction with the geometric one. Roughly speaking, this amounts to proving the convergence of some formal power series solutions of differential equations. The necessary ideas go back to Schmid’s 1967 thesis [Schmid67]; the result was proved completely in [Wong].

We begin with the finite-dimensional module F_{γ} for G_{ξ}. By Corollary 5.17, G_{ξ} is a real reductive group, with Cartan involution the restriction of θ and maximal compact subgroup K_{ξ} = G_{ξ} ∩ K. In particular, K_{ξ} acts on F_{γ}. At the same time, Proposition 10.1(e) provides a representation of the complex Lie algebra 𝔭_{ξ} on F_{γ}. The same result ensures that the representations of 𝔭_{ξ} and K_{ξ} enjoy a compatiblity analogous to that required for (𝔤, K) modules in Definition 2.19; we call F_{γ} a (𝔭_{ξ}, K_{ξ})-module accordingly. Recall now that the differential of γ is iξ + δ (cf. (10.4)). In analogy with the definition of M_{λ} in (9.5), we therefore set

W_{γ} = Hom_{U(𝔭_{ξ})}(U(𝔤), F_{γ}).  (10.8)(a)

The Lie algebra 𝔤 acts on W_{γ} by right multiplication on U(𝔤). The group K_{ξ} acts by combining the adjoint action on U(𝔤) with the action on F_{γ}:

(h · w)(u) = h · (w(Ad(h^{−1})u))  (h ∈ K_{ξ}, w ∈ W_{γ}, u ∈ U(𝔤)).  (10.8)(b)

This action need not be locally finite, so we define

W_{γ}^{K_{ξ}} = {w ∈ W_{γ} | dim(span(K_{ξ} · w)) < ∞}.  (10.8)(c)

Up to this point there is no need to restrict to the compact subgroup K_{ξ}; we could just as well have kept track of an action of G_{ξ}. (The K_{ξ}-finite vectors turn out automatically to be G_{ξ} finite.) But it is the K_{ξ} action we will soon need.

Lemma 10.9. The space W_{γ} may be identified with the space of formal power series for sections of the vector bundle F_{γ} at the base point ξ ∈ G · ξ. The actions of 𝔤 and K_{ξ} satisfy compatibility conditions analogous to those in Definition 2.19, making W_{γ}^{K_{ξ}} a (𝔤, K_{ξ})-module. If G is of inner type G_{ℂ}, then the Dixmier algebra D_{ξ} acts naturally on W_{γ}, extending the action of U(𝔤) and preserving the subspace W_{γ}^{K_{ξ}}; in this way W_{γ}^{K_{ξ}} becomes a (D_{ξ}, K_{ξ}) module (Definition 3.13).

This elementary result is a holomorphic version of Lemma 9.3. In the setting of Problem 3.15, one can take W_{γ}^{K_{ξ}} as one of the D_{ξ} modules W_{i}(G · ξ).

The next step is to construct from W_{γ}^{K_{ξ}} a (D_{ξ}, K)-module. We will use a functor Γ = Γ_{𝔨,K_{ξ}}^{𝔨,K} introduced

by Zuckerman for passing to the subspace ΓW_{γ}^{K_{ξ}} of K-finite vectors in W_{γ}^{K_{ξ}}. One way to understand this approach is in terms of Theorem 10.5. The “(s + 1)-completeness” property says roughly that G · ξ looks like a Stein manifold away from the compact subvariety K · ξ. Holomorphic bundles on Stein manifolds have many global sections. This means that (morally) the obstruction to globalizing a formal power series section of F_{γ} is mostly in the direction of K · ξ. Now it is not difficult to see that a K-finite formal power series section of F_{γ} must represent a holomorphic section over K · ξ. In light of the (s + 1)-completeness, this suggests that elements of ΓW_{γ}^{K_{ξ}} should represent global holomorphic sections of F_{γ}. That is, we might expect

ΓW_{γ}^{K_{ξ}} ≃ H^{0}(G · ξ, F_{γ})^{K}.

This turns out to be true, and not too difficult to prove. The problem, as we already observed after (10.4), is that both sides are usually zero; we need analogues not of holomorphic sections but of higher cohomology.

Zuckerman’s great observation was that the functor Γ is only left exact, and that it has right derived functors Γ^{i}. Although it is harder to justify precisely, one might still hope formally that

Γ^{i}W_{γ}^{K_{ξ}} ≃ H^{i}(G · ξ, F_{γ})^{K}.

This statement was proved in [Wong].

Here is the definition of Γ. (Compare Definition 9.9, which is similar but simpler.)

Definition 10.10 ([Vogan81], Definition 6.2.4). Suppose K is a compact Lie group with complexified Lie algebra 𝔨, and H is a closed subgroup of K. Suppose W is a (𝔨, H)-module (defined as in Definition 2.19). We want to define a (𝔨, K)-module Γ_{(𝔨,H)}^{(𝔨,K)}W = ΓW. To begin, let K̃_{0} be the universal cover of the identity component of K, so that we have

1 → Z → K̃_{0} → K_{0} → 1.

Here Z is a discrete central subgroup of K̃_{0}. Define

Γ̃W = {w ∈ W | dim U(𝔨)w < ∞}.

As in Definition 9.9, Γ̃W carries a locally finite representation of K̃_{0}; and we can define

Γ_{0}W = {w ∈ ΓW | z · w = w, all z ∈ Z}.

This is a subspace of W carrying a locally finite representation π_{0} of K_{0}; it is also preserved by the representation τ of H on W. We may therefore define

Γ_{1}W = {w ∈ Γ_{0}W | π_{0}(h)w = τ(h)w, all h ∈ H ∩ K_{0}}.

This subspace is invariant under the representations π_{0} and τ. Now define K_{1} to be the subgroup of K generated by K_{0} and H. There is a unique representation π_{1} of K_{1} on Γ_{1}W that extends both π_{0} and τ. Finally, set

ΓW = Ind_{K_{1}}^{K}Γ_{1}W,

a locally finite representation of K.

The reader may try to understand geometrically each of the steps in the construction of ΓW when (for example) W is the space of H-finite formal power series sections of a bundle E on K/H. In this case ΓW may be identified with the space of K-finite global sections of E. One interesting step is the last one, of induction from K_{1} to K. The point there is that the index m of K_{1} in K is just the number of connected components of the homogeneous space K/H. Ordinarily one could not hope to understand sections on different connected components using Taylor series; but the group action allows us to do just that. The space Γ_{1}W may be identified with K_{1}-finite sections of E over the identity component K_{1}/H ≃ K_{0}/(K_{0} ∩ H). Induction more or less replaces this space by a sum of m copies of it; each copy corresponds to sections supported on one of the components of K/H.

Proposition 10.11 (Zuckerman; see [Vogan81], Chapter 6). The functor Γ of Definition 10.10 is a left exact functor from the category of (𝔨, H)-modules to the category of (𝔨, K)-modules. It has right derived functors Γ^{i}, which are non-zero exactly for 0 ≤ i ≤ dim_{ℝ}(K/H).

Suppose now that K is the maximal compact subgroup of a reductive group G. If W is a (𝔤, H)-module (defined in analogy with Definition 2.19), then Γ^{i}W becomes naturally a (𝔤, K)-module. Similarly, if A is a Dixmier algebra and W is an (A, H)-module, then Γ^{i}W is naturally an (A, K)-module.

The last assertion (about Dixmier algebras) is not part of Zuckerman’s original ideas; it is more or less a folk theorem from the early 1980’s. It may be proved using the method of [Wallach], section 6.3.

Theorem 10.12 (see [Vogan84]). In the setting of (10.8), write Γ = Γ_{𝔨,K_{ξ}}^{𝔨,K} for the functor of Definition 10.10. Write s = dim_{ℂ}(K · ξ) = (1/2) dim_{ℝ}(K/K_{ξ}).

a) Γ^{i}W_{γ}^{K_{ξ}} = 0 for i ≠ s.

b) Γ^{s}W_{γ}^{K_{ξ}} carries a natural non-degenerate (𝔤, K)-invariant Hermitian form.

c) The form in (b) is positive definite.

d) Suppose G is of inner type G_{ℂ}. Then there is a natural Hermitian transpose on D_{ξ} making Γ^{s}W_{γ}^{K_{ξ}} into a unitary (D_{ξ}, K)-module.

e) Γ^{s}W_{γ}^{K_{ξ}} is irreducible as a (D_{ξ}, K)-module.

In lieu of a proof, here are some historical remarks. The vanishing theorem in (a) is due to Zuckerman for most γ; a proof may be found in [Vogan81]. The proof for all γ as in (10.4) appears first in [Vogan84].

The form in (b) was constructed by Zuckerman, but his proof of its invariance under 𝔤 was incomplete. Repairs were provided in [Enright-Wallach].

Part (c) is (a special case of) the main result of [Vogan84]. The proof there is a reduction to the special case when G_{ξ} is a compact Cartan subgroup of G. In that case Schmid in [Schmid75] had essentially identified Γ^{s}W_{γ}^{K_{ξ}} with one of the discrete series representations constructed by Harish-Chandra (using deep analytic techniques). Harish-Chandra’s discrete series representations are irreducible and unitary; so any non-zero invariant Hermitian form on them must be definite. A purely algebraic proof of (c) was later found by Wallach (see [Wallach]).

Part (d) is more or less a folk theorem; it can be proved in the same way as (b). The irreducibility of Γ^{s}W_{γ}^{K_{ξ}} even as a (𝔤, K) module was proved by Zuckerman for most γ (see [Vogan81]). The general result in (e) is perhaps an unpublished result of Bernstein; it follows from the generic case using the translation technique in [Vogan90], Corollary 7.14.

Theorem 10.7 follows from Theorem 10.12 and the result of Wong stated before Definition 10.10.

References.

[Abraham-Marsden] R. Abraham and J. Marsden, Foundations of Mechanics. Benjamin/Cummings, Reading, Massachusetts, 1978.

[Arnold] V. Arnold, Mathematical Methods of Classical Mechanics. Springer-Verlag, New York-Heidel-berg-Berlin, 1978.

[Beilinson-Bernstein] A. Beilinson and J. Bernstein, “Localisation de 𝔤-modules,” C. R. Acad. Sci. Paris 292 (1981), 15–18.

[Borel-Wallach] A. Borel and N. Wallach, Continuous Cohomology, Discrete Subgroups, and Representations of Reductive Groups. Princeton University Press, Princeton, New Jersey, 1980.

[Borho] W. Borho, “Definition einer Dixmier–Abbildung für 𝔰l(n, ℂ),” Invent. Math. 40 (1977), 143– 169.

[Borho-Brylinski82] W. Borho and J.-L. Brylinski, “Differential operators on homogeneous spaces I,” Invent. Math. 69 (1982), 437–476.

[Borho-Brylinski85] W. Borho and J.-L. Brylinski, “Differential operators on homogeneous spaces III,” Invent. Math. 80 (1985), 1–68.

[Collingwood-McGovern] D. Collingwood and W. McGovern, Nilpotent orbits in semisimple Lie algebras. Van Nostrand Reinhold, New York, 1993.

[Conze-Berline–Duflo] N. Conze-Berline and M. Duflo, “Sur les représentations induites des groupes semi-simples complexes,” Compositio math. 34 (1977), 307–336.

[Dixmier] J. Dixmier, “Ideaux primitifs dans les algèbres enveloppantes,” J. Algebra 48 (1978) 96–112. [Enright-Wallach] T. Enright and N. Wallach, “Notes on homological algebra and representations of Lie

algebras,” Duke Math. J. 47 (1980), 1–15.

[Guichardet] A. Guichardet, “Théorie de Mackey et méthode des orbites selon M. Duflo,” Expo. Math. 3 (1985), 303–346.

[Harish-Chandra53] Harish-Chandra, “Representations of semi-simple Lie groups I,” Trans. Amer. Math. Soc. 75 (1953), 185–243.

[Harish-Chandra56] Harish-Chandra, Representations of semisimple Lie groups, V,” Amer. J. Math. 78 (1956), 1–41.

[Humphreys] J. E. Humphreys, Introduction to Lie Algebras and Representation Theory. Springer-Verlag, Berlin-Heidelberg-New York, 1972.

[Joseph] A. Joseph, “Kostant’s problem and Goldie rank,” 249–266, in Non-commutative Harmonic Analysis and Lie Groups, J. Carmona and M. Vergne, eds., Lecture Notes in Mathematics 880. Springer-Verlag, Berlin-Heidelberg-New York, 1981.

[Knapp-Vogan], A. Knapp and D. Vogan, Cohomological induction and unitary representations. To be published by Princeton University Press, Princeton, New Jersey, 1995.

[Kostant] B. Kostant, “Quantization and unitary representations,” in Lectures in Modern Analysis and Applications, (C. Taam, ed.), Lecture Notes in Mathematics 170. Springer-Verlag, Berlin-Heidelberg-New York, 1970.

[Lepowsky] J. Lepowsky, “Algebraic results on representations of semisimple Lie groups,” Trans. Amer. Math. Soc. 176 (1973), 1–44.

[Rawnsley-Schmid-Wolf] J. Rawnsley, W. Schmid, and J. Wolf, “Singular unitary representations and indefinite harmonic theory,” J. Funct. Anal. 51 (1983), 1–114.

[Schmid67] W. Schmid, “Homogeneous complex manifolds and representations of semisimple Lie groups,” 223–286 in Representation theory and harmonic analysis on semisimple Lie groups, P. Sally and D. Vogan, eds. Mathematical Surveys and Monographs 31. American Mathematical Society, Providence, Rhode Island, 1989.

[Schmid75] W. Schmid, “Some properties of square-integrable representations of semisimple Lie groups,” Ann. of Math. 102 (1975), 535–564.

[Schmid-Wolf] W. Schmid and J. Wolf, “A vanishing theorem for open orbits on complex flag manifolds,” Proc. Amer. Math. Soc. 92 (1984), 461–464.

[Vogan81] D. Vogan, Representations of Real Reductive Lie Groups. Birkhäuser, Boston-Basel-Stuttgart, 1981.

[Vogan84] D. Vogan, “Unitarizability of certain series of representations,” Ann. of Math. 120 (1984), 141–187.

[Vogan86] D. Vogan, “The orbit method and primitive ideals for semisimple Lie algebras,” in Lie Algebras and Related Topics, CMS Conference Proceedings, volume 5, D. Britten, F. Lemire, and R. Moody, eds. American Mathematical Society for CMS, Providence, Rhode Island, 1986.

[Vogan87] D. Vogan, “Representations of reductive Lie groups,” 245–266 in Proceedings of the International Congress of Mathematicians 1986, volume I. American Mathematical Society, Providence, Rhode Island, 1987.

[Vogan87b] D. Vogan, Unitary Representations of Reductive Lie Groups. Annals of Mathematics Studies, Princeton University Press, Princeton, New Jersey, 1987.

[Vogan88] D. Vogan, “Noncommutative algebras and unitary representations,” 35–60 in The Mathematical Heritage of Hermann Weyl, Proceedings of Symposia in Pure Mathematics 48. American Mathematical Society, Providence, Rhode Island, 1988.

[Vogan90] D. Vogan, “Dixmier algebras, sheets, and representation theory,” 333-395 in Operator Algebras, Unitary Representations, Enveloping Algebras, and Invariant Theory, A. Connes, M. Duflo, A. Joseph, and R. Rentschler, eds. Birkhäuser Boston, 1990.

[Vogan94] D. Vogan, “The unitary dual of G_{2},” Invent. math. 116 (1994), 677-791.

[Wallach] N. Wallach, Real Reductive Groups I. Academic Press, San Diego, 1988.

[Wong] H. Wong, “Dolbeault cohomologies and Zuckerman modules associated with finite rank representations,” Ph.D. dissertation, Harvard University, 1992; to appear in J. Funct. Anal.

[Zierau] R. Zierau, “Unitarity of certain Dolbeault cohomology representations,” 239–259 in The Penrose transform and analytic cohomology in representation theory, M. Eastwood, J. Wolf, and R. Zierau, eds. Contemporary Mathematics 154. American Mathematical Society, Providence, Rhode Island, 1993.