ON INFINITE DIMENSIONAL ALGEBRAIC TRANSFORMATION GROUPS

VLADIMIR L. POPOV^{∗}

Abstract. We explore orbits, rational invariant functions, and quotients of the natural actions of connected, not necessarily finite dimensional subgroups of the automorphism groups of irreducible algebraic varieties. The applications of the results obtained are given.

1. Introduction. The following well-known result (see, e.g., [Bor91, Prop. I.2.2]) is one of the indispensable tools in the theory of algebraic groups:

Theorem. Let φ_{i}: T_{i} → G (i ∈ I) be a collection of morphisms from irreducible algebraic varieties T_{i} into an algebraic group G, and assume that the identity element of G lies in X_{i} := φ_{i}(T_{i}) for each i ∈ I. Then the subgroup A of G generated, as an abstract group, by the set M := ⋃_{i∈I} X_{i} coincides with the intersection of all closed subgroups of G containing M. Moreover, A is connected and there is a finite sequence α_{1}, …, α_{n} in I such that A = X_{α_{1}}^{e_{1}} ⋯ X_{α_{n}}^{e_{n}}, where e_{i} =±1 for each i.

Here we show that the analogous construction, applied in place of G to Aut(X), where X is an irreducible algebraic variety, yields a group, though not in general algebraic, but whose natural action on X surprisingly retains some basic properties of orbits and invariant fields of algebraic group actions. This leads to some applications. The main results are formulated in Section 3.

In what follows, variety means algebraic variety in the sense of Serre over an algebraically closed field k of arbitrary characteristic (so algebraic group means algebraic group over k). The standard notation and conventions of [Bo91] and [PV94] are used freely. Given a rational function f ∈ k(X) and an element σ ∈ Aut(X), we denote by f^{σ} the rational function on X defined by f^{σ}(σ(x)) = f(x) for every point x in the domain of definition of f.

2. Definitions and notation. Let T be an irreducible variety. Any map

φ: T → Aut(X), t ↦ φ_{t},

determines a family {φ_{t}}_{t∈T} in Aut(X) parameterized by T. We put

φ_{T} := φ(T)

∗ Supported by grants ���� �������������, ��������������, and the program Contemporary Problems of Theoretical Mathematics of the Russian Academy of Sciences, Branch of Mathematics.

If I is a nonempty collection of families in Aut(X), then the subgroup of Aut(X) generated, as an abstract group, by the set ⋃ φ_{T} with the union taken over all families {φ_{t}}_{t∈T} in I will be called the group generated by I.

We shall say that a family {φ_{t}}_{t∈T} in Aut(X) is

— injective (see [Ram64]) if φ_{t} ≠ φ_{s} for all t ≠ s;

— unital if id_{X} ∈ φ_{T};

— algebraic (see [Ram64]) if

φ̃: T × X → X, (t, x) ↦ φ_{t}(x)  (1)

is a morphism. Given a family {φ_{t}}_{t∈T} in Aut(X), the family {φ_{t}^{−1}}_{t∈T} in Aut(X) will

be called the inverse of {φ_{t}}_{t∈T}. If {φ_{t}}_{t∈T} , …, {ψ_{s}}_{s∈S} is a finite sequence of families in Aut(X), the family

{φ_{t} ∘ ⋯ ∘ ψ_{s}}_{(t,...,s)∈T ×···×S}  (2)

in X will be called the product of {φ_{t}}_{t∈T} , …, {ψ_{s}}_{s∈S}. The inverses and products of families contained in a subgroup G of Aut(X) are contained in G as well. The inverses and products of algebraic (resp., unital) families are algebraic, see [Ram64] (resp., unital).

Let I be a collection of families in Aut(X). We shall say that a family {φ_{t}}_{t∈T} in Aut(X) is derived from I if {φ_{t}}_{t∈T} is a product of families each of which is either a family from I or the inverse of such a family.

A subgroup G of Aut(X) is called (see [Ram64]) a finite dimensional subgroup if there is an integer n such that dim T ⩽ n for every injective algebraic family {φ_{t}}_{t∈T} in this subgroup; the smallest n satisfying this property is called the dimension of G. If G is not finite dimensional, it is called an infinite dimensional subgroup of Aut(X).

If for every element g ∈ G there exists a unital algebraic family {φ_{t}}_{t∈T} in G such that g ∈ φ_{T}, then G is called (see [Ram64]) a connected subgroup of Aut(X).

If {φ_{t}}_{t∈T} is an algebraic family such that T is a connected algebraic group and φ̃ (given by (1)) is an action of T on X, then φ_{T} is a connected finite dimensional subgroup of Aut(X). By [Ram64, Thm.], every connected finite dimensional subgroup of Aut(X) is obtained in this way. Such subgroups are called connected algebraic subgroups of Aut(X).

Given a nonempty subset S of Aut(X), we put

S(x) := {g(x) | g ∈ S}.

Given a subgroup G of Aut(X) and a G-invariant subset Y of X, we shall say that a family {φ_{t}}_{t∈T} in G is an exhaustive family for the natural action of G on Y if G(y) = φ_{T} (y) for every point y ∈ Y.

3. Main results. In Theorems 1, 2, 3, 4 and Corollaries 1, 2 below we do not assume finite dimensionality of G. If G is finite dimensional, then the statement of Theorems 1 and 2 become trivial and that of Theorems 3, 4 and Corollaries 1, 2 turn into the well-known classical results of the algebraic transformation group theory (see, e.g., [PV94, Sect. 1.4, 2.3]); in particular, Theorem 4 becomes classical Rosenlicht’s theorem [Ros56].

Theorem 1. Let X be an irreducible variety and let G be a subgroup of Aut(X). Then the following properties are equivalent:

(i) G is a connected subgroup of Aut(X); (ii) G is generated by a collection I of unital algebraic families in Aut(X).

Theorem 2. Let X be an irreducible variety and let G be a subgroup of Aut(X) generated by a collection I of unital algebraic families in Aut(X). Let Y be a G-invariant locally closed subvariety of X. Then there is a family derived from I and exhaustive for the natural action of G on Y.

Theorem 3. Let X be an irreducible variety and let G be a connected subgroup of Aut(X). Let Y be an irreducible G-invariant locally closed subvariety of X. Then there exists an integer m_{G,Y} and a dense open subset U of Y such that dim G(y) = m_{G,Y} for every point y ∈ U.

Theorem 4. Let X be an irreducible variety and let G be a connected subgroup of Aut(X). Let Y be an irreducible G-invariant locally closed subvariety of X. Then for some G-invariant dense open subset U of Y there exists a geometric quotient, i.e., there are an irreducible variety W and a morphism ρ: U → W such that

(i) ρ is surjective, open, and the fibers of ρ are the G-orbits in U; (ii) if V is an open subset of U, then

ρ^{∗}: k[ρ(V)] → {f ∈ k[V] | f is constant on the fibers of ρ|_{V} }

is an isomorphism of k-algebras.

Corollary 1. Let X be an irreducible variety and let G be a connected subgroup of Aut(X). Let Y be an irreducible G-invariant locally closed subvariety of X. Then there exists a finite subset of k(Y)^{G} that separates G-orbits of points of a dense open subset of Y.

Corollary 2. Let X be an irreducible variety and let G be a connected subgroup of Aut(X). Let Y be an irreducible G-invariant locally closed subvariety of X. Then the transcendence degree of the field k(Y)^{G} over k is equal to dim X − m_{G,Y} (see Theorem 3). In particular, k(Y)^{G} = k if and only if there is an open G-orbit in Y.

Here are some applications of these results.

Theorem 5. Let X be a nonunirational irreducible variety. Then there exists a nonconstant rational function on X which is G-invariant for every connected affine algebraic subgroup G of Aut(X).

Theorem 5 shows that there is a certain rigidity for the orbits of any connected affine algebraic group G acting regularly on an irreducible nonunirational variety X: every such orbit should lie in a level variety of a certain nonconstant rational function on X not depending on G and its action on X.

We shall say that Aut(X) is generically n-transitive if there exists a dense open subset V of X such that for every point x, y ∈ V^{n} lying off the union of the “diagonals”, there exists an element g ∈ Aut(X) such that g(x) = y for the diagonal action of Aut(X) on X^{n}.

Theorem 6. Let X be a nonunirational irreducible variety.

(i) If the group Aut(X) is generically 2-transitive, then Aut(X) contains no nontrivial connected affine algebraic subgroups.

(ii) If, moreover, there is no dominant morphism Z → X, where Z is an abelian variety, then Aut(X) contains no nontrivial connected algebraic subgroups.

The other applications are discussed in Section 9.

4. Proof of Theorem 1. (i)⇒(ii): For every element g ∈ G, fix a unital algebraic family {φ_{t}}_{t∈T} in G such that g ∈ φ_{T}; connectedness of G implies that such a family exists. Then G is generated, as an abstract group, by ⋃ φ_{T} with the union taken over all the fixed families.

(ii)⇒(i): Since the inverse of any family in G is also a family in G, we may (and shall) assume that if a family belongs to I, then its inverse belongs to I too. Then for every element g ∈ G, there exists a finite sequence of families {φ_{t}}_{t∈T} , …, {ψ_{s}}_{s∈S} from I such that g = φ_{t_{0}} ∘ ⋯ ∘ ψ_{s_{0}} for some t_{0} ∈ T, …, s_{0} ∈ S. Hence g is contained in the product of families {φ_{t}}_{t∈T} , …, {ψ_{s}}_{s∈S} defined by (2). Therefore, G is connected. □

5. Algebraic families. This section contains several general facts utilized in the proofs of Theorems 2, 3, and 4.

Lemma 1. Let X be an irreducible variety, let G be a connected subgroup of Aut(X), and let Y be a G-invariant locally closed subvariety of X.

(i) Every product of unital families in Aut(X) contains each of them. (ii) If a family {φ_{t}}_{t∈T} in G is exhaustive for the natural action of G

on Y, then every family {ψ_{s}}_{s∈S} in G such that φ_{T} ⊆ ψ_{S} is also exhaustive for this action.

(iii) If G is generated by a collection I of unital algebraic families, then

G is the union of all families derived from I. (iv) G|_{Y} := {g|_{Y} | g ∈ G} is a connected subgroup of Aut(Y).

(v) If F is a finite set of algebraic families in G, then G contains a unital algebraic family {φ_{t}}_{t∈T} such that φ_{T} ⊇ ψ_{S} for every {ψ_{s}}_{s∈S} in F.

Proof. (i) and (ii): This is immediate from the definitions.

(iii): The proof is similar to that of implication (ii)⇒(i) of Theorem 1. (iv): If {φ_{t}}_{t∈T} is a unital algebraic family in G containing an element

g ∈ G, then {φ_{t}|_{Y} }_{T} is a unital algebraic family in G|_{Y} containing the element g|_{Y} ∈ G|_{Y}. Whence the claim.

(v): Due to (i), the proof is reduced to the case where F consists of a single family {ψ_{s}}_{s∈S}. In this case, take an element g ∈ ψ_{S}. Since G is connected, it contains a unital algebraic family {μ_{r}}_{r∈R} such that g^{−1} ∈ μ_{R}. The product of {ψ_{s}}_{s∈S} and {μ_{r}}_{r∈R} is then the sought-for family {φ_{t}}_{t∈T}. □

Lemma 2. Let X be an irreducible variety and let G be a connected subgroup of Aut(X). Let Y be a G-invariant locally closed subvariety of X and let Y_{1}, …, Y_{n} be all the irreducible components of Y. Then every Y_{i} is Ginvariant.

Proof. Let {φ_{t}}_{t∈T} be a unital algebraic family in G. For every t ∈ T, since φ_{t} ∈ Aut(X) and Y is φ_{t}-invariant, φ_{t} permutes Y_{1}, …, Y_{n}. Put

T_{ij} := {t ∈ T | φ_{t}(Y_{i}) = Y_{j}}.

For every point x ∈ Y_{i} consider the morphism

φ̃_{x}: T → X, t ↦ φ̃(t, x) = φ_{t}(x)  (3)

(see (1)). Then, for every Y_{j},

T_{ij} = ⋂_{x∈Y_{i}} φ_{x}^{−1}(Y_{j}).  (4)

Since Y_{j} is closed, (4) implies closedness of T_{ij} in T. Unitality of φ_{t} implies T_{ii} ≠ ∅. From T = ⨆_{j=1}^{n} T_{ij} and irreducibility of T we then infer that T = T_{ii} for every i. Thus, Y_{i} is φ_{t}-invariant for every i and t. Theorem 1 then completes the proof. □

Lemma 3. Let X be an irreducible variety and let G be a connected subgroup of Aut(X). If {φ_{t}}_{t∈T} is an algebraic family in G, and x is a point of X, then

(i) G(x) is an irreducible locally closed nonsingular subvariety of X; (ii) φ_{T} (x) is a constructible subset of G(x).

Proof. (i): This is proved in [Ram64, Lemma 2].

(ii): This follows from the definition of algebraic family and Chevalley’s theorem on the image of morphism. □

Corollary 3. Let X be an irreducible variety and let G be a connected subgroup of Aut(X). Then k(X)^{G} is algebraically closed in k(X).

Proof. Let f ∈ k(X) be the root of t^{n} + f_{1}t^{n−1} + ⋯ + f_{n} ∈ k(X)^{G}[t] and let a ∈ X be a point where f and every f_{i} are defined. Then f(G(a)) is a finite subset of k since it lies in the set of roots of t^{n} + f_{1}(a)t^{n−1} + ⋯ + f_{n}(a) ∈ k[t]. Irreducibility of G(a) then implies that this subset is a single element of k, i.e., f|_{G(a)} is a constant. This means that f ∈ k(X)^{G}. □

Lemma 4. Let X be an irreducible variety and let G be a connected subgroup of Aut(X). Let Y be a G-invariant locally closed subvariety of X. Let {φ_{t}}_{t∈T} be a unital algebraic family in G such that φ_{T} (y) is dense in G(y) for every point y ∈ Y. Then the product of the inverse of {φ_{t}}_{t∈T} and {φ_{t}}_{t∈T} is the exhaustive algebraic family {ψ_{s}}_{s∈S} for the natural action of G on Y.

Proof. By the definition of {ψ_{s}}_{s∈S},

ψ_{s} = φ_{t_{1}}^{−1} ∘ φ_{t_{2}} for s = (t_{1}, t_{2}) ∈ S = T × T.  (5)

Take any points y_{1}, y_{2} ∈ Y such that G(y_{1}) = G(y_{2}). The density assumption then yields the equality \overline{φ_{T} (y_{1})} = \overline{φ_{T} (y_{2})}, where bar stands for the closure in X. By Lemma 3, this implies

φ_{T} (y_{1}) ∩ φ_{T} (y_{2}) ≠ ∅;

whence, φ_{t_{1}}(y_{2}) = φ_{t_{2}}(y_{1}) for some t_{1}, t_{2} ∈ T. Therefore, ψ_{s}(y_{1}) = y_{2} for ψ_{s} defined by (5). Hence ψ_{S}(y_{1}) = G(y_{1}) for every point y_{1} ∈ Y. □

6. Proof of Theorem 2. First, we shall show that it suffices to prove the following “generic” version of Theorem 2:

Theorem 2^{∗}. Let X, G, I, and Y are the same as in Theorem 2 and let Y be irreducible. Then there exist a dense open G-invariant subset U in Y and a unital algebraic family {φ_{t}}_{t∈T} in G such that

(i) {φ_{t}}_{t∈T} is derived from I; (ii) φ_{T} (y) is dense in G(y) for every point y ∈ U.

Indeed, assuming that Theorem 2^{∗} is proved, we can complete the proof of Theorem 2 as follows.

The group G is connected by Theorem 1. Therefore, every irreducible component of Y is G-invariant by Lemma 2. From this and Lemma 1(i),(ii) we infer that it is sufficient to prove Theorem 2 for irreducible Y. In this case we argue by induction on dim Y.

Namely, the case dim Y = 0 is clear. Assume that the claim of Theorem 2 holds for irreducible G-invariant subvarieties in X of dimension < dim Y and consider the set U from Theorem 2^{∗}. Let Z_{1}, …, Z_{n} be all the irreducible components of the variety Y \U. By Lemma 2, every Z_{i} is G-invariant. Since dim Z_{i} < dim Y, the inductive assumption implies for every i = 1, …, n the existence of a unital algebraic family {ψ_{s_{i}}^{(i)}}_{s_{i}∈S_{i}} in G such that

(a) {ψ_{s_{i}}^{(i)}}_{s_{i}∈S_{i}} is derived from I;

(b) {ψ_{s_{i}}^{(i)}}_{s_{i}∈S_{i}} is exhaustive for the natural action of G on Z_{i}.

On the other hand, Theorem 2^{∗} and Lemma 4 imply the existence of a unital algebraic family {λ_{r}}_{r∈R} in G such that

(c) {λ_{r}}_{r∈R} is derived from I;

(d) {λ_{r}}_{r∈R} is exhaustive for the natural action of G on U.

The claim of Theorem 2 now follows from (a), (b), (c), (d) and Lemma 1(i),(ii). This completes the proof of Theorem 2 assuming that Theorem 2^{∗} is proved. □

We now turn to the proof of Theorem 2^{∗}. Consider the map

τ_{Y} : G × Y → Y × Y, (g, y) ↦ (g(y), y).  (6)

Its image Γ_{Y} is the graph of the natural action of G on Y:

Γ_{Y} = {(y_{1}, y_{2}) ∈ Y × Y | G(y_{1}) = G(y_{2})}.  (7)

Claim 1. Maintain the above notation.

(i) There exists a family {φ_{t}}_{t∈T} derived from I such that τ_{Y} (φ_{T} × Y) contains a dense open subset V of \overline{Γ}_{Y}, where bar stands for the closure in Y × Y.

(ii) \overline{Γ}_{Y} is irreducible.

Proof of Claim 1. If {ψ_{s}}_{s∈S} is an algebraic family in G, then the subset τ_{Y} (ψ_{S} × Y) of Γ_{Y} is the image of the morphism

S × Y → Y × Y, (s, y) ↦ (ψ_{s}(y), y)

of irreducible varieties (see (1)). Chevalley’s theorem on the image of morphism then implies that \overline{τ_{Y} (ψ_{S} × Y)} is an irreducible subvariety of \overline{Γ}_{Y} and τ_{Y} (ψ_{S} × Y) contains a dense open subset of \overline{τ_{Y} (ψ_{S} × Y)}.

From dim \overline{Γ}_{Y} ⩾ dim \overline{τ_{Y} (ψ_{S} × Y)} we conclude that there exists a family \{φ_{t}\}_{t∈T} derived from I on which the maximum of dim \overline{τ_{Y} (ψ_{S} × Y)} is attained when {ψ_{s}}_{s∈S} runs over all families derived from I. If {ψ_{s}}_{s∈S} is a family derived from I such that φ_{T} ⊆ ψ_{S}, then the maximality condition and irreducibility of \overline{τ_{Y} (ψ_{S} × Y)} imply that

\overline{τ_{Y} (ψ_{S} × Y)} = \overline{τ_{Y} (φ_{T} × Y)}.  (8)

Take an element g ∈ G. By Lemma 1(iii),(i), there is an algebraic family {ψ_{s}}_{s∈S} in G such that φ_{T} ⊆ ψ_{S} and g ∈ ψ_{S}. From (8) and (6) we then conclude that Γ_{Y} ⊆ \overline{τ_{Y} (φ_{T} × Y)}. Since \overline{τ_{Y} (φ_{T} × Y)} ⊆ \overline{Γ}, we get \overline{τ_{Y} (φ_{T} × Y)} = \overline{Γ}_{Y}. This completes the proof. □

Endow X × X with the action of G via the second factor:

g · (x_{1}, x_{2}) := (x_{1}, g(x_{2})), x_{i} ∈ X, g ∈ G.  (9)

The second projection X × X → X, (x_{1}, x_{2}) ↦ x_{2} is then G-equivariant and, by (7), Γ_{Y} and \overline{Γ}_{Y} are G-invariant.

Claim 2. {φ_{t}}_{t∈T} and V in Claim 1 can be chosen so that V is G-invariant.

Proof of Claim 2. Maintain the notation of Claim 1 and consider in \overline{Γ}_{Y} the G-invariant dense open subset

V_{0} := ⋃_{g∈G} g · V.  (10)

Since V_{0} is quasi-compact, its covering (10) by open subsets g · V, g ∈ G, contains a finite subcovering:

V_{0} = ⋃_{i}^{n} g_{i} · V for some elements g_{1}, …, g_{n} ∈ G.  (11)

By Lemma 1(iii), every g_{i} is contained in a family derived from I. Taking a product of {φ_{t}}_{t∈T} with these families, we obtain a family {ψ_{s}}_{s∈S} derived from I such that

φ_{T} ∘ g_{i}^{−1} ⊆ ψ_{S} for every i = 1, …, n.  (12)

Since V ⊆ τ_{Y} (φ_{T} × Y), from (6) and (9) we obtain

g_{i} · V ⊆ {(φ_{t}(y), g_{i}(y)) | t ∈ T, y ∈ Y }.  (13)

This yields

\begin{aligned}τ_{Y} (ψ_{S} × Y) & = \{(ψ_{s}(y), y) | s ∈ S, y ∈ Y \} \\ & = \{(ψ_{s}(g_{i}(y)), g_{i}(y)) | s ∈ S, y ∈ Y \} \\ & ⊇ \{(φ_{t}(g_{i}^{−1}(g_{i}(y))), g_{i}(y)) | t ∈ T, y ∈ Y \} (by (12)) \\ & ⊇ g_{i} · V (by (13)). \end{aligned} (14)

Thus V_{0} ⊆ τ_{Y} (ψ_{S} × Y) by (11) and (14). So, replacing {φ_{t}}_{t∈T} and V by, resp., {ψ_{s}}_{s∈S} and V_{0}, we may attain that V in Claim 1 is G-invariant. □

To complete the proof of Theorem 2^{∗}, consider the second projection

π_{Y} : \overline{Γ}_{Y} → Y, (y_{1}, y_{2}) ↦ y_{2};  (15)

it is a G-equivariant surjective morphism of irreducible varieties. Let {φ_{t}}_{t∈T} and V be as in Claim 1 and let V be G-invariant by Claim 2. Since V is a dense open subset of \overline{Γ}_{Y}, by Chevalley’s theorem on the image of morphism π_{Y} (V) contains a dense open subset of Y. Let U be the union of all dense open subsets of Y lying in π_{Y} (V). Since V is G-invariant and π_{Y} is Gequivariant, π_{Y} (V) is G-invariant. Therefore, U is also G-invariant.

Take a point y ∈ U. Since V ⊆ Γ_{Y}, π_{Y}^{−1}(y) ∩ Γ_{Y} = {(g(y), y) | g ∈ G}, and V ⊇ {(g(y), y) | g ∈ φ_{T} }, we have

∅ ≠ V ∩ π_{Y}^{−1}(y) = V ∩ Γ_{Y} ∩ π_{Y}^{−1}(y) = V ∩ {(g(y), y) | g ∈ G}  (16)
⊆ {(g(y), y) | g ∈ φ_{T} }.  (17)

By Lemma 3, {(g(y), y) | g ∈ G} is an irreducible locally closed subset of \overline{Γ}_{Y}. From (16) we then infer that V ∩ \{(g(y), y) | g ∈ G\} is a dense open subset of {(g(y), y) | g ∈ G}, and from (17) that φ_{T} (y) is dense in G(y). This completes the proof of Theorem 2^{∗} and hence that of Theorem 2. □

7. Proof of Theorem 3. Maintain the notation of the proof of Theorem 2. There is shown that the restriction of π_{Y} to V is a dominant morphism of irreducible varieties V → Y whose fiber over every point y of a dense open subset U of Y is isomorphic to a dense subvariety of G(y). Hence, the dimension of this fiber is dim G(y). The claim now follows from the fiber dimension theorem [Gro65, 5.6]. □

8. Proof of Theorem 4. By Lemma 1(iv), it suffices to give a proof for Y = X. We shall use the idea utilized in [Lun73, 4] for proving the existence of generic stabilizer for reductive group actions on smooth affine varieties. Below is maintained the notation used in the proof of Theorem 2.

Since any subfield of k(X) containing k is finitely generated over k, replacing X by an appropriate invariant dense open subset of X we can (and shall) find an irreducible affine normal variety Z and a surjective morphism

ρ: X → Z

such that ρ^{∗}(k(Z)) = k(X)^{G}. This equality implies that ρ is a separable morphism, see, e.g., [Bor91, AG, Prop. 2.4].

The construction yields that

(q_{1}) G(x) ⊆ ρ^{−1}(ρ(x)) for every point x ∈ X.

By the fibre dimension theorem and Theorem 3, further replacing X and Z by the appropriate open sets, we can (and shall) attain the following properties:

(q_{2}) for every point z ∈ Z, the dimension of every irreducible component

of ρ^{−1}(z) is equal to dim X − dim Z; (q_{3}) dim G(x) = dim G(x′) for every points x, x′ ∈ X.

Lemma 3(i) and (q_{3}) imply that G(x) is closed in X for every point x ∈ X. By Grothendieck’s generic freeness lemma [Gro65, 6.9.2], after replacing

Z by a principal open subset, we can (and shall) assume that

(q_{4}) there exists an affine open subset X_{0} of X such that ρ(X_{0}) = Z and

k[X_{0}] is a free ρ^{∗}(k[Z])-module. Below, for any subsets S ⊆ X and R ⊆ X × X, we put

S_{0} := S ∩ X_{0}, R_{0} := R ∩ (X_{0} × X_{0}).

Finally, replacing X by the invariant open set ⋃_{g∈G} g(X_{0}), we can (and shall) assume that

(q_{5}) the intersection of X_{0} with every G-orbit in X is nonempty.

Consider now in X ×X the G-invariant (with respect to action (9)) closed subset

X ×_{Z} X := {(x_{1}, x_{2}) ∈ X × X | ρ(x_{1}) = ρ(x_{2})}  (18)

and its affine open subset (X ×_{Z} X)_{0}.

Claim 3. (X ×_{Z} X)_{0} is dense in X ×_{Z} X.

Proof of Claim 3. Take a point (x_{1}, x_{2}) ∈ X ×_{Z} X. From (18) and (q_{1}) we infer that G(x_{1})×G(x_{2}) ⊆ X ×_{Z} X, and from (q_{5}) and Lemma 3(i) that (G(x_{1})×G(x_{2}))_{0} is a dense open subset of G(x_{1})×G(x_{2}). Therefore, since (x_{1}, x_{2})∈G(x_{1})×G(x_{2}), the closure of (G(x_{1})×G(x_{2}))_{0} in X ×_{Z} X contains (x_{1}, x_{2}). Whence the claim, because (G(x_{1})×G(x_{2}))_{0} ⊆(X ×_{Z} X)_{0}. □

Consider now the set

Γ := Γ_{X}  (19)

defined by (7). By (q_{1}), we have Γ ⊆ X ×_{Z} X. Since X ×_{Z} X is closed in X × X, this yields \overline{Γ} ⊆ X ×_{Z} X (see Claim 1(i)).

Claim 4. \overline{Γ} = X ×_{Z} X.

First, we shall show how to deduce Theorem 4 from Claim 4.

By (19) and Claims 1(ii), 4, the variety \overline{Γ} = X ×_{Z} X is irreducible. Consider its dense open subset V from Claim 2 and morphism π_{X}: \overline{Γ} → X defined by (15) for Y = X. If B is an irreducible component of \overline{Γ} \ V such that π_{X}(B) is dense in X, then, by the fiber dimension theorem, dim π_{X}^{−1}(x) > dim π_{X}^{−1}(x) ∩ B for every point x ∈ X lying off a proper closed subset of X. This and property (q_{3}) imply that V ∩ π_{X}^{−1}(x) is dense in π_{X}^{−1}(x) for every such x. On the other hand, π_{X}^{−1}(x) = ρ^{−1}(ρ(x)) × x by (18) and, as explained at the end of the proof of Theorem 2, V ∩ π_{X}^{−1}(x) is a dense open subset of G(x) × x. Since G(x) ⊆ ρ^{−1}(ρ(x)), this shows that G(x) is dense in ρ^{−1}(ρ(x)). Closedness of G(x) in X then implies that G(x) = ρ^{−1}(ρ(x)) for every point x ∈ X lying off a proper closed subset. This means that replacing Z by its open subset and X by the inverse image of this subset, we can (and shall) assume that ρ is an orbit map, i.e., the fibers of ρ are the G-orbits in X. Since ρ is a surjective separable morphism and Z is a normal variety, by [Bor91, Prop. II.6.6] this implies that ρ: X → Z is the geometric quotient. Thus the proof of Theorem 4 is completed provided that Claim 4 is proved. □

So it remains to prove Claim 4.

Proof of Claim 4. We divide it into three steps.

1. In view of Claim 3, it suffices to prove density of Γ_{0} in (X ×_{Z} X)_{0}. Since (X ×_{Z} X)_{0} is an affine variety, the latter is reduced to proving that if a function f ∈ k[(X ×_{Z} X)_{0}] vanishes on Γ_{0},

f|_{Γ_{0}} = 0,  (20)

then f = 0. To prove this, note that closedness of (X ×_{Z} X)_{0} in X_{0} × X_{0} implies the existence of a function h ∈ k[X_{0} × X_{0}] such that

h|_{(X×_{Z}X)_{0}} = f.  (21)

In turn, since k[X_{0}×X_{0}] = p_{1}^{∗}(k[X_{0}])⊗_{k} p_{2}^{∗}(k[X_{0}]), where p_{i}: X_{0}×X_{0} → X_{0}, (x_{1}, x_{2}) ↦ x_{i}, there are functions s_{1}, …, s_{m}, t_{1}, …, t_{m} ∈ k[X_{0}] such that

h = ∑_{i=1}^{m} p_{1}^{∗}(s_{i})p_{2}^{∗}(t_{i}),  (22)

2. By an appropriate replacement of h and s_{1}, …, s_{m}, t_{1}, …, t_{m} we may attain that t_{1}, …, t_{m} are linearly independent over ρ^{∗}(k[Z]). Indeed, by property (q_{4}), there are functions b_{1}, …, b_{r} ∈ k[X_{0}], linearly independent over ρ^{∗}(k[Z]), such that

t_{i} = ∑_{j=1}^{r} c_{ij}b_{j} for some c_{ij} ∈ ρ^{∗}(k[Z]), i = 1, …, m  (23)

In view of (22) and (23), we have

h = ∑_{j=1}^{r}(∑_{i=1}^{m} p_{1}^{∗}(s_{i})p_{2}^{∗}(c_{ij}))p_{2}^{∗}(b_{j}).  (24)

Take a point x = (x_{1}, x_{2}) ∈ (X ×_{Z} X)_{0}. Since ρ(x_{1}) = ρ(x_{2}), we have

c_{ij}(x_{1}) = c_{ij}(x_{2}) for all i, j.  (25)

¿From (24) and (25) we then obtain

\begin{aligned}h(x) & = ∑_{j=1}^{r}(∑_{i=1}^{m} s_{i}(x_{1})c_{ij}(x_{2}))b_{j}(x_{2}) \\ & = ∑_{j=1}^{r}(∑_{i=1}^{m} s_{i}(x_{1})c_{ij}(x_{1}))b_{j}(x_{2}). \end{aligned}  (26)

Hence, if we put

\begin{aligned}d_{j} & := ∑_{i=1}^{m} s_{i}c_{ij} ∈ k[X_{0}], \\ h̃ & := ∑_{j=1}^{r} p_{1}^{∗}(d_{j})p_{2}^{∗}(b_{j}) ∈ k[X_{0} × X_{0}], \end{aligned}  (27)

then we have h(x) = h̃(x) by virtue of (26). Given (21), this yields

h̃|_{(X×_{Z}X)_{0}} = f.  (28)

¿From (27) and (28) we conclude that replacement of s_{1}, …, s_{m} and t_{1}, …, t_{m} by, respectively, d_{1}, …, d_{r} and b_{1}, …, b_{r} is the one we are looking for.

3. Thus, keeping the notation, we shall now assume that t_{1}, …, t_{m} in (22) are linearly independent over ρ^{∗}(k[Z]).

Take an element g ∈ G and let W be the domain of definition of the rational function

ℓ = ∑_{i=1}^{m} s_{i}t_{i}^{g} ∈ k(X).

Since X is irreducible, W ∩g(W)∩X_{0}∩g(X_{0}) is a dense open subset of X. Let x be a point of this subset. Then the rational functions ℓ, s_{i}, t_{i}^{g} ∈ k(X) are defined at x and

a := (x, g^{−1}(x)) ∈ Γ_{0}.  (29)

¿From this we obtain

ℓ(x) = ∑_{i=1}^{m} s_{i}(x)t_{i}^{g}(x) = ∑_{i=1}^{m} s_{i}(x)t_{i}(g^{−1}(x))

=^{by (29)} (∑_{i=1}^{m} p_{1}^{∗}(s_{i})p_{2}^{∗}(t_{i}))(a) =^{by (22)} h(a) =^{by (21)} f(a) =^{by (20)} 0.

So ℓ vanishes on a dense open subset of X; whence ℓ = 0. Thus, it is proved that

∑_{i=1}^{m} s_{i}t_{i}^{g} = 0 for every g ∈ G.  (30)

Since Z is affine and ρ^{∗}(k(Z)) = k(X)^{G}, the field of fractions of ρ^{∗}(k[Z]) is k(X)^{G}. This implies that t_{1}, …, t_{m} are linearly independent over k(X)^{G}. In turn, by Artin’s theorem [Bou59, §1, no. 1, Thm. 1], this linear independency yields the existence of elements g_{1}, …, g_{m} ∈ G such that

det(t_{i}^{g_{j}}) ≠ 0.  (31)

Combining (30) and (31) we obtain s_{1} = … = s_{m} = 0. From this, (22), and (21), we then infer that f = 0, as claimed. □

9. Distinguished connected subgroups of Aut(X). Some collections I of unital algebraic families in Aut(X) are naturally distinguished. They generate distinguished connected subgroups Aut(X)_{I} of Aut(X) that are of interest.

The first example is the collection U of all unital algebraic families in Aut(X). We shall denote Aut(X)_{U} by Aut(X)^{0} and call the identity component of Aut(X). The group Aut(X)/Aut(X)^{0} will be called the component group of Aut(X).

Theorem 7. Let X be an irreducible variety such that Aut(X) is a finite group. Then Aut(X)^{0} = {id_{X}}.

Proof. Let {φ_{t}}_{t∈T} be a unital algebraic family in Aut(X). Take a point x ∈ X. Irreducibility of T implies irreducibility of the image I_{x} of morphism (3). Finiteness of Aut(X) (resp., unitality of {φ_{t}}_{t∈T}) implies finiteness of I_{x} (resp., x ∈ I_{x}). This yields I_{x} = {x}, i.e., φ_{T} = {id_{X}}; whence the claim. □

The component group of Aut(X), in contrast to that of an algebraic group, may be infinite.

Remark 1. If k is uncountable, then the same argument as in the proof of Theorem 7 shows that if Aut(X) is countable (this may happen, see Examples 1, 2 below), then Aut(X)^{0} = {id_{X}} and hence the component group of Aut(X) is countable.

Example 1. Let X be a surface in A^{3} defined by the equation x_{1}^{2}+x_{2}^{2}+x_{3}^{2} = x_{1}x_{2}x_{3} + a where a ∈ k. By [Èl’-H74], if a is generic, then Aut(X) contains a subgroup of finite index which is a free product of three subgroups of order 2.

Example 2. Let char k = 0 and let X be a smooth irreducible quartic in P^{3}. Then Aut(X)^{0} = {id_{X}} by [Mat63], and, according to the classical Fano–Severi result, for a sufficiently general X there is a bijection between Aut(X) and the (countable) set of solutions (a, b), a > 0 of the Pell equation x^{2} − 7y^{2} = 1 (see [MM64, pp. 353–354]).

Example 3. Let X be the underlying variety of an algebraic torus G of dimension n > 0. The automorphism group Aut_{gr}(G) of the algebraic group G is embedded in Aut(X) and is isomorphic to GL_{n}(Z). The map G → Aut(X), g ↦ ℓ_{g}, where ℓ_{g}: X → X, x ↦ gx, identifies G with a subgroup of Aut(X). By [Ros61, Thm. 3],

Aut(X) = Aut_{gr}(G) ⋉ G.  (32)

Let {φ_{t}}_{t∈T} be a unital algebraic family in Aut(X). It follows from [Ros61, Thms. 2 and 3] that there exist a morphism α: T → G and the elements s ∈ G, g ∈ Aut_{gr}(G) such that φ̃(t, x) = ℓ_{α(t)s}(g(x)) for every t ∈ T, x ∈ X (see (1)), i.e., φ_{t} = ℓ_{α(t)s} ∘ g. From this, (32), and unitality of {φ_{t}}_{t∈T} we infer that g = {id_{X}}. Hence φ_{T} ⊆ G. This proves that Aut(X)^{0} = G and the component group of Aut(X) is isomorphic to GL_{n}(Z).

Example 4. By [Ram64, Cor. 1], Aut(X)^{0} is a connected algebraic group if X is an irreducible complete variety (and, in fact, more generally, semicomplete variety).

Theorem 8. Aut(A^{n}) = Aut(A^{n})^{0} for n ⩽ 2.

Proof. Let J(A^{n}) and T (A^{n}) be, resp., the de Jonquières subgroup and the tame subgroup of Aut(A^{n}). Recall [Ess00] that if x_{1}, …, x_{n} are the standard coordinate functions on A^{n}, then J(A^{n}) consists of all g ∈ Aut(X) such that

g^{∗}(x_{i}) = x_{i} + f_{i}, f_{i} ∈ k[x_{i+1}, …, x_{n}]  (33)

(it is meant that f_{n} ∈ k). Arbitrary polynomials f_{i} may occur in (33). The subgroup of Aut(X) generated by J(A^{n}) and Aff(A^{n}) is T (A^{n}).

For every t ∈ k = A^{1}, let g_{t} be the element of J(A^{n}) defined by

g_{t}^{∗}(x_{i}) = x_{i} + tf_{i}, f_{i} ∈ k[x_{i+1}, …, x_{n}].

Then {g_{t}}_{t∈A^{1}} is the unital algebraic family in J(A^{n}) containing g. Hence J(A^{n}) is a connected subgroup of Aut(X). Connectedness of Aff(A^{n}) then implies that T (A^{n}) is a connected subgroup of Aut(X) too.

The claim now follows from the equalities Aut(A^{1}) = Aff(A^{1}) and Aut(A^{2}) = T (A^{2}) (the first is easy and the second follows from the so-called Automorphism theorem, cf. [Ess00, 5.1.11]). This completes the proof. □

Conjecture 1. Aut(A^{n}) = Aut(A^{n})^{0} for every n.

A series of examples is obtained taking I to be a part of the collection G of all algebraic families {φ_{t}}_{t∈T} such that T is a connected algebraic group and φ̃ defined by (1) is an action of T on X. In this case, Aut(X)_{I} is a subgroup of Aut(X) generated, as an abstract group, by a collection of some connected algebraic subgroups of Aut(X). For char k = 0, the subgroups Aut(X)_{I} of this type were studied in [AFKKZ13, Sect. 1] where they are called “algebraically generated groups of automorphisms”. Propositions 1.3, 1.5 and Theorem 1.13 of [AFKKZ13] are the special cases of respectively the above Lemma 3, Theorem 2, and Theorem 4.

Some interesting parts I of G are obtained as collections of all families {φ_{t}}_{t∈T} in G such that the algebraic group T has a certain property.

For instance, requiring that T is affine one obtains the collection G_{aff}. Theorems 5 and 6 give examples of dependency between the groups Aut(X)_{G}, Aut(X)_{G_{aff}} and geometric properties of X. Here is another example.

Example 5. If Aut(X)_{G_{aff}} ≠ {id_{X}}, then X is birationally isomorphic to the product of A^{1} and a variety of dimension dim X − 1. This follows from [Mat63, Cor. 1].

Developing the idea of [Pop11, Def. 1.36], one obtains another example of interesting collection of families taking I to be the collection G(F) of all families {φ_{t}}_{t∈T} in G such that T is isomorphic to a fixed connected algebraic group F.

For F = G_{a} this yields the important subgroup Aut(X)_{G(G_{a})} in Aut(X) introduced^{1} in [Pop05, Def. 2.1] and called in this paper “∂-generated subgroup”. Its close relation to constructing a big stock of varieties with trivial Makar-Limanov invariant was demonstrated in [Pop11]. Later in [AFKKZ13] transitivity properties of Aut(X)_{G(G_{a})} (called in this paper “the special automorphism group of X”) were studied. By [Pop11, Lemma 1.1], Aut(X)_{G(G_{a})} coincides with the subgroup of Aut(X) generated by all connected affine subgroups of Aut(X) that have no nontrivial characters.

Another interesting case is F = G_{m}. Since the union of all maximal tori of a connected reductive group is dense in it, Aut(X)_{G(G_{m})} coincides with the subgroup of Aut(X) generated by all connected reductive subgroups of Aut(X). This implies that

Aut(X)_{G_{aff}} = Aut(X)_{G(G_{a}) ⋃ G(G_{m})}.

Indeed, let H be a connected affine algebraic group with a maximal torus T and the unipotent radical R_{u}(H), and let π: H → H/R_{u}(H) be the canonical projection. By [Bor91, Prop. 11.20], π(T) is a maximal torus in H/R_{u}(H). Conjugacy of maximal tori and density of their union in H/R_{u}(H) yield H/R_{u}(H) = π(S) for the subgroup S in H generated by all maximal tori. Whence the claim.

10. Proof of Theorem 5. Since G lies in Aut(X)_{G_{aff}} , by Corollary 2 it suffices to show that neither of Aut(X)_{G_{aff}} -orbits is open in X.

Assume the contrary and let O be a Aut(X)_{G_{aff}} -orbit open in X. Take a point x ∈ O. By Theorem 2, a certain family {φ_{t}}_{t∈T} derived from G_{aff} is exhaustive for the action of Aut(X)_{G_{aff}} on X. Then O is the image of morphism (3). Since O is open in X, this morphism is dominant. On the other hand, the definitions of derived family and G_{aff} imply that T is a product of underlying varieties of connected affine algebraic groups. But such underlying varieties are rational (see [Pop13, Lemma 2] for a four-lines proof; we failed to find an earlier reference for a proof valid in arbitrary characteristic). Hence T is a rational variety. This and dominance of morphism (3) then imply that X is unirational — a contradiction. □

1At the irrelevant assumption X = A^{n}.

10. Proof of Theorem 6. (i): Assume the contrary and let C be a nontrivial connected affine algebraic subgroup of Aut(X). Then there exists a point x ∈ V such that V ∩ C(x) is an irreducible locally closed set of positive dimension. Hence there exists a point y ∈ V ∩ C(x), y ≠ x. By the condition of 2-transitivity, for every point z ∈ V, z ≠ x, there exists an element g ∈ Aut(X) such that g(x) = x, g(z) = y. This implies that for the subgroup H := g^{−1} ∘ C ∘ g we have z ∈ H(x). Therefore, for the connected subgroup G of Aut(X) generated by all conjugates of C in Aut(X) we have V ⊆ G(x); whence G(x) is open in X. From this, arguing as in the proof of Theorem 5, we deduce that X is unirational — a contradiction.

(ii): Assume the contrary and let A be a connected algebraic subgroup of Aut(X). Since, by (i), A contains no nontrivial connected affine algebraic subgroups, the structure theorem on algebraic groups [Bar55], [Ros56] implies that A is an abelian variety. The same argument as in the proof of (i) then shows that one of the orbits O of the connected subgroup G of Aut(X) generated by all conjugates of A in Aut(X) is open in X and there exists a surjective morphism Z → O, where Z is a product of several copies of the underlying variety of A. Since Z is then the underlying variety of an abelian variety too, this contradicts the assumption on X. □

References

[AFKKZ13] I. Arzhantsev, H. Flenner, S. Kaliman, F. Kutzschebauch, M. Zaidenberg, Flexible varieties and automorphism groups, Duke Math. J. 162 (2013), no. 4, 767–823.

[Bar55] I. Barsotti, Structure theorems for group-varieties, Ann. Mat. Pura Appl. (4) 38 (1955), no. 4, 77–119.

[Bor91] A. Borel, Linear Algebraic Groups, 2nd Enlarged Edition, Springer, New York, 1991.

[Bou59] N. Bourbaki, Algèbre, Chapitre V, Hermann, Paris, 1959. [Èl’-H74] M. H. Èl’-Huti, Cubic surfaces of Markov type, Math. USSR-Sbornik 22

(1974), no. 3, 333–348.

[Gro65] A. Grothendieck, EGA IV, 2, Publ. Math. I.H.É.S. 24 (1965), 5–231.

[Lun73] D. Luna, Slices étales, Bull. Soc. Math. France 33 (1973), 81–105.

[Mat63] H. Matsumura, On algebraic groups of birational transformations, Atti Accad. Naz. Lincei Rend. Cl. Sci. Fis. Mat. Natur. 34 (1963), no. 8, 151–155.

[MM64] H. Matsumura, P. Monsky, On the automorphisms of hypersurfaces, J. Math. Kyoto Univ. 3 (1964), no. 3, 347–361.

[Pop05] V. L. Popov, Open problems, in: Affine Algebraic Geometry, Contemp. Math. 369, Amer. Math. Soc., Providence, RI, 2005, pp. 12–16.

[Pop11] V. L. Popov, On the Makar-Limanov, Derksen invariants, and finite automorphism groups of algebraic varieties, in: Affine Algebraic Geometry: the Russell Festschrift, CRM Proceedings and Lecture Notes 54, Amer. Math. Soc., Providence, RI, 2011, pp. 289–311.

[Pop13] V. L. Popov, Rationality and the FML invariant, J. Ramanujan Math. Soc. 28A (2013), special issue, 409–415.

[PV94] V. L. Popov, E. B. Vinberg, Invariant theory, in: Algebraic Geometry IV, Encyclopaedia of Mathematical Sciences, Vol. 55, Springer-Verlag, Berlin, 1994, pp. 123–284.

[Ram64] C. P. Ramanujam, A note on automorphism groups of algebraic varieties, Math. Annalen 156 (1964), 25–33.

[Ros56] M. Rosenlicht, Some basic theorems on algebraic groups, Amer. J. Math. 78 (1956), 401–443.

[Ros61] M. Rosenlicht, Toroidal algebraic groups, Proc. Amer. Math. Soc. 112 (1961), no. 6, 984–988.

[Ess00] A. van der Essen, Polynomial Automorphisms and the Jacobian Conjecture, Progress in Mathematics, Vol. 190, Birkhäuser Verlag, Basel, 2000.

Steklov Mathematical Institute, Russian Academy of Sciences, Gubkina 8, Moscow 119991, Russia

National Research University Higher School of Economics, Myasnitskaya 20, Moscow 101000, Russia

   E-mail address: popovvl@mi.ras.ru