Abstract:Any abstract (not necessarily continuous) group automorphism of a simple, compact Lie group must be continuous due to Cartan (1930) and van der Waerden (1933). The purpose of this paper is to study a similar question in nilpotent Lie groups. For many simply connected 2-step nilpotent Lie groups, we show any abstract group automorphism is continuous "up to discontinuity due to the center and field automorphisms of $\mathbb{C}$." Such groups include (1) a generic simply connected 2-step nilpotent Lie group of type $(p,q)$ with $p$ large, (2) all twelve simply connected 2-step nilpotent Lie groups of dimensions 6 or less except for one example, and (3) Iwasawa N-groups of simple Lie groups of rank 1, i.e., "Heisenberg groups". To our knowledge, the exception in (2) is the first nilpotent Lie group whose automorphism group is not of the type described above. In the proof for (3), we use tools from Riemannian geometry, even though the result is purely algebraic. All of the three cases are derived from one key result that gives a sufficient condition for the automorphism group to be of the type described above.

On automorphisms of certain free nilpotent-by-abelian Lie algebras

Automorphisms in Certain Nilpotent-by-Abelian Varieties of Groups

On the automorphisms of a nilpotent π-group of finite rank

On Commuting automorphisms of nilpotent Lie algebras

Automorphisms of the Category of Free Non-Associative Algebras with Unit

A note on automorphisms of free nilpotent groups

Automorphisms of categories of free algebras with unit for classical varieties of non-associative algebras

A Classification of Free and Free-Like Nilpotent Groups

Local derivations and automorphisms of nilpotent Lie algebra

On the automorphisms of a nilpotent group of finite rank(Ⅱ)

The Automorphism Group of a Class of Nilpotent Groups with Infinite Cyclic Derived Subgroups

Abstract Group Automorphisms of Two-Step Nilpotent Lie Groups and Partial Automatic Continuity

On generic properties of nilpotent algebras

Model-theoretic properties of nilpotent groups and Lie algebras

Commutative post-Lie algebra structures and linear equations for nilpotent Lie algebras

On post-Lie algebras structures coming from simply transitive NIL-affine actions

Some three-dimensional non-nilpotent Leibniz algebras: automorphism groups

The 2-nilpotent multiplier of n-Lie algebras and its applications

On index of certain nilpotent Lie algebras

Multiplicative Lie algebra structure on nilpotent group of class $2$

Automorphism groups of some 3-dimensional Leibniz algebras