Abstract:We review our recent theoretical advances in phase transition of cold atoms in optical lattices, such as triangular lattice, honeycomb lattice, and Kagomé lattice. By employing the new developed numerical methods called dynamical cluster approximation and cellular dynamical mean-field theory, the properties in different phases of cold atoms in optical lattices are studied, such as density of states, Fermi surface and double occupancy. On triangular lattice, a reentrant behavior of phase translation line between Fermi liquid state and pseudogap state is found due to the Kondo effect. We find the system undergoes a second order Mott transition from a metallic state into a Mott insulator state on honeycomb lattice and triangular Kagomé lattice. The stability of quantum spin Hall phase towards interaction on honeycomb lattice with spin-orbital coupling is systematically discussed. And we investigate the transition from quantum spin Hall insulator to normal insulator in Kagomé lattice which includes a nearest-neighbor intrinsic spin-orbit coupling and a trimerized Hamiltonian. In addition, we propose the experimental protocols to observe these phase transition of cold atoms in optical lattices.

GOSSAMER SUPERFLUID AND ANTIFERROMAGNETIC STATE OF COLD ATOMS IN OPTICAL LATTICES

Artificial Staggered Magnetic Field for Ultracold Atoms in Optical Lattices

Staggered-Vortex Superfluid of Ultracold Bosons in an Optical Lattice

Quantum Phase Transition of Cold Atoms Trapped in Optical Lattices

Magnetic Quantum Phase Transition of Cold Atoms in an Optical Lattice

QUANTUM PHASE TRANSITIONS OF COLD ATOMS IN HONEYCOMB OPTICAL LATTICES

Cold Atoms in a Two-Dimensional Triangular Optical Lattice As an Artificial Frustrated System

Spin-Orbit Driven Transitions Between Mott Insulators and Finite Momentum Superfluids of Bosons in Optical Lattices

Quantum Phase Transitions of Fermionic Atoms in an Anisotropic Triangular Optical Lattice

Superfluid-Mott-Insulator Transition in an Optical Lattice with Adjustable Ensemble-Averaged Filling Factors

Observation of Two-Dimensional Mott Insulator and Π-Superfluid Quantum Phase Transition in Shaking Optical Lattice

Observation of 2D Mott insulator and $\pi$-superfluid quantum phase transition in shaking optical lattice

Superfluid-Mott-Insulator Transition of Spin-2 Cold Bosons in an Optical Lattice in A Magnetic Field

Quantum Phases of Ultracold Bosonic Atoms in a Two-Dimensional Optical Superlattice

Exploring Exotic Superfluidity of Polarized Ultracold Fermions in Optical Lattices

Fulde-Ferrell-Larkin-Ovchinnikov Or Majorana Superfluids: The Fate Of Fermionic Cold Atoms In Spin-Orbit-Coupled Optical Lattices

Simulation and Detection of Dirac Fermions with Cold Atoms in an Optical Lattice

Observation of 2D Mott insulator and $π$-superfluid quantum phase transition in shaking optical lattice

Tunable Superfluidity and Quantum Magnetism with Ultracold Polar Molecules.

A New Non-Abelian Topological Superfluid of Cold Fermi Gases in Anisotropic and Spin-Dependent Optical Lattices

Generating Topological Optical Flux Lattices for Ultracold Atoms by Modulated Raman and Radio-Frequency Couplings