Abstract:The possible existence of a quantum spin liquid (QSL) phase, an exotic state of matter with long-range quantum entanglement and fractionalized excitations, in $\alpha$-RuCl$_3$ has sparked widespread interests in exploring QSLs in various Kitaev models under magnetic fields. Recently, a $K$-$J$-$\Gamma$-$\Gamma'$ model has been proposed to accurately describe the compound by fitting the measured thermodynamic data, where $J$ is Heisenberg interaction, and $\Gamma$, $\Gamma'$ are off-diagonal exchanges on top of the dominant Kitaev coupling $K$. Based on this effective model, an intermediate QSL phase in presence of an out-of-plane fields along the $[1 1 1]$ direction, between the low-field zigzag order and high-field polarized phase, has been predicted. By combining density matrix renormalization group (DMRG), exponential tensor renormalization group (XTRG), and variational Monte Carlo (VMC) calculations, we address the nature of this QSL phase in the honeycomb $K$-$J$-$\Gamma$-$\Gamma'$ model under the $[1 1 1]$-direction field. Our DMRG calculations find the algebraic-like decay of spin correlation function, the finite spin scalar chiral order, and lattice nematic order. Together with the XTRG results of power-law specific heat at low temperature, our findings naturally suggest a gapless nematic chiral spin liquid. On the other hand, our VMC study finds a gapped nematic chiral spin liquid with the variational energy very close to that obtained in DMRG. As VMC finds a very small gap that is beyond the current resolution of both ground-state DMRG and finite-temperature XTRG calculations on finite-width cylinders, we resort the full clarification for the nature of the intermediate QSL to future studies. Lastly, we discuss the implications of our results to the recent experiment on $\alpha$-RuCl$_3$ and the QSL-like phase in a similar $K$-$\Gamma$-$\Gamma'$ model.

Multinode quantum spin liquids in extended Kitaev honeycomb models: the view from variational Monte Carlo

Multinode Quantum Spin Liquids on the Honeycomb Lattice

Symmetry-protected gapless spin liquids on the strained honeycomb lattice

Gapless Quantum Spin Liquid in a Honeycomb Γ Magnet

Quantum liquids of the S=3/2 Kitaev honeycomb and related Kugel-Khomskii models

Effect of ring-exchange interactions in the extended Kitaev honeycomb model

Unifying Kitaev magnets, kagome dimer models and ruby Rydberg spin liquids

Unveiling the S=3/2 Kitaev honeycomb spin liquids

Quantum spin liquid from electron-phonon coupling

One Proximate Kitaev Spin Liquid in the K-J-Γ Model on the Honeycomb Lattice

Z2 spin liquid in S=1/2 Heisenberg model on Kagome lattice: A projective symmetry group study of Schwinger-fermion mean-field states

Assembling Kitaev honeycomb spin liquids from arrays of 1D symmetry protected topological phases

Chiral spin liquid in a generalized Kitaev honeycomb model with $\mathbb{Z}_4$ 1-form symmetry

Quantum phase transition between spin liquid and spin nematics in spin-1 Kitaev honeycomb model

Topological Phase Transition and Single/multi Anyon Dynamics of Z 2 Spin Liquid

Quantum spin liquids on the diamond lattice

Nematic chiral spin liquid in a Kitaev magnet under external magnetic field

Topological phase transition and single/multi anyon dynamics of Z2 spin liquid

Possible Kitaev Quantum Spin Liquid State in 2D Materials with S=3/2

Classification of spin-$1/2$ fermionic quantum spin liquids on the trillium lattice

Interplay of Kitaev Interaction and Off-diagonal Exchanges: Exotic Phases and Quantum Phase Diagrams