Abstract: Highly frustrated spin systems represent a central and challenging problem in condensed mater physics. To this problem, we introduce an algorithm based on mixed projected entangled pair states (m-PEPS), which is a novel type of tensor network. We use the famous Kitaev model on an infinite honeycomb lattice, which can be solved exactly, as a benchmark. With very limited parameters and finite scaling, our calculation results are in good agreement with the exact results, indicating the efficiency of our algorithm. After presenting the benchmark, we investigate the Kitaev-Heisenberg model, which was proposed to describe the effective magnetic momentum interaction in iridate Na$_2$IrO$_3$ which may be used to realize the spin liquid phase. However, our calculations suggest that the gapless spin liquid phase is not robust at the thermodynamic limit, and thus this phenomenon is very difficult to observe.

A Study of Highly Frustrated Spin Systems with mixed PEPS in Infinite Honeycomb Lattice