Topological phase transitions in (Bi_1-xIn_x)₂Se₃ and (Bi_1-xSb_x)₂Se₃

We study the phase transition from a topological to a normal insulator with concentration x in (Bi1-xInx)2Se3 and (Bi1-xSbx)2Se3 in the Bi 2Se3 crystal structure. We carry out first-principles calculations on small supercells, using this information to build Wannierized effective Hamiltonians for a more realistic treatment of disorder. Despite the fact that the spin-orbit coupling (SOC) strength is similar in In and Sb, we find that the critical concentration xc is much smaller in (Bi 1-xInx)2Se3 than in (Bi 1-xSbx)2Se3. For example, the direct supercell calculations suggest that xc is below 12.5% and above 87.5% for the two alloys, respectively. More accurate results are obtained from realistic disordered calculations, where the topological properties of the disordered systems are understood from a statistical point of view. Based on these calculations, xc is around 17% for (Bi1-xIn x)2Se3, but as high as 78%-83% for (Bi 1-xSbx)2Se3. In (Bi 1-xSbx)2Se3, we find that the phase transition is dominated by the decrease of SOC, with a crossover or "critical plateau" observed from around 78% to 83%. On the other hand, for (Bi1-xInx)2Se3, the In 5s orbitals suppress the topological band inversion at low-impurity concentration, therefore accelerating the phase transition. In (Bi1-xIn x)2Se3 we also find a tendency of In atoms to segregate. © 2013 American Physical Society.