Polonium is the only element to crystallise into a simple cubic structure at ambient conditions. Moreover, at high temperatures it undergoes a structural phase transition into a less symmetric trigonal configuration. It has long been suspected that the strong spin-orbit coupling in Polonium is involved in both peculiarities, but the precise mechanism by which it operates remains controversial. Here, we introduce a single microscopic model capable of capturing the atomic structure of all chalcogen crystals: Selenium, Tellurium, and Polonium. We show that the strong spin-orbit coupling in Polonium suppresses the trigonal charge and orbital ordered state known to be the ground state configuration of Selenium and Tellurium, and allows the simple cubic state to prevail instead. We also confirm a recent suggestion based on ab initio calculations that a small increase in the lattice constant may effectively decrease the role of spin-orbit coupling, leading to a re-emergence of the trigonal orbital ordered state at high temperatures. We conclude that Polonium is a unique element, in which spins, orbitals, electronic charges, and lattice deformations all cooperate and collectively cause the emergence of the only elemental crystal structure with the simplest possible, cubic, lattice.