Moore's law does not describe the size of semiconductors, but the size of transistors on a semiconducting chip. The chip itself remains a whole piece of semiconductor, several millimeters (millions of atoms) large, but the density of transistors (i.e., the elements of computer logic) on it increases.
In principle, a single atom (of size of about 0.1nm) could serve as a logic gate, although the necessary wiring, cooking etc. would require more place. What in reality is the crux of the Moore's law is that by now we are able to produce structures that are smaller than the electron coherence length (a few hundred nanometers), which means that we cannot use anymore the semi-classical theory if semiconductors, which served well for more than half a century, and which is grounded in boltzmann equation. Instead we need to use full quantum description - starting with Landauer-Büttiker formalism for non-interacting electrons, and so on.