RAM in common computers is addressed (i.e. referred to by programs) using sequences of bits that correspond to powers of two. When you're using a 32-bit operating system, it means that programs have (at most) 32 bits available to them to describe each address. This 32-bit limit, fundamentally, lies in hardware: the x86 family of processors originally reserved just 32 bits for addresses.
The total number of different unique sequences you can organise 32 bits in is 4,294,967,296. For computers, this means you can only refer to that many different addresses in the memory. To point to more addresses (so that each address still remains unique) you must have more bits.
That large total number translates to 4 GB. As for why, in practice, it can actually be a quarter less than that, the 3 GB barrier Wikipedia article will explain.
Physical Address Extension or PAE, on hardware level, is an extension the 32-bit addressing in x86 processors: PAE processors have 36 bits for memory, thus extending the range of addresses available for the operating system (which then divides this memory among the programs). When you're installing a PAE kernel, you're in effect installing low-level operating system support for this extended x86 hardware.
When you have a 64-bit processor (as most modern processors are), you can run operating systems and applications that are built to address memory using those 64 bits. This gives them a total of "18446744073709551616 different values, a number in excess of 18 quintillion". In theory, at least, that means you won't run out of addresses before you have more than 16,8 terabytes of RAM.
