As you haven't mentioned the version of VirtualBox nor the version of the system you were installing, I'll describe the solution I've found for VB v4.1.12 and Ubuntu 12.04 LTS guest machine.
This OS version was installed with X Server from quantal (12.10) and displayed error message about low-graphics mode. When you remove this error window using Esc and log in using console, try to uninstall current X Server from quantal from the guest machine:
sudo apt-get purge xserver-xorg-core-lts-quantal
sudo apt-get autoremove
As soon as this will be done, default core X Server will be installed automatically. If this will not happen, try to install it manually (including all its dependencies):
sudo apt-get install xserver-xorg-core
After reboot guest OS should work properly.