I managed to install basic Ubuntu system using the Ubuntu Server edition (http://www.ubuntu.com/download/server).
When asked about packages that should be installed, I left all the check-boxes empty, which leaves mostly barebone Ubuntu, which boots to console.
Then following this guide: http://www.linux.com/community/blogs/133-general-linux/803937-installupgrade-linux-kernel-to-3183-stable-in-ubuntulinux-mintpeppermint , Updated the kernel version to 3.18-3.
After that to get xfce4, I used:
sudo apt-get install xfce4-desktop
The display driver still has issues though. Portions of the screen are not repainted and sometimes even the terminal stops responding but it's a start.
