This is an issue that I looked into pretty seriously because I had an older laptop that was not running very smoothly at all. It took forever for programs to open and execute various tasks, and got slower and slower during the time I would use it, forcing me to reboot periodically just to refresh the system. I experimented between Xubuntu and Ubuntu, and kept track of times to measure the actual difference.
The technical answer is, yes, Xubuntu is faster than regular Ubuntu.
But, the more accurate answer is, no, probably not in any way that will make a difference to you.
If you just opened up Xubuntu and Ubuntu on two identical computers and had them sit there doing nothing, you would see that Xubuntu's Xfce interface was taking up less RAM than Ubuntu's Gnome or Unity interface.
But, so what? You don't just run computers to have them sit there. And here's where the rubber hits the road.
Xubuntu isn't just Xfce, it's also a whole suite of bundled applications to handle most tasks you want to do. For example, if you wanted to do word processing, Xubuntu comes with Abiword preinstalled. On regular Ubuntu, you're offered LibreOffice Writer by default.
Abiword may take less resources, but you pay for that with less features. And while I probably don't use 90% of the features available in Writer, the difference was noticeable enough that I found I would rather use Writer.
Which brings us to the real issue when determining whether Xfce is faster or not. Even though underneath Writer, Xubuntu's Xfce interface is using less resources than Ubuntu's Gnome or Unity, the question you're coming down to is this: Is Xfce using less resource to a degree that would make a difference in how fast LibreOffice Writer runs?
The answer is, no, not really. LibreOffice uses so much resource on its own that it quickly eats up any savings in using Xcfe. For all intents and purposes, when running something like LibreOffice on Xubuntu, the savings that Xcfe offers you is so negligible as to not matter.
So, if you're content to stay within the suite of applications offered by Xubuntu, then you might see some difference.
If, however, your hope is to run more or less the same programs as you would with Gnome or Unity but are hoping to get an appreciable performance boost, then there is no meaningful difference. By "meaningful difference", I mean a difference in how fast programs open or execute tasks to a degree that a human can tell the difference. I don't know about you, but saving .03 seconds if it takes a minute and a half to open a program doesn't mean anything to me.
In the end, I found that buying more RAM for my laptop - going from 512MB to 1GB - was the real solution I was looking for. The dream is that by getting an optimized desktop manager you can breathe new life into an old machine, and I'm sure some people have a situation where they pulled that off. But my experience was that trying to tweak the interface and applications ultimately had too many feature drawbacks for too little return.
So the question about which is faster, Xubuntu or Ubuntu, comes down to this: what applications do you want to use? If you are willing to go with the whole Xubuntu suite and all the bundled applications that are meant to be more minimal in resource use, then you might notice a performance improvement. But as soon as you start using bigger applications, the interface savings will become moot.
