If your machine is not one of the fastest, I would assume that the system-monitor CPU-graph window itself is eating the CPU time.
I have Intel Duo 2x2.0GHz and I can't get the displayed load below maybe 50% if that graph window is open.
Go to the settings and adjust the refresh delay to 1sec or so. That should make a huge difference, but at least for me, it does not save this setting.
Or check the CPU usage with other tools like top from the terminal or indicator-multiload (usually has to be installed first), which displays a small graph in your panel.