I think the server firewall will sufficiently do this job but it's pretty risky:
https://help.ubuntu.com/12.04/serverguide/firewall.html
The problem is, if you block all traffic except from your IP address, what happens if your IP address changes? You're locked out from your machine. If you know you have 2 routes into the server both with static IP addresses, you should be fine, but use this with caution and be aware that the only solution might involve reformatting the machine and losing all data should you lock yourself out.
It's probably possible using the firewall to only permit traffic through certain ports in certain ways, so you could (for instance) set your SSH port to something completely non standard, and leave that port open to any IP (this will give you a fallback if you lock yourself out). Port scanners will find the open port, but if the people gaining access aren't skilled as per your implication, it will stop them.