I had the similar problem with rapidscene.ws

It was running on 1GB RAM VPS and the memory usage was just 300MB but the CPU Load was 2-3.

Then I tried tweaking my.cnf and installed eAccelrator, Memchace, etc. but it didn't help much. I asked XSLTel to see if he can do it and KABOOM ! He optimized the server within 12 minutes (exact).

I asked him what he did, and the reply was that he changed the PHP Apache Handler and installed PHP-CGI Handler and since then the load hasn't gone over 0.5 and my VPS hasn't been down (its been 2 month without 0 downtime)