The load will be due to HDD IO. If you have multiple people using RL, raring and unraring, uploading and downloading, the HDD can't keep up.

Of course, on a VPS your HDDs are shared with others. If you're such a heavy user that the load on your VPS goes so high, it might be time to get a dedicated server.