Just like others have said you get SSH (is basically a username / password and IP) to connect and via Putty (or any other SSH program) you can connect to your server. This is mostly the case when your using Linux/Unix operating systems. You will only be able to see the "command text" instead of the GUI (graphical interface) until you install it.

Do not be afraid of the command line (black/white) only there are many tutorials for all operating systems on how to get a GUI up and running. If you do not want to try there is always Windows which some VPS will provide you with a VPS Windows that will have "Remote Desktop Connection" and you do not need to install anything for a GUI.