Chances are, your OS is caching the memory for later use and that shows up as used memory.

Looking at your load averages, your CPU utilization and the number of running/sleeping processes, I don't suppose you should worry about the memory utilization.