Disable the plugin and delete it from your dashboard. If you are using a caching plugin like wordpress total cache, remember to clear everything after deleting the plugin.