The delete button is only available for sitemaps that you have submitted from your account and not from other sources like some other plugin



You can only delete sitemaps in "By me" category

You may disallow sitemap and block it in robot.txt though i have no idea whether it could be harmful or not