Ya, I also upgraded from 3.x to 4.x since I felt it was good at SEO by default. But the problems with 4.X I feel is

1. They release a new 4.x version for 10 days, so not stable.
2. Skins is a major problem with 4.x.
a. Skins are not as customizable as 3.x due to the use of only Style Vars in 4.x, it makes it more difficult for customization.
b. Also most skins are not adaptable to future versions, so you need to update the skin for the latest version and again customize it.
3. Still has a lot of vulnerabilities.

I would recommend not to downgrade to 3.x since I hope there will be a stable one 4.1, at the same time not to upgrade to 4.x till it becomes stable.