Say I wanted to host Wordpress on a VPS: What is the main advantage of doing that over, say, a $5.00/month subscription to a shared hosting provider who often have one click installs of Wordpress and you also don't have to harden that Wordpress instance (e.g Apache gets automatically updated on a shared hosting provider, among other things).
What would you go for personally. VPS or the shared hosting option?
A VPS, by contrast, is going to give you more computing power. If a large number of visitors is a concern, a VPS will be better.
Either way, you can get a VPS with low resources from Digital Ocean or linode for very cheap, so price shouldn't be a huge issue.
Personally, I'd go for a VPS. It's fun to set up a web server and control the entire process. If that's not what interests you about making a website, shared hosting is absolutely a fine option.
Is your bottleneck time or money? And what about in 2 years.
Additionally: easyengine will make most of it easier for you: https://easyengine.io/
Don't forget basic Unix hardening ( fail2ban, ... )