so after all of that, my question is: how do you exactly down vote a story on HN without feeling icky cause you flagged it?
Flagging puts stuff in the moderation queue to be reviewed for abuse / spam. Please don't abuse it for other purposes.
Downvotes exist, but are hidden until you've posted more than some threshold I forget the exact value of.
The idea is to help prevent the downvote button from being even more of a "I don't like what you're saying but can't muster a convincing response to it" button than it already is, while making it convincingly difficult to manipulate via spam accounts.
It's much better at the second than the first, people being what they are, but it does seem to help with both at least enough to justify the feature's continued existence.
If something made front-page, there are enough people upvoting it that your downvote will not kick it off the first page anyway.
On-Topic: Anything that good hackers would find interesting. That includes more than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity.
Off-Topic: Most stories about politics, or crime, or sports, unless they're evidence of some interesting new phenomenon. Videos of pratfalls or disasters, or cute animal pictures. If they'd cover it on TV news, it's probably off-topic.
If it's not on topic (interesting) or if it's off topic (politics) feel free to flag it. Also if it has things like clickbait or self-promotion.