How do you say “No” when you're the right person for the job?

I'm very much a jack of all trades, working as a Product Manager which is the job I didn't know I wanted until I got it.

What I like most is the variety of responsibilities I have. I'm better at doing a large number of things than a large number of people are at doing those things. I'm not amazing at anything except that. I can learn quickly to do something up to a relatively high standard.

I'm passionate about doing things right and when people ask me to help them on X and Y, I genuinely want to because I think/know that I'll do a good job, even if it's outside my normal tasks.

Sometimes I feel like there is no one else I'd trust to do those tasks right and I feel like that may be the main problem. I like helping people too obviously.

So at the end of the day I find myself genuinely thinking that all the things I'm working on should be on my shoulders. In reality I get burnt out because the things that matter to me don't move enough, I just don't realize it in the moment.

Are there any tricks to figuring out on the spot whether you should take on a task or not?

My title is a bit of lie, it should read "when you think you're the right person"..