Outside of insecurities about age, it's the same for everyone. All that really matters is what you accomplish and how respectable you are. Many advisors actually prefer older students since industry experience usually results in much more productive students (at least in biology, I don't know how other fields work).
Financial considerations: pay is crappy, so it'll be tough to support a family if you haven't saved up. Health problems can also get expensive if you're in the US and they're only going to get more likely at your age. Make sure the insurance they provide is good when you're interviewing.
Otherwise: totally worth it. You can be 40 with a PhD or 40 without a PhD, but you'll be 40 no matter what.