Maybe it's because I don't talk to many friends and family to begin with, but with the typical networking advice it feels too easy for my social activities to blow up considerably and I'm not trying to do that.
Slow down for a second. What do you mean by this?
Do you not feel like you have agency over how much time you invest in networking?
It reads like you think that you’ll somehow get swept away (by what?) if you even dip a toe in to it, and that you won’t be able to safely assert your own boundaries once it gets going.
Is that really how you feel? If so, are you sure it’s something that happens?
A lot of people who might say something like that are actually just kinda nervous about doing something they’re not accustomed to and so invent sketchy rationalizations to cover for it. Maybe that’s not you, but it sounds like it could be.
It's really worth socializing at least a little bit with everyone and anyone - just get to know their name... and slowly what they do on the weekend, if they have a partner/kids, etc., just for chit chat. It can be a few minutes at a time, water cooler type stuff.
You don't have to go out to lunch / drinks with people or get to know them deeply, but it really pays to not just sit at your desk and ignore everyone except for your manager/immediate coworkers.
* Keep in touch with past colleagues. Use linkedin. Congratulate them when they post something. This will take a few years.
* Go to 1 meetup/month. Talk to people there, but don't make plans outside of the meetup. This will take a few months.
* Write something regularly. Create a newsletter from it. This will take a few months to years.
* Post to HN anonymously. Keep making insightful comments. This will take a few weeks to months, depending on your level of insight.
All of these will grow your professional network without excess social activity.