I'm a software engineer with about 20 years of experience. I've worked in large and small orgs; on a website/service with 100-500 million monthly active users, and another with 10-50m MAUs; and on Big Science. I work mainly in distributed systems, and have branched out to database implementation in the past 5 years. I really love distsys.
The main economic forces that I currently see operating on my career are: (a) the ascendance of the small handful of cloud computing providers; and (b) the remaking of many facets of the economy by AI, much of this to take place in the coming decades.
On (a): I'm happy in my current job, but have found that only companies at the largest scale want to pursue the distsys approaches that I'm most interested in (e.g., gossip/CRDTs, consensus algorithms, etc.). Outside of those companies, it's difficult to get funded for anything besides, for example, leader election, or adopting a 3rd party distributed data store or cache.
OK, so on to my question... Should I:
(1) Stay where I am. (I'm lucky to work on something that I enjoy, even if it's not an exact match.) (In other words: don't worry about your 1st world problems.)
(2) Try to find distsys work at one of those large cloud computing providers.
(3) Branch out into another area, e.g., distributed machine learning, or data/compute management/scheduling systems for AI.
(4) Something else I haven't thought of.
What do you think?
I'm stuck in thinking about what to work on. Thanks in advance for your help!
Similarly there is not a market for that many large distributed systems.
What there is an endless market for is new applications from the medium to small side. For instance there is the resurgence of "low code", "no code", and many participants who feel underserved with current technology compared to what Google gets.
See this guy
as a transitional figure between "modern data centers" and "buy any 1 liter PC off ebay and see what happens".
The Baltimore Police Department keeps track of incident reports in a big Lotus Notes database, so do many 3-letter and more-letter agencies in the area south of there. Lotus Notes had some algorithms for merging changes that people make to "objects" with a very JSON-like structure that meant people built amazing things in the 1990s. (e.g. 10 years ahead of Salesforce.com!)
It must have been patents that kept the technology out of other products but it seems those must be expired by now.
Can this be disproven? Are there companies of smaller scale that are attempting to solve these problems?