How to be a software engineer who works on challenging problems?
I've been a software engineer in a midsized Canadian company for the past 2.5 years. I mainly do application level CRUD work. This work is pretty easy and we're mostly solving business problems, and never really get involved with technically challenging work. I always wanted to work on harder problems that deal with things lower in the stack. However, I'm not sure the process to go about this. My undergraduate degree isn't in CS, nor did I do any research in undergrad. Past few years I've been rigorously teaching myself CS fundamentals. I've read multiple books about databases and distributed systems. I've played around with these technologies in small side projects. But I have no idea how to pursue a career exploring these concepts in a lower level. Any advice for someone like me?
Personal take: Apply for lots of jobs, at companies that have these problems, on the roles/teams that solve them. Some'll have room for a relatively junior engineer who can teach themselves. You should talk about technical challenges as a big reason you're excited by the role, but not to the exclusion of all the usual "care about the business/user" stuff. Practice talking succinctly about your side projects and what you learned/demonstrated, and be able to get detailed if asked.
You'll learn more quickly on the job, from people with practical experience, and get paid to do it. Your lack of CS degree isn't necessarily an issue, any more than for jobs where you write CRUD. (Of course, some employers will filter you out. I don't know if you could meet visa requirements in a different country, work experience sometimes helps with this)
In addition to the great advice already given:
1. Don't wait for your job or your industry to give you challenging problems.
2. Figure out which problem domains you find the most attractive.
Just start doing it. There is always more work for people who can solve the problems that others can’t. No degree is required. Most of what is taught in school is eclipsed by a few years of hard problem solving and coding every day.
Try contributing to open source. You can easily find very interesting and challenging projects and there's always more tasks than people who can work on them. Many projects have great documentation/code comments and welcome new contributors. Also, a lot of people would be happy to explain things to you on mailing lists, IRC, etc.
SWE does fuck all in the scheme of things
Learn battery tech and renewable energy
Try embedded, although embedded systems have a lot of CRUD stuff they tend to be more challenging, require innovation and definitely require low level knowledge of the system especially on the low end of the systems, those with low memory and weak CPUs.
I don’t know that there’s any way to get handed a challenging problem on your JIRA board. But if you work at a place that’s big and complex enough, and you have visibility into enough projects, you will notice challenging problems lying around untouched. Maybe no one realizes they exist. Maybe they are too deep in the sprint grind to take serious swings at them. In any event, make a case that the problem has business value and that you have a good idea how to solve it. If you are sufficiently credible and your leadership is sufficiently forward-thinking, it will be approved.
This is one model for Staff-plus careers, by the way.
99.98% of all "programming" is CRUD and line of business _applications_ (the clue is in the name).
It is not realistic to turn a CRUD job into a low level / "challenging" systems engineering job. If you want a job doing those things, you need to be qualified, apply for, and be hired for such a job, and CRUD experience won't get you very far.
No amount of flight attendant experience will get you a pilot job. No amount of nursing experience will get you a doctor job. Same thing applies here, even if companies doing CRUD work like to interview as if and otherwise pretend that their jobs are a stepping stone to such things.
There are a ton of interesting distributed systems problems in cryptocurrency. More generally, open source projects will often share issues for beginners which might be good to start with. Find a project you like and see if there's any way you can help.
Most projects label their low-hanging-fruit issues with "good first issue". Start there and filter down as required. https://github.com/search?q=label%3A%22good+first+issue%22&t...
Work for a defense company. There's a lot of hardcore engineering there.
Just learn leetcode and join faang position about these things.