Why do product devs switch into data eng/data ops roles?

In data eng roles I've had to build systems that ingest tons of data quickly and learnt multithreading, used profilers to fine tune perf, learnt how SQL databases work under the hood and tricked query planners, learnt how to structure map reduce style jobs to exploit parallelism on huge clusters. Admittedly this was a few years ago when the field was more nascent. Maybe today it's less engineering heavy if you can get away with gluing together a few off the shelf solutions like Kafka/Snowflake.

In SWE roles I've had occasional deep dives into tech, but a lot of it is about knowing the business logic and delivering simple things quickly. It's like write SQL query, maybe change a model, write some simple business logic in a hairy codebase. Since it's simple, there is more pressure to deliver quickly as well. Sometimes doing things the “right” way would have been more interesting but also more time consuming, so we decide to just put a bandaid over it. It feels like there is less agency as well, just implementing whatever features/roadmap the PMs hand down.

But I see other people on this thread have had the exact opposite experience.. so I guess it really just comes down the particular role, company and your interests.

I moved into data engineering because there were a lot of interesting problems to solve for our company.

We wanted to move from reporting on a daily granularity, with a 24 hour lag, to being able to support a minute granularity, with a 5 minute lag.

It was challenging, fun, and there was plenty of engineering fundamentals involved, just a different kind.

The work is often dull. Often it's ETL plumbing disparate historical systems together to generate reports, maybe some machine learning, sorry I should say "AI" \s. There's certainly some challenging engineering though, especially distributed systems challenges, some caused by the old historical systems you need to integrate to get data out reliably and correctly.

Unless you are working on greenfield projects I'd say the only reason is 1) It interests you 2) It pays more

I'm dying to find a non data engineering job but they all come with 30-50% pay cut.

(1) Problems are interesting for builders

(2) Industry still feels nascent; mission of self-service still hasn't yet been proven

(3) Your users are other software engineers, analysts, and data scientists. So opinionated power-users.

Happy to go more in-depth about this, I've helped several folks make the transition: beta.senpai.so/ataki12

Endlessly re-learning JavaScript frameworks and CSS gets very boring after awhile.

Data engineering roles and backend development have a lot of overlap and sometimes they're basically different titles for the same role. So for some people it really isn't that big of a career change.

SWE roles and you are taking webdev as examples? What are the solid engineering fundementals of legions of people doing CRUD development in React and Vue for example? All of your arguments can be turned around if we replace data engineering with webdev.

I switched from a fullstack role to data engineering because I wanted to dig deeper into how to process streams (as an example) and build something that other engineers, and through the use of dashboards rest of the company, rely upon and in that way 10x myself. But also because I wanted to see if I can create the need for a machine learning engineer role at my company since that is what I would like to try out next.

Because the business doesn't have a clue what they need, but there's a chance other engineers putting in requests to you have a pretty good idea of what they need.

pure backend/data roles are interesting because when building infra for multiple consumers, you can give many teams leverage over their problems. It’s like building a great tool they didn’t have before. For some people this feels immensely satisfying. Especially when they get immediate feedback from fellow devs.

In search, for example, I’ve seen teams turn Elasticsearch reindexing from a task that takes weeks down to hours. So many teams can build on that and it unlocks tremendous abilities for them to even care about improving search.

> Data roles heavily focus on particular enterprise products (Kafka, Redshift, Snowflake) at the expense of engineering fundamentals.

I’ve heard the opposite, ha! That “App dev roles rely on just prebuilt web frameworks and canned RDMSs, not the breadth of the full menu of fundamental problems out there” :-p.

In reality there’s a huge menu of databases and data processing systems that backend/data roles play with from spark, flink, Kafka, analytical systems and so on…

It could be that you are still a product engineer, it's just that your customer becomes other engineers in your company and your impact and experience can be magnified. As you gain experience you become more capable of successfully delivering products that other engineers rely on.