HACKER Q&A
📣 quokkafriend

Steps from Analyst to Data Engineering?


For someone who has strong SQL skills and some scripting experience, what would be the best approach to shifting into data engineering?

A constraint is that current place of employment does not offer transfer or mentorship opportunities.

If you were to recommend the top 2-3 actions to take to enter the field and gain employment, what would they be? (self-study resources, courses, projects, bootcamps)


  👤 claytonjy Accepted Answer ✓
I took a similar route, albeit much less intentionally and spanning almost a decade. Software QA -> Business Intelligence Analyst -> Data Scientist -> Data Engineer.

Here what I'd recommend today:

1. get very comfortable with Python. Scripting isn't enough, you'll need good OO principles, understand how to manage projects/libraries/dependencies, etc. This will take the longest, so start it first.

2. Read and re-read Designing Data-Intensive Applications by Kleppmann. This is the bible of data engineering and far outclasses anything else currently available.

3. Get your hands dirty with modern tools and the whole data lifecycle. DBT, Airflow, Snowflake, Postgres should be obvious (feel free to substitute prefect, clickhouse, etc. if desired). You'll also want familiarity with a cloud stack and how to manage it (terraform, pulumi, or CDK). A public portfolio project would be great, but being able to talk confidently about the how and why of these things is probably enough.

The hard part is getting that next job. Look for junior roles at big companies, and mid-level roles at startups who don't understand the data ecosystem yet (almost any startup whose product is not ML or ELT). The former will give more mentorship, the latter will be easier to get if you can talk the talk in an interview.


👤 jstx1
Are you doing any data engineering at you current job? Anything you can lean into more? My first data analyst job was essentially a mistitled data engineering job.

One challenge that you need to overcome is that many data engineering roles require experience with specific tools and that junior positions aren't quite as common.

I also think that you should consider backend software development as an option. It's a similar role (depending on the type of DE role you compare to) but with better opportunities for progression and more career flexibility; it's also been around much longer than DE which is a relatively new job title. Another weird thing is that some DE roles have lower status in some companies. Especially if you get to work with data scientists, there's sometimes the perception that DS are the smart people and DE are the grunts that do the plumbing. The whole thing is obviously very misplaced but it does exist in some places and you should be aware of it.


👤 giantg2
If you already have experience, just apply.