What does an ML Engineer do?

> I was shocked to learn that my new employer has a very different job definition for ML Engineer than the one I was familiar with.

Job titles do not have consistent meaning across regions or industries. People use buzzwords and hype to recruit funding and talent.

You've learned an important lesson: use interviews to gather information about companies and teams. "Can you describe in broad strokes a typical project for this role?" is a perfectly reasonable question to ask a hiring manager.

If it were my business or my team I would want an ML Engineer to relentlessly knock down any barriers to getting models into production.

The deployment of ML is still per-paradigmatic in that people today deploy products based on luck, blood, sweat and tears. Many projects fail and many more never get started.

I am sure that somebody somewhere had a team that gets results consistently, I can partially picture what it would take to do so, but that's not the usual condition. (That team is so productive that it doesn't need to hire more people!)

For instance, you might not want to do any of these things, but any of them could be essential to putting something in production:

   * manually classifying 10,000 images or documents
   * figuring out how to run a program in the Java and C++ debuggers at the same time
   * be able to convince somebody that their research programme is a dead end
   * survive a 40 minute build process
   * push back against a 40 minute build process and reform it
   * reproduce a result from a scientific paper
   * get a "data scientist" (who thinks he is too good to listen to you,  or even to the CEO) to standardize their python distribution so their work isn't wasted
   * find the scalability bottleneck in a system,  where it is,  not where you want it to be.

Many teams fail at ML because there was an essential task that nobody wanted to do that didn't get done.

People who do these things today will be able to say tomorrow that they were part of a successful team. Other people will expect great praise because they diddled around with Word2Vec for a summer.

Some day the social structures will be in place and they might have a role made for you that will make you productive but for now you must ask: who do you want to be?

My subjective view coincides with your new employer. I think that's pretty common.

ML Scientist and Data Scientist versus ML Engineer and Data Engineer.

Sounds like their title for the job should’ve been ML Infra or ML DevOps engineer to indicate it’s a supporting role versus a primary practitioner. Annoying, sorry OP.

I'm on the same boat, but about SRE.

Basically every company have their own definition. For some companies, a SRE is just a traditional software engineer who happens to know how to use Kubernetes, where for others, a SRE is a "modern systems guy", with focus on IaC and automation.

Guess you're experiencing the same but regarding ML.

Designing the systems that run the 'ML pipeline', or end to end data collection > train -> deploy -> infer -> improve. I don't see 'ML Engineers' as advancing the state of the art but they should be fluent in recent techniques, know how to apply them, optimize them, know hardware is needed to efficiently train, etc.

I think it often depends on how large the team is. On a team with many developers per model, ML Engineer to me means someone who does the engineering to support models and is distinct from a researcher or a scientist. ML engineers will not do the science side of the equation (model architectures, optimizing scientific metrics like accuracy) except for when their engineering work impacts the science side (e.g. figuring out how to optimize model throughput without impacting scientific metrics).

On smaller teams, the titles mean less - but I rarely see ML engineer positions for small teams.

ML involves both science and engineering. ML engineer by definition sits on the engineering side of the fence, but how much of the science they are involved in will vary by company.

How did you get through a job description and the interview process without this coming up? Was it a bait and switch?

It may be definition differences between the two companies but here's the first thing that came to mind: by any chance is your new employer a much larger organization? In smaller companies you get to wear many hats and typically have responsibility over a wider stretch of . In larger companies things tend to be much more siloed. So in a larger organization, the things you did in your previous position would likely be spread across multiple positions and teams.

My understanding is ML engineer gets machine learning models running in production. They would own the infrastructure that enables teams to deploy ML models, interface with engineering teams to agree on contracts/endpoints/UI/etc, and work on scaling models when/if that time comes. I would view Data Scientist and ML engineer as distinct sister roles, and not as a subset.

I've worked in environments where ML engineers were "data science plus", and those were really unhealthy work environments - it ended up feeling like a prestige war with lots of cookie licking and turfing. When you're at the point of needing differentiated roles, you really need someone to focus on infrastructure to deploy ML models and someone else to focus on managing stakeholders, wrangling data, and developing proof of concepts.

At our company, ML Engineers do both the modeling work and the larger engineering that allows those models to interact with the product. So all the models are built by ML engineers, as well as a lot of the production data engineering that gets any needed data to and from the models in production.

I would say, some combinations of:

- Helping with feature engineering at scale

- Putting models into production

- All of the environmental stuff (Framework versions, security patching, etc)

- Monitoring models and model drift

- Supporting hot/swapping models with zero downtime

- Model persistence, A/B testing and evaluation in production

- Distributed hyperparameter tuning

I work for a company where half the company consists of ML Developers. They match your definition. Your job is what we call ML Ops.

Kind of interested how you got hired to a completely different role though and how that didn't come up in the interview.

In my experience, if you have a PhD you can get a job working with the models. It will likely be called something fancier.

Without the PhD, I feel like you're just a data jockey. Just clean the data and create APIs for the PhDs and consumers to use.

Most of the time they allow the company to say they work on this new flashing buzzword tech while burning through capital doing stuff a few if else statements could easily do

>... never works on machine learning directly.

Meaning, does not push the train button and watch loss curves?

What's out of this role's scope at the new company?

I think it depends on the company. In my experience most companies have it halfway between the things you described (some data engineering + some modeling). Companies with less established infra generally have more data engineering work.

Imo unless you are a rich company or have a well funded research arm, it mostly seems wasteful to make someone's job pure modeling.

We call that role ML Ops at our work. It's a mix of data eng and dev ops.

I'd say what you've mentioned are the 2 typical definitions.

Either its an expert at building and training ml models, or its an expert at putting ml models in production.

Yeah I think in general “ML engineers” are building models (can include feature engineering) and then deploying them into production.

Its whatever you want it to be, which is the greatest thing.

Ex ML Engineer at a company.