How's the current state of hiring in the LLM field?

I'm a DS with ~10 yoe mostly in classical ML and I've been wondering how much/how fast I need to pick up "high level" LLM skills. I've seen the way the winds are blowing wrt to the DS job title and so have been moving towards more of engineering roles. But it seems like every job posting right now for ML engineering positions is asking for "pytorch, DL, LLM" experience, even for companies where getting into the nuts and bolts of inference would not really be a core value prop. Like I really find it hard to believe that financial/insurance companies (my primary domain) are building and fine tuning their own models for chat/RAG applications or whatever (or if they are, should they be?). I haven't really seen huge value add from even the most sophisticated RAG setups yet.

I use LLMs every day for code assistance so I'm not totally naive to the whole thing, but I seriously wonder what direction to take my career, exactly. Should I be going all-in on learning CUDA? Is knowing how to wire up a few API calls enough for "soft" applications? Should I in fact be keeping up with the latest RAG techniques (which seem to change every time a new foundational model gets released)? Should I just stay out of the whole thing and double down on classical ML applications in my domain?

The work seems to be exclusively about using openAI apis and hacking response pipelines together, but the hiring processes are focused on getting ML specialists.

So yea, it’s weird

If you find LLMs interesting, then by all means. But you shouldn't focus on LLMs as a path to career success.

In most cases business just want to use an LLM but don't need any expertise in how they work. 80% of use cases are "I want to write an English sentence to make a SQL query". With a little bit of fine tuning or RAG you get there.

Being a data scientist who understands LLMs would be far more likely to be a good career move.

As an outsider (offering prototyping project help to clients), the main thing I am seeing at the moment is AI-knowledge experts as project managers. As in: a company wants to go about an AI project, and needs someone who can guide them through the jungle of tools (services, budgeting, operating costs) as well as figuring out potential partners (individual developers to AI-integration companies). People on the inside of these companies aren't trained 'enough' to fully grasp the status quo, what's possible or not possible, where the risks currently are and where it is all heading, despite being super eager to be involved on those things. That's at least what I am hearing, less actual AI-development work, more "how to get AI into our business" work. Just my 2cts anyway.

> Is it worth entering the field today starting from scratch?

Here's a personal anecdote: I'm only really good / competent in technical areas that truly interest me.

In technical areas that I've pursued only for career / money reasons, my performance has been sub-par. Especially when compared to that of developers who are truly interested.

I don't really know how much my experience generalizes to other developers. Maybe it's related to my ADD.

Looks like the line between Data Scientist, ML Engineer, AI Engineer are blurred with the onset of LLMs/GenAI (especially if you were previously working on classical NLP). Absolutely worth entering the domain, it's still day 0, but be prepared to keep yourself updated, unlearn things fast and pick up new tricks.

We have open positions for putting LLMs to use (won't link, not a work account). Seems difficult to hire people with sufficient knowledge of our niche and of LLMs.

Whether they will continue to grow is hard to say. Putting open models to productive use is difficult. Like stable diffusion can make a pretty image but can it make the image you want? For example, can it make a 2d video game sprite, can it make the same character consistently in different poses.

That's a personal project problem I have but it is the same vibe as LLMs with code. Like they can produce some code, but a more complex algorithm or larger program is tough. I'm not sure if there will be an upper limit on what they can do. Right now it is a lot of engineering to get them to do stuff with no human in the loop.

Not a lot of companies are training models. There are probably a number that are trying to integrate OpenAI's API in some way.

They can do powerful things and it is a good skill to learn, but it feels like the market right now is for PhDs from top schools or low paid data labelers. Maybe others will have different experience.

For resources, check out [1], [2], and [3]. The third being my least favorite, but others like it.

1. https://karpathy.ai/zero-to-hero.html

2. https://deeplearning.ai

3. https://fast.ai

For every 1000 prompt engineers who are doing little more than hooking up an API to a front end and a bit of prompt engineering, there's really only 1 decent engineer who can put together things like RAG and fine tuning properly.

I'm yet to see a truly strong LLM application where the results are not significantly better than a smart person with chatGPT open and a bit of patience to craft some prompts.

What are the most impressive things available that are not just chatGPT?

Same problem as data science jobs have had for the last decade:

Job requirements are for applied researchers.

Job work is Excel/OpenAI wrangling.

Some people are promoting the title "AI Engineer", which I think a sensible title... kind of like "full stack developer", not defined by a single technology, but requires knowing a number of technologies (LLMs especially) and being able to put those together into a working product.

Having worked with LLMs a lot I would NOT agree with people who call them "just another tool". Some people are experts working with cryptography, or creating scalable distributed systems, or working on network protocols, etc... these are real and specialized skills. Working with LLMs is similar. Starting from a foundation of "can get stuff working" (full stack) is awfully important, and I don't think LLMs reward narrow specialization.

Also prompt engineering is real, and I don't think it's going anywhere. Working in collaboration with domain experts to do that prompting is essential, but a lot of output issues benefit from a combination of both prompting and changes to pipeline, text processing, and other code-based approaches. There's real benefit to being good with words, a close reader, able to get in the head of the LLM, learning its fixations and misconceptions, templating thought processes, etc. Ideas that prompt engineering will disappear with better models and fine tuning are, IMHO, naive and misunderstand the interplay of prompt and LLM. If you want to get something from an LLM you will still need to know how to ask!

I don't think there is any real demand for high end LLM researchers/mathematitions etc. who would really build LLMs.

On the other hand, only a small amount of companies have the money, interest to actually at least finetune LLMs. They have their people already in place.

Than you have people who should integrate LLMs and tbh thats something everyone can do who was able to integrate any other api. I don't think there will be middle class companies looking for LLM experts.

Unless you're talking about research, LLM's are considered a tool, not a new field.

I have mostly worked in creating automation tools in Python(scrapers, bots, etc) and since the inception of chatGPT I have tried to use LLMs in existing solutions; obviously for automation purposes. Like recently I integrated a few customGPTs with web applications as currently GPTs do not support monetization nor you can distribute to others(unless the other person have a GPT4 account). If you opt for this route then there is definitely a need. Frameworks like CrewAI and langchain agents are going to help businesses to automate their internal processes.

In the broader field of machine learning (I would leave Artificial Intelligence for the marketers) don't think there are as many jobs as much there's hype.

A handful of companies are doing LLMs. Rest are doing internal fine tuning of available models.

LLMs have proved themselves not to be that reliable that they can be left in production alone.

Hence is the adoption. Limited to areas where they don't have significant impact such as "Change the tone , rewrite, summarize" context menus in many products.

Are you asking about creating LLMs or using LLMs to create apps?

The growth and opportunities will be at the application layer. Training and fine-tuning LLMs doesn't really have that much value. Closed source (OpenAI) models and Open source models will be the foundation. As I said running some training/fine-tuning on top of these models and knowing how to do evaluations wont have that much value. Learn how to build useful applications on top of these things instead.

My first full time role was to build chat bots using RASA, Dialogflow and some other similar tools. Now the past year or so, it has largely become a GenAI role, making bots, and other exciting tools. So far things are kind of great in my current role, albeit people tend to have unrealistic expectations of these models.

Hiring in the LLM field varies, but there are opportunities if you're interested in law. It's hard to say if it'll boom like data science. But if you love law and are ready to learn, it could be worth it.

There is certain to be good money made in applying all this to business use cases.

…but not sure that necessarily makes it a good foundation for a CS career. Especially with much of this moving towards almost no code prompting territory

“Do you think they will grow in the coming years like it did for data scientists?”

Maybe. But then the demand might also shrink quickly after a couple of years. Is it worth that risk? Only you can decide.

AI Engineer is going to become about as descriptive as "full stack engineer"