But, I now understand the problem. At my organisation we have only 2 DBAs, and we can't give all of our time to this task.
To be a DBA is an endless battle against the entropy some devs try to create. The data model should really always be validated by someone with more knowledge and experience. Also, some young devs come up with crazy ideas (for instance, we should not use foreign keys). Others want to load insane amounts of data to process in the database, instead of preprocessing, or using a staging database. There is a continuous stream of bad ideas being implemented.
And when a bad idea gets implemented, it is really hard to undo it and it tends to just create more chaos: you need materialised views to get around bad modelling, or weird views to compensate for duplicate data, etc.
So, yes, a DBA is a really important role. With regards to it being a good job, it depends if your company takes it seriously and puts you in the development process, otherwise you'll be in for a lot of stress.
I would argue that it's way more accurate to say this role is actually outsourced in most organizations with all the tradeoffs that come with that strategy. I begged my former employer to subscribe to a good PostgreSQL support company. Guess what the one we picked is constantly looking to hire new DBA to sustain growth...
I was in database engineering/administration position for the last 10 year on a small company. It was exhausting because management wouldn't care to rely one guy to manage the whole data stack. It was rewarding because the CEO saw me as probably smarter than I was because I knew SQL well enough to produce outstanding value as a single individual.
Often reading HackerNews I felt like an imposter because NoSQL was the new hype and database was seen as a thing of the past... well trends reverse... and people will always need reliable systems to handle business critical data.
So maybe in a few year after few major Cloud data horror story DBA will become the new trend. No one know.
But learning how to handle data? That will always be handy no matter the underlying technology.
It is astonishing how much a poorly tuned data request / SQL, or poorly designed data model / table, can impact performance by multiple orders of magnitude. It is also fascinating how much addition of specific index or tuning statistics can help execution. There comes a scale where throwing more hardware on performance problem is prohibitive over hiring a good DBA, as much as current thinking is that "hardware is cheap". But on a regular basis I'm seeing experienced, serious, dedicated developers create SQL that is functional correct and reads 10 billion rows into buffer in order to spit out a single row answer (sometimes of course that's necessary, but usually not :)
He got let go for basically having a really bad temper, and we decided not to look for a new DBA. I was sad he was gone but also glad to not be getting screamed at. It feels silly, but I've got really unresolved and mixed feelings about the whole thing. I genuinely really liked the guy, loved talking music and movies with him, but he would heel turn in an instant and just ruin your day.
Honestly wish we had a DBA, I ended up picking up a lot of his responsibilities. I'd love to have someone I could learn from. Basically, I'm getting real sick of optimizing reports.
It's important to understand why it existed in the first place. A long time ago, if you were building business software on a database at all, it was a very costly commercial database. Imagine 6 figures a year for the software licenses, the same amount again for a support contract, and a similar amount for the hardware to run it on. If a company is spending $1M+ a year on a critical piece of software infrastructure, that is very complicated in addition to being very expensive, they will usually be very open to spending more on top of that in salary to employ experts to look after it. Those experts are your DBAs.
A funny thing happened when MySQL and Postgres made databases "free": companies (irrationally, IMHO) stopped being willing to pay for experts to run their database software. That duty became a secondary responsibility of developers or operators or both, none of whom really wanted to do it or cared to learn how to do it well. There are counterexamples, sure. Some companies get big enough, and lean on their databases hard enough, that they cultivate true competence in them, but it's rare.
And then along comes AWS RDS (and similar), automating away the most tangible aspects of what DBAs did: HA, backup/restore, configuration, provisioning. There's still plenty left for DBAs to do, but it's stuff people don't understand the value of. As above, the residual DBA duties silently fall to developers and operators.
Also, being able to run databases for $0 in licensing on small-ish, commodity servers has led us to use more and more of them, rather than giant, central, shared databases. This amplifies all of the above effects. It also has important 2nd-order effects on the social organization of our software-building endeavors. When databases were shared by many apps, it made sense to think of them as their own services. When each database is built and run to service exactly one app, that makes less sense, plus it's also easier to push everything onto the devs (as above).
Make no mistake, all these companies still need DBAs. Their schema designs are awful, their query performance is worse, who knows if their backups work, etc etc etc. But we don't see it, because there are so few people working in software today that understand what a DBA is, or ever met (and worked with) a good one.
It's rewarding work if you can get it. I still love doing it. But outside of the Oracle and Microsoft ecosystems, it's not really a career.
So I hope you find from other people that this is still a good role to pursue and that you do become a dba and work as a specialist with highly skilled teams and make the code world a better place.
That being said, that’s a startup opinion. There are lots of F500s that will want them in their data centers. So like cobol programmers, I don’t think there will be a problem getting employment somewhere in the foreseeable future.
Personally, I'd take a proper DBA over modern data engineers any day - there's a ton of value in knowing how to set up, maintain and use an effective SQL database. It's just not a place you find shiny new tech, so people regard it as boring and old. I'm biased though - the beginning of my career I was a DBA in a boring industry, but it taught me A TON that still has an impact on my skill set 30 years later even though I'm now working in totally different areas.
I haven't seen as many DBAs in the last 15 years. In that time, two things have happened:
1. Servers have become faster (obviously). I spent a lot of time in the late 90s and early 2000s optimising databases, from the data model and query plans to physical hardware. We forget how slow hard drives were, and how challenging a database cluster is to run on 4GB of RAM. These days we don't spend much time worrying about database scalability for most cases. There was a time where many businesses were pushing the upper limits of database server performance for hardware available at the time.
2. SQL databases have become less central to the solution. Everything used to be modelled in third normal form and the SQL database was the 'single version of the truth'. Now we may put some data in SQL, some in a document database, and some on a queue to be saved in a datalake.
If you are asking specifically about SQL DBAs then yes, the career is fading. However, as others on this thread have pointed out, there is always a career for people who love data, and know how to work with it.
I would likely not suggest a "DBA" as a career, but it is a very valuable skill set... To get a job at anywhere but a large business... I feel you need one other hat at least.
Drop me a line: vignesh@ or reply here.
I think just in general data engineer is a better term to find the same role across a lot of companies today.
Major products are released and operating right now with incredibly low performant queries and uses of databases. The infrastructure people and services keep the databases running.
Even at large companies, you'll often see databases being run by "data engineering" and SRE teams rather than pure DBA teams. This isn't just a naming difference: the background and mindset expected in these roles is engineering, not system administration.
You're going to be most likely to find traditional DBA roles at companies that run a lot of legacy on-premise software: banks, defense contractors, government jobs, older large tech companies like IBM and HP, etc.
I don’t plan to continue in administration work when I move on from my current role—partly because I’ve found the development to be more fun, and partly because I’m seeing the writing on the wall.
From a database developer's point of view, I have always admired the work of DBA, although I can develop the kernel of the database, I can also write a SQL engine, never been good at writing complex and efficient SQL, totally different skills.
I have noticed a trend in the database field in recent years: Everything goes to the cloud.
Because of cloud is more and more popular, and the cloud is providing fully managed database service, it is hard to say the trend of traditional DBA job is getting less and less, the DBA expert is still extremely popular, However DBA will alive and can be more welcome, because there are more and more new technologies are evolving fast, just like what we're proudly building: TiDB, an HTAP database provides SQL interface, support complex analytic workloads, as well as the OLTP load. DBA could reuse their skills, and even better, they are not just DBA, they can also act as a data expert, help application developers to optimize SQL queries, and schema design, and help them to design and optimize systems.
A good example is my recent little side project: https://ossinsight.io/, although the architecture is simple, it contains some complex SQL in there, I turned to my DBA friends to get this application done correctly and efficiently.
As others have pointed out, traditional DBA roles are fading away due to cloud offerings abstracting many of the problems DBA’s would otherwise be responsible for. Hardware is cheaper and more available which means that companies throw more RAM/CPU at their problems, whilst areas like index planning, performance tuning, query optimisation, and resource monitoring take a back seat. In our experience, this eventually catches up to you in the form of poor performance and unhappy customers.
We believe that tech can augment most of the insights offered by a traditional DBA and help avoid these pains early on, especially for smaller teams with limited database expertise. We’re currently in private beta, but will be opening up more broadly in the coming weeks. We’d love to get your feedback on our approach - sign up for the beta or AMA if you have any questions!
* Developer Relations Engineer: https://jobs.lever.co/ottertune/486e1fdb-ceb6-44d4-872f-c66e...
This is a super important position to us, so you can email me directly if you want to discuss: pavlo@ottertune.com
I’ve worked at 5 small companies in the last 15 years, and freelanced for a few years. The dev team size was: (15, 5, 5, 4, 8). None of those companies had a DBA. One company we had a full time IT/sysadmin, but otherwise that role was filled by people with a title like “software engineer”.
In truth, I think we often would have benefited from the expertise of a DBA, but we never made the decision to hire someone for that role full time.
And all the move to cloud is not straight too , there are lift and shifts where the same DB runs on another machine. There are lot of "microservices" that data from these that need housekeeping that will go on anyway.
How about NoSQL DBA's ? We moved a lot of data to a NoSQL datastore (HBASE) and added Phoenix on top of it (SQL flavour). Sure enough there were app queries that did need a relook at! , newer query patterns emerged , secondary indices , data re-indexing , adding more RegionServers , Splitting regions , tuning guideposts, changing the GC parameters for the stop the world GC's was all needed. While a lot of these are taken care in PaaS , not every application runs on CosmosDB , DynamoDB or the like. There is always a niche and it is here to stay!
However for some orgs this becomes a brick wall they eventually smash into.
The cloud vendors now provide moderately ok database offerings on a variety of virtual hardware that allows you hide a lot sub optimal data modelling, index usage and inefficient data modelling.
Any database workload is performant while all the data fits in ram.
I agree with some other comments that there is a bit of a shift left approach to the DBA role now. I find that dedicated DBA skill sets are somewhat useful up until a point then they struggle to have context of the application stack beyond.
If you want a solid DBA career you may want to look at a SRE track/pathway that includes some deep focus on databases.
You will want to know the how various index types work, when and why to use them. Data types. Intricacies of various database mechanics that impact performance and scaling workloads. Solid foundations in highly available system design, maintenance and operation. A willingness to be on-call. A willingness to write code to a software engineer level or send in changes to the dev team/s to help optimise queries, drivers, connection pool settings/timeouts or ORM usage (shudder).
I think the days of a pure DBA role are coming to a close in technology lead organisations. Maybe not in banking/insurance or other orgs where DBAs found a home to sit in the back office doing minute query tuning on a reporting query etc.
If more developers have read and understood the content in https://use-the-index-luke.com/ then there would be less need for DBAs.
Our "customer" is the developer or cloud system architect.
Today I think a "DBA" in a small/middle sized company would be also touching cloud resources and data funneled through analytics and other sources. Basically the "big data" fad we had a while ago becoming just "data", and that data has to be administered, optimized, and backend/frontend devs also need guidance and education on how to best deal with it.
In general, I have the feeling "administrator" roles are disappearing to be either more specific in the biggest orgs, or way broader in the middle sizes/run of the mill orgs.
In the meanwhile, there will be somewhat new framework that makes DBAs life easier and bring a broader entrance for anyone who want to manage their database well.
Having said all that, I could see it would be hard to decide right now, as a new IT person, to get into being a DBA. We wear so many hats, and are generally looked to for anything that needs to be done. As others have stated, a lot of companies don't seem to care that their queries are absolute shit, they just throw more hardware at the issue.
1) Maintenance - rebuilding indexes, managing disks 2) Performance - identifying problems, training devs 3) Architecture - helping people change DB architecture at scale 4) Automation - DB-as-code and automatic migrations
Some of these go away or reduce with the cloud but an awful lot of companies don't or can't use the cloud. Even if we theoretically could, we might have a large existing database on a live system that would not be quick to migrate to the cloud (the copy process would take long enough) only to find out later that there is some issue with performance or pricing, so we leave it where it is on-prem.
However, if you have expertise in specific databases, start a consulting company. That can be a very lucrative job, and you won't be tied to your employer's mismanagement - I've seen numerous times where a company will hire more managers and fire engineers (or cut the costs in hardware) due to increased costs by those managers. And they end up badly pretty quickly.
The whole "shift-left" movement seems to have swept DBAs along with it. Good to know yer DBs but it might be better to expand your offerings.
I would suggest to keep your highly specialized database skill and invest in infrastructure as code methodologies (e.g. terraform) so that you can be the enabler/trainer for other teams on how to setup/upgrade their database instances and also help them with database migrations.
So I would say you can transition to a highly specialized consultant (maybe in house) on how to optimize a database to keep your skill and learn infrastructure as code skills.
I think DBA will finally transform to DA: Data Architect, Data Admin, Data Analyst. As for Data'Base' Administration, there are going to be more and more tools, SaaS, and PaaS to eliminate thosea dmin work...
There are people working in data analytics / warehousing / "big data" but I don't think that's the same as a traditional DBA role where you're smashing out SQL.
I suspect DBAs are still around in older enterprises or maybe large companies with very complex databases - but I don't think you'd see many in modern software development.
Do you like the work? Do you want to work in large orgs? Keep doing it! If you don’t, there’s plenty of room to branch out and I assure you if you’re half decent at the job your insights will translate well to many other meaningful contributions.
- How do you define good job?
- You assume it was a good job by using still. Why was it ever a good job?
- What is your age?
- How long do you plan to stay in the industry before retiring?
- Are you passionate about it?
- Do you enjoy it?
- What do you want to achieve?
- What pay you expect?
- How much money you need for your lifestyle?
I could go on forever but in order to answer your own question (if it was not asked just to promote your business cloudsql or whatever) your need to answer these first.
But the job title as such is dead as a doornail -- along with "systems administrator", "webmaster", etc. You need to call yourself a "data engineer", "data warehouse architecht" or something similar.
It's silly, yes. But this is a silly industry.
I’m looking for a DBA - MariaDB (galera cluster) and TiDB.
Contact info on my profile.
I'm not saying this is correct or better, just saying that's the way it is and it does set a precedent and model for other companies.
Olso there isn't a need for DBA if you are using other storage strategies.
But in many, many cases there is going to be a large OLAP or even OLTP DB out there, in dire need for someone to love her and take care of her.
Just call yourself sysops or database engineer depending on if you actually want to maintain DBMS or write sql. Or come up with dbops or sqlops lol. Everyone else is doing it.
DB is a huge system right now, a functional ones has multiple subsystems, possibly handled by different teams.
Can't really see what a traditional DBA fits into this.
I blame a lot of performance and scaling problems on ORMS which are fundamentally a bad abstraction for sql and enable devs to brute force something that "works" instead of dinging an elegant and high performance solution to a problem. Most devs don't really understand how to use case statement or aggregate queries for instance. When i was building out analytics pipeline, my original vision was to use some sort of elastic search. We're scrappy and bootstrapping though so instead I came up with a set of triggers that populated special tables that pre-crunch certain bits of data from the original source. Between that and judiculous use of indexing, we are able to do realtime queries with no delay. If you know how to layout your information correctly, the database can be more than fast enough for most use cases.
I think everyone working on database abstraction libraries should look at Ecto. It doesn't try to hide sql. instead it embraces the semantics of sql and extends it to make it more ergonomic. It become easy to escape hatch into raw sql for things the libary doesn't support and even better, I don't have some werid under the hood transformation going on that messes with my intuition for the sql being generated.
tldr: stop treating sql as a glorified k/v store. you can do a lot of cool stuff inside the db and benefit from a crazy speed boost
Databases are such an important part of modern systems it makes sense to have an expert on them on board. I think that would pay off many time over.
However, I'm in a small team and I think it's important that everyone can also help out with a little bit of everything depending on what's going on.
They come with different skillsets and sadly, some very different competences[1] Ignore the latter point and you'll think of them as equal therefore interchangeable and start outsourcing to save money. Ok, buy cheap pay twice, or more likely, much more.
[1] My replacement at my last job didn't seem to understand MSSQL server even had a query optimiser. His knowledge was 20+ years obsolete at least.
Probably not.