What would you do for developer experience

Before you go changing anything, I'd go talk to all the people who are there - understand where they came from, how the systems got to where they are, why they made their decisions, what they like and dislike, and what their biggest complaints are as well as what they love.

Then look at solving their actual problems. Because if you just come in and declare how you want it to be, you aren't improving their experience, you are just inflicting your own preferences on an existing ecosystem.

Dev experience is about people, not tech. You will use tech to improve it, but your devs are your customers - talk to them.

Are you a principal eng? The principal engs I’ve known express things in terms of measurable business problems. You have a list of practices with no success criteria.

My context has been startups with engineering teams of 15-60 engineers.

In general, I've found CI/CD templates, IAC templates and app templates to be an anti-pattern. It goes to the heart of scalable vs replicable.

> Put simply: something replicable can be copy-pasted (with variations as needed) to grow impact linearly in relation to effort and cost. Something scalable can create impact at a rate that increases faster than the rate at which your effort and costs increase.

In the contexts I've worked in, templates enabled us to rapidly grow and expand our business and technology in unmaintainable ways.

We scaled to 100 services, but our revenue and headcount didn't scale linearly to support the services over time.

---

In general, I lean towards solutions that facilitate distributing and updating templates over solutions that facilitate copying/pasting templates.

- Kubernetes this meant a common Helm service chart for the organization -- Now we could distribute and update kubernetes manifests to the org - CICD this meant building plugins that we distributed to teams - Application templates I've historically moved folks back into monoliths but the microservice chassis pattern looks like a solution here -- https://microservices.io/patterns/microservice-chassis.html

Your calculus might look different depending on your internal capabilities and resources. Good luck!

I'd basically first audit the entire infrastructure. Doing this will tell you about other problems (if any), so that you can take proper decisions.

Second I'd also talk to my customers - ie the developers on the challenges, pain points they are facing.

Next - you'd have two options. One is to build something (semi-automated or fully automated) or write scripts to achieve those tasks. Or you can consider buying an Internal Developer Platform. There are a lot of solutions out there that can solve for your potential needs. The answer depends on whether you have the internal expertise and bandwidth to do something like this. If not, buying an IDP would be a better idea.

> cloud is not meaningfully leveraged

I would very much warn about this. "Leveraging" cloud is a massive double-edged sword.

I've been in places that "leveraged" cloud. In practice this meant that every application had "gaps" filled in by cloud-only services that are impossible to run locally, can have their own state separate from application state and are a black box when things go wrong.

There is great value in being able to run your entire stack locally and having superuser access on all the parts. You may however not see that value until "spooky action at a distance" starts happening and the lack of access/local reproductibility impedes troubleshooting.

Most of the developers and managers won't know what change needs to happen. If they did, they would have done it, and you wouldn't have been hired. They can give you insight into the lay of the land, but you'll get nowhere by considering everyone's prescription; the majority of them will be wrong and a minority will be right but you'll have little way to tell which is which.

Whether or not metrics will capture it, the efficiency of the organization can best be increased by centralizing the teams around one common toolkit to cover most use cases. You'll have to be judicious in deciding which cases can be exempted.

No matter what you provide for these teams, they will tend to stick to what they know unless they are forced to switch. Some can probably be persuaded, but your success will likely depend on having the backing to force them to switch one by one. The first ones will be the most difficult and then you can gain momentum.

Use your metrics cynically to this end. This all probably sounds rather dark, but leadership is tasked with leading, and inefficiency and disorder will rule the day until someone steps in and leads.

It depends a lot on company culture but ideally talk to a decent sample size of the employees across teams and find their perceived pain points. Then measure things: time to build locally, time for CI, time for tests to run, release frequency. Gather the combined data from measuring objective things and developers subjective experience and work it out from there.

If your role crosses over with SRE and platform then you need to get the data for these things to.

Make sure you understand why X is like it is for all X that you want to change. Know the history and the battles won and lost that got it like that.

Move slow and fix things!

Of course if the culture is all “quarterly goals, show progress etc.” then you may need to adapt for that too. Maybe get some quick win projects out that are no brainer things. Parallelize tests or project master branches etc.

By the way your list of changes might be perfect or awful. It is like a diet plan… it depends on the person and the reason for it.

This is a large task list, and many of these things will require a lot of time and effort to implement properly. Especially given the language spread, where a different solution is often required for each language, you'll need to prioritize effectively to make effective positive change (not easy, but important).

Those changes should ideally make developers' jobs easier, but it should also improve the security and quality of the systems. The priorities depend on not just what developers see as friction, but also what's important to the business.

You don't always get to make everyone happy, but one example at my company is that we implemented [OpenID Connect](https://docs.github.com/en/actions/deployment/security-harde...) with our GitHub Actions and Artifactory installation to enable passwordless auth for uploading build artifacts as part of a CI build. This let developers not have to deal with managing a password for Artifactory, and also made security happy because that's a lot less passwords floating around.

Another idea that's helped the comradery and quality of our teams is language-specific working groups. My company has a few thousand developers, so establishing a monthly meeting where you share ideas, and facilitating discussions of recommended paths is useful. Your job as developer advocate can be a facilitator of the objectives and tasks, and run the meeting. Again, it's a significant time commitment, but has been very valuable for us. To emphasize; you're paving paths, not building walls. This is key to avoid preventing teams from deviating as needed, but also attract those that are looking for a smoother way of getting where they're going.

Did you talk to the developers what they want to improve, their pain points?

> create a quarterly summary and survey of engineers.

Be very careful about creating performance, compliance, or progress metrics that can be followed in its wording but not in its spirit.

For example, if the developers do not like writing documents but you decide to somehow track a documentation metric per team or person, that might lead to grossly misleading "positive" progress.

Unless the developers see a value in what additional things they need to do, imposing metrics will not go well.

First of the most important question: why? When proposing such changes, you need to carefully answer this.

Why do you want to implement those changes? Is it because as it stands currently it actually doesn’t work, or just because the status quo isn’t best practice?

> create a shared wiki and require teams to put access directions, openAPI specs, and each environment urls.

Imho this is a great change and pretty much universally applicable, but it’s not an easy one. Wikis quickly go out of date when there is no buy in from everybody involved. Also, you need to really think about the structure. The search function of some wiki tools (confluence) is famous for being awful, so make sure the wiki is well structured *and* remains that way.

> create ci/cd templates, starting with a paved path into k8s

CI/CD templates are great for some projects and awful for others. Depending on the standardization in your org in terms of tooling you may find it impossible to achieve any meaningful technology coverage for templates. My company has done a pretty good job in providing just enough scaffold to make building a CI/CD pipeline easy with libraries. We have libraries that make stuff like setting up internal package repositories, internal root CAs etc very easy and nothing more. People are quite happy with that approach.

However, a paved path into K8s and deeper into the cloud ecosystem of your choice (as implied by your status quo) is a kind of political and architectural decision I would not dare to touch in the pursuit of better developer experience.

K8s is a real beast and kind of overpowered for 95% if the instance is being used with. You need a real need for it to justify the investment required in terms of migration effort as well es engineering in maintenance (note that while managed K8s offers lift some off the maintenance burden off your shoulder, there is still a substantial amount of responsibility on your part).

> - create a quarterly summary and survey of engineers

Is this your entire job or is this some additional responsibility you plan on taking on. If the later, I’d suggest you stay away from this. I’ve seen this approach being tried several times and the result was always the same. After an initial spike in enthusiasm engagement is starting to die off while the effort required for upkeep keeps on rising.

What's the testing situation like? Also, is this a body shop / consulting company?

1. Talk to folks about what they actually want. People will find a way to resist/gum up things they don’t actually want to do. 2. Document the shit out of anything you change. Because you’re introducing a new standard on how to deploy, things will slow way the hell down as people have to figure out how to adopt these new patterns. 3. Then, dog food it. Spin up 5 new dummy services, and double down up improving the process, documentation, figure out where you can automate.

All of this is also to say, if a new principle engineer joined up after I had been there for 5-10 years, and suddenly told me we need to pivot to changes no one asked for, I’d either push back as hard as possible, or look for the door if I wasn’t specifically courted and take my tribal knowledge with me. Your job is to advocate FOR developers, not to impose.

> around 100-200 engineers. I don’t know exactly, because they don’t even have a full list!!!

Uh, perhaps start by making a full list?

talk with the devs, see what's the pain point, etc... and take it from there.

Have you been put in charge of dev ops or devex?

What’s the market sector?

As engineer in a .NET/javascript/python shop, I can tell you that literally ALL your suggestions would freak the shit out of me.

- k8 => why? I’ve never worked with k8, but I’ve heard enough to know it’s an overengineered shit show that solves a problem I don’t have. Probably most people think like me.

- iac template: don’t even know what that is.

- shared wiki: and this would be the 14th repo that is meant to contain all doc, and that would be totally incomplete and out of date before anyone reads anything in it. And if you had written any docs, you’d know wiki is the worst thing to ask your eng to work with. You want docs, write it yourself!

- scan repos for secrets, and make all access internal => how is that helping me exactly?

- linting and security scans on all code => well imposing linting is certainly going to help me reach that deadline. You want clean code? Clean it yourself, cause I surely won’t clean garbage code from a guy who left 10 years ago! Security scans? Meh.

- quarterly summary and survey: not my problem. Wait? Are you seriously asking me to fill in a painfully long form every quarter? Can’t wait for the “46% of you didn’t hand off your survey in time, you should do this over the weekend” email.

- standardized app templates: AH, at last something for me. Is that helping in any way? Probably not, cause I already have a standard way of doing things, so this again is some top-down imposition on how to do things. Unless your standardize way comes with a ton of stuff that would be painful to do in my way of doing things, this is more stuff preventing me to do my job efficiently.

—-

The problem is that you should ask yourself: what can _I_ do to help those guys? What can _management_ do to help those guys?

The best people I’ve worked with would take the pain points away. Like one of my boss would come in one morning and say: oh I’ve written the documentation for API x, and people would look at him and say “what? Thanks!”. Or one other came in and said “I hired some external database guy who’s going to go through all your sql queries and optimize them for you”.

“Ask not what your team can do for you, ask what you can do for your team” is a bit cheesy, but it’s basically the job you’ve taken. And no dev exp is not devops. At all.

You should really dive in the life of these people, take one-week rotations working very closely with all the various teams. See how they’d “onboard” you if you were a new employee. The first thing you want is a general overview of the place.

But you should really change your attitude towards your team. Comments you’ve posted here kind of show that you think you know better than the 200 guys who’ve been here before you. Don’t be that guy. We’re all engineers. We’re all smart. Things are the way they are because things happened. Learn humility.

I’ve been at my currently company for almost 20 years. I’ve seen a lot of people come and go, most of them thought they were going to fix everything… none of them did.

The problem is everyone comes in with their own ideas of what things “should” be like, this comes from their past or what they read online last week, not based on the needs of the organization. Have you talked to those 100-200 engineers and listened to what their actual issues are? You could spend a lot of time trying to push them onto k8 where it doesn’t make sense, or when other very basic problems aren’t solve for yet. You could setup all these new things and no one will use them, because they don’t know how, education could be needed, and depending on where they are starting from, you may need to do this it steps. Taking them from crawling to sprinting could take years of realistic steps toward an ultimate goal. If you dump it on them all at once, they’ll just keep doing what they’re doing and wait for you to leave. You don’t know what that goal should be yet, because you just took the job. You may think you know, but you don’t.

I’ll give you a perfect example of this problem in action. I did the math last week and on average, in my years at the company, we change our CMDB every 3.8 years. This usually aligns with a new leader seeing asset management as a problem and decides a new tool is going to fix it. We spent a couple years re-writing everything we have to work with the new CMDB, almost get to a point of stability, and then that person leaves and someone else starts the process all over again. Our problem has never been with the tool, it has always been an issue with the data. It’s still the problem. However, none of the new people who come in ever take the time to listen to the people who work there, so they migrate an existing problem to a new tool, cause thousand of hours of needless work, and solve nothing. They assign someone to own the tool, but no one is ever accountable for the quality of the data. Good luck with automation when the data sucks… the lack of data and standards mean all our code is 10x longer than it needs to be to cover all the edge cases, and new edge cases show up all the time. This is one problem, I can give you a long list of other problems that follow a similar pattern, some are specially done under the umbrella of “developer experience” and all they’ve done is create chaos and confusion among developers who just want to focus on their app.

Don’t start implementing anything until you talk to people, and let their pain points and requirements influence the direction. Actually listen to them, don’t just go through the motions. This process will also build trust with those in the organizing. You’re new, they don’t know you, they have no reason to trust you, and you’re trying to change systems you don’t understand. That’s going to go poorly.

It sounds like step 1 is putting a system together to know who all the employees are and what their roles are. That can be used as a way to get to know who you’re actually dealing with, and can then facilitate the conversations to find out what you’re dealing with. Having visibility into who all the teams are, publicly, can also help facilitate teams talking to each other and breaking down silos.

Also remember, the technology isn’t the product. Your company has something they make and sell, and the customer doesn’t care if it’s on VMs, K8, or any of that. Make sure you don’t load people up with so much stuff that they don’t have time to work on the actual product being sold. The technology should help get things done, not be the blocker that prevents new features from going out the door. Keep the workload and timelines related to the idealistic state realistic and the priorities clear.