How do you do software testing?

It depends what type of ticket we'll be playing. If it's a bug, we'll probably write or augment an existing test that tests that "X does/doesn't happen". If it's a feature we'll start with the unit and integration tests to show it works, and then write the code that does that thing.

Our team is all Python so we rely on pytest and mock etc. For testing AWS related stuff we typically use Moto, which is an AWS mocking library, but sometimes to have it run locally with localstack, and then against the dev/uat environments once the code has been pushed.

End-to-end tests exist as well that cover the major user stories but these can be a little bit flaky - and this is the main challenge right now.

I find I get great results with UI testing by observing someone use the software and not tell them how to use it. Then refine away all the bumps.

My team's main software is a backend network service that feeds off of a queue and sends data out into the wider internet. It is the tail end of a pipeline of work with another team responsible for the incoming work. We call out to a couple dependent services as part of our flow.

We write unit and unit-integration tests to cover the error cases and success cases. These make sure that our code flows the way we expect, handles errors talking to a dependent service or data source, ensures retries happen, errors and events are logged, etc.

We have integration tests. These are written in a test framework and leverages docker compose. It spins up all our dependencies and mocked/stubbed sinks for data we send out to the world, and we have utilities that populate our incoming edge with data. These tests are black box: given this input, we expect this output. These tests ensure that known payloads produce known artifacts.

When developing, we write the tests. Not all stories/tickets need integration tests, but most do. Unit tests are universally expected to be written (we don't strive for 100% line coverage, but coverage needs to be reasonable). For QA, another developer of the same team reviews the code, the tests, and validates in any other way that the code meets our definition of done, meets its acceptance criteria, and generally does what it is expected to do. This developer will often manually execute the code to be sure it does what the code and tests say it will. They are responsible for asking the "what if" questions (what if $foo fails? what if assumption $bar changes? do we have telemetry in place to debug this in the future?) and ensure that the code is good to go to production.

The ticket/story pull request(s) are blocked from merging to master until the automated tests pass. After they are merged to master, the automated tests run again. The master branch tests create any artifacts needed for deployment.

We leverage buildkite for our builds. After master is green, buildkite plugins allow us to automatically create canary deploys, deploy the whole cluster, and/or roll back a deploy. Deploys start by going into our staging environment.

Our staging ecosystem is where all team's latest code lives. We have a set of acceptance tests that are run here as a matter of course. These run tests as if a customer initiated an action, and ensure our system behaves as expected. If these tests do not pass, we do not roll the code to production.

Anyone on our team can deploy any time, though we try to keep that during business hours and not on Fridays. Deploys are logged/announced in a slack channel in case something later goes wrong, and someone needs to do some detective work to find out what happened when.

For the production roll out, as mentioned earlier, we will deploy canary nodes. We then observe logs, alerts, and metrics. Depending on the scope or risk of a deploy, canary deploy nodes may be long lived or not. The deploy is rolled through the rest of the cluster and that is that.

Tools: Go, docker, docker-compose (redis, mysql, other services), dnsmasq for simulating the outside world, buildkite, kubernetes, argo cd, and more.

QA team: as alluded to, we don't have a "qa team." We separate the QA activity on our team. As we are responsible for the live monitoring and running of our services, we are in the best position to more rapidly test our own software.

Challenges: we had to build a lot of tooling, but it was very much worth it. We can deploy multiple services multiple times per day. We still have more things we would like to have under automated test and automated monitoring so we can do better "less high touch" deploys. Our biggest challenges arise due to scale. We cannot generate enough load in dev or staging environments and can only see some classes of problems when we go out to the production clusters. Customer's can do all kinds of strange things with the data they give us, and we cannot attempt to capture the combinatorial complexity they can throw our way. Sometimes, an error case has to build for days or weeks to even be noticeable in our metrics. Some of our caches can be very long lived and sometimes a cache invalidation bug might slip through; those are the worst. Any time a bug is discovered, it is added to one of our test suites to guard against regressions.

Overall, I like where our team is from a testing perspective. There is always more you can do, but you have to strike a balance between providing value to the org and protecting ourselves from ourselves. You can go too deep into testing land for marginal benefit. I think we are at a decent balance where we can refactor ruthlessly when needed and have good confidence in anything we release. I would like to see more from our team on performance testing / monitoring over time. We have telemetry on all this stuff, but not tied to our build system.

I kinda just slammed this out stream of conscience style; hope it rolls ok.