Beyond the amorphous "do test reports inspire developer confidence?" I realize I don't know how to think about different testing thresholds.
I am taking over engineering at a young startup. Their approach to testing has been some automated smoke tests, and "90%" code coverage for unit tests. 90% seems too high for the speed tradeoff we have to make, but where to draw the line? Why 90% and not 84%?
Should we just try to find something that works and let production bugs be our guide? I would also like to explore other methods of testing I've never used, e.g. crowd testing, but I'm unsure how to fit that into my mental model of knowing well-tested software when I see it.
Any insight or articles are welcome. Thanks!
As it's a young startup I would focus on writing tests for the business-critical use cases, which should allow a team the confidence to "move fast and break things" knowing comfortably that the day-to-day is still functioning as per usual.
Ultimately, I would say most code should be covered with unit tests, but functionality covered by automated integration/end-to-end testing. To me that would be 'well-tested'.
(and no flakiness!)
Prototype is code that is built for the sake of proving something - it might be to gauge user behavior, or convince investors/customers you can do this cool stuff. It might be because it's hard to plan something and easier to hack it together.
Prototype code is built disposable. You can throw it away and replace it with an optimized thing later, and they're decoupled enough to be disposed. For pre-PMF startups, nearly 100% of code is prototype.
For prototypes, I write tests, but it's a manual checklist. It's also usually user behavior focused, such as "user tries to upload 15 images from third party photo gallery and gets an error limiting them to 5 images at a time".
There's some point where it's more efficient to write tests than keep pressing buttons.
Once you decide a feature is concrete (probably someone is paying for it), then you can refactor that and do some TDD on it.
Google and read about the Modern Testing Principles [1], it's a set of principles and a community around it (join the Slack channel through the same link) that tries to bring change to the " test reports inspire developer confidence" state of mind. There's not a lot of tangibles around it yet but it's a good start.
As for coverage percentage, there are multiple papers showing that (usually) anything above 80%-85% does not contribute to the quality of the product. Also remember that unit test are not great at finding "good" regression bugs, especially those subtle bugs related to timing, concurrency or external resources (e.g. your DB).
I can throw dome ideas but you'll need to be more specific for them to be useful-
- Use Data, from CI tests through just-before-big-release tests to production bugs having more Data and better ways to utilize it will help you find problems earlier and solve them faster.
- Don't be afraid of some level of manual testing, sometimes it's the most cost effective solution especially for start ups. Mobiles are the best example, it's very costly and difficult to build a stable test environment especially if you want to have good phone models coverage.
- Cover bottom up (aka "The Test Pyramid"), have more unit tests, a bit less integration tests (contract tests, sub sets of modules where the rest is mocked), have good but less API tests and finally have only a handful of E2E tests and even less UI tests.
- None of the above is set in stone though if it works for you, for example if your UI is simple and stable enough go ahead and use it as your main interface.
Finally remember that testing is hard and knowing that you have good results is even harder than for your actual product, so iterate and learn as you go.
[1] https://www.angryweasel.com/ABTesting/modern-testing-princip...
The coverage tends to revolve around ~75% but that's not a number I actively think about. Instead I think about it in terms of interactions, possible errors etc..
Note that there's no "percent coverage" whatsoever in that question. Instead, it's a question of what behaviors of your code you need to test.