How do you insure against regressions in performance critical codepaths?

Question

If you work on performance-critical systems like database software or language runtimes, what sort of tooling do you use to insure against regressions in the critical codepaths in your system? Are such checks performed in CI or right before some sort of a release or elsewhere?Tangentially, for folks who spend a lot of time benchmarking systems on non-bare metal instances on AWS EC2, how often do you have to deal with inconsistent benchmark results due to noisy neighbor issues?

al2o3cr · Accepted Answer

CI can help for some situations, but frequently the reason a particular code-path gets slow isn't visible in CI - could be anything from slow third-parties to slow production-scale databases.Most application performance monitoring tools can set up alerts for specific metrics; at one client, they used that to ensure that an endpoint called by external lead-generation code was returning a result within a specified time budget. (otherwise they wouldn't win any leads, which would be Very Bad)