How do you do monitoring/observability for dev/staging environment?

Question

If you are using APM tools like DataDog, NewRelic - do you also use them for staging environment? How do you optimise performance at staging - or the task of optimization is left to the production env. as long as things are working fine?If you use APM tools for staging/dev environment also - doesn't it increase your cost significantly?

eoinbmorg · Accepted Answer

We have all our staging systems report to DD and Sentry, tagged with an `env:stage/prod/whatever` tag to slice the metrics by. This helps maintain parity for alerting too (although pagerduty is not enabled for stage. Sometimes it's tough to get right because the lower volume of traffic on stage makes the alert metric queries very noisy. For example, alerts that fire for error rates may not resolve if no new successful requests come in for a few hours on stage.

linsomniac · Answer

Our staging environment and our dev database both run on very under-powered servers, by design. The idea being that we notice performance problems, at least as far as coding and queries. Catching the most egregious cases.We do feed data into influx via telegraf, elasticsearch, and sentry, and also do system monitoring via Icinga2. But in dev/stg it is not treated as actionable.