HACKER Q&A
📣 perlgeek

How do you handle cron mails?


I'm part of an organization that does a lot of IT operations, much automated, some manually. Lots of it is "traditional" metal or VMs, not k8s.

We get a lot of automated events; the two biggest sources are our monitoring system, and failed cron jobs.

What do you guys use to store and manage such events?

Currently we have a ticket system (OTRS) that can be used by email, but it is approaching its capacity limits, and it's simply not designed for this task.

Ideally I'm looking for:

* A way to make events visible to anybody interested

* Some sort of deduplication: if a cron job sends an email every 5 minutes, there should be a way to deal with them all at once

* Some sort of triaging: a way to mark an event or stream of event as not important / dealt with / whatever, and only give me a few of events that haven't been triaged

* Escalation: some events must trigger alerting to the appropriate teams

I'm sure we're not unique in needing a way to deal with this, but my searches haven't turned up a ready solution for this.

  👤 sneak Accepted Answer ✓
My system could be improved, but I wrap cronjobs in a shell script that doesn't output anything on non-error exit codes. It doesn't do de-dupe, it's a quick hack.

Wrapper:

https://git.eeqj.de/sneak/hacks/src/branch/master/cronify/cr...

In /root/.forward:

    |"/usr/bin/envdir /etc/environment.d /usr/local/bin/email_to_webhook"
That email_to_webhook can be found here:

https://gist.github.com/sneak/2b7d5348efcf00e7537a69dccef93d...

The script isn't mine, I found it somewhere.

I try to set up my cronjobs so that they don't output anything to stdout (they can log to syslog, logs are collected) if the process doesn't need attention. If it shows up in slack, I deal with it.

To do what you're trying to do, I'd figure out some way of getting my cron output into Sentry, or maybe just Sentry-ify all of my cron processes like you would do with any other program, and then ignore their output/return codes when running from cron.

👤 tiernano
not sure if this will solve all your problems, but i am using https://healthchecks.io/ for my cron stuff (backups mostly). It only pings me (Email and pushover, but other options available) when a cron task misses a schedule... it can talk to other services, so it might help...