How to log full HTTP requests

Question

I want backups for all data entering my system at the HTTP level. It happens fairly often in my career that the worst data loss bugs happen in ways where you're not covered by backups. The data doesn't get saved, validation goes amok, it gets saved them gets deleted. In all these cases having backups of the data as it enters the system would enable recovery of it.Are there tools like this? It's not enough to just save it, it would need to be searchable as well. The data may be sensitive so it would need to be encrypted too.

patoroco · Accepted Answer

If you&rsquo;re behind a reverse proxy, like NGINX, you could log the traffic and customize the log lines to be more verbose.After you could index these log files into a search engine, like Elastic Search and perform queries / dashboards through Kibana.This is the typical stack, so I&rsquo;m not sure if you&rsquo;re looking something more sophisticated.