HACKER Q&A
📣 gitgud

How do you cache “rate-limited” 3rd party APIs?


Often in side projects I'm using 3rd party API's where the data is rate limited e.g; Weather apis, Social media apis etc..

Is there any easy solution like a proxy server where you funnel API requests through it and it only hits the actual Api when the cache expires or something?

Preferably, I want a simple and general solution which I can easily plug into a new project. Docker container or easily deployable would be great.

Anyone have any ideas?


  👤 dylz Accepted Answer ✓
> Is there any easy solution like a proxy server where you funnel API requests through it and it only hits the actual Api when the cache expires or something?

My lazy solution is nginx or varnish. You can set both to locally cache, set a burst r/s, etc. Listen on localhost or on your docker container, proxy_pass to the underlying API.

You can set it to disregard caching headers and always cache for a specific amount of time based on the request. If your URLs look like /weather/94704?key=11942890238, you can use proxy_cache_key to disregard the actual key and only cache against /weather/94704 for example (by dropping $is_args$args).

A docker container where you inject a nginx conf, or set up variables that overwrite like PROXY_PASS_TO_URL=, PROXY_AUTHORIZATION_BEARER=, ..., but the actual cache rules might need some tweaking.