HACKER Q&A
📣 gabrielsroka

Tools to call REST API and export to CSV?


I've written my own in JavaScript, Python, PowerShell, and I've used Microsoft Power BI/Power Query, but they are slow, don't work well, require custom code, etc.

I need to:

1. call REST APIs (with Authorization HTTP header) and get JSON (arrays of objects), also parent-child endpoints (parent has IDs of child objects) -- nested loops

2. paginate using the Link HTTP response header, observe rate limits (HTTP response headers, to avoid 429 errors) and concurrency limits

3. (nice to have) select certain fields from the JSON object; also, data can be "L-shaped" (not every record has the same fields)

4. export to CSV

Open source, cross-platform would be good. Needs to be easy for not-very-technical end users to use.

  👤 nhayfield Accepted Answer ✓
datatables.net is a good modular javascript library that handles pagination and child tables and export to csv/excel. you will still need to write some code for formatting the api calls into a useable format.
👤 majkinetor
What was wrong with powershell ? You could make it faster if you run using jobs/runspeaces and/or distribute on several machines. Should be dead easy.