How to return large response over an API?

Question

I am using a third party service which would transcribe an audio file and give me the entire transcription in a json format, I would read this and make a new json file by transforming it and send to s3. I am using python/django for it along with json, requests and boto3 module, however I noticed as the application kept running the memory consumption kept increasing.To fix this I had to write it as a streaming response to a json file and use ijson (https://pypi.org/project/ijson/) to read the content which decreased memory utilisation a lot. So while looking around, I found many people had the same issue, so this got me thinking how would I have sent this content if i had built the third party api myself.Some previous questions asked on stackoverflow,https://stackoverflow.com/questions/2400643/is-there-a-memory-efficient-and-fast-way-to-load-big-json-files-in-pythonhttps://stackoverflow.com/questions/11057712/huge-memory-usage-of-pythons-json-moduleHow do you send a response via an api for very large content, along with their advantages/disadvantages.

aardvarkr · Accepted Answer

Sounds like this is a personal problem and not a them problem. They&rsquo;re sending you a json and you&rsquo;re complaining that your memory usage is exploding. That&rsquo;s on you. If you&rsquo;re not actively cleaning up after yourself idk how you can blame the external api for your internal memory issues. That&rsquo;s my two cents at least. You can&rsquo;t possibly design an api that fixes a user&rsquo;s poor design.

tiew9Vii · Answer

If you can chunk the response json-seq/xml (sax parser) may be worth looking at.
The server can incrementally stream a chunk and client incrementally consume a chunk keeping flat memory usage.
I think gRpc also supports streaming but don’t know much on it.
JSON is a bad format for large files as generally you need to read the entire file in to memory before you can use it as you observed.

pestatije · Answer

Json is the default go-to standard for web services nowadays. But it doesn't mean it is the best format for your requirements. I'm not sure what an audio transcription is exactly, but if you can "stream" it you don't need json at all. Just use some basic serializer and stream that instead.

PEJOE · Answer

If you know the structure of the file so that you are comfortable reading it a bit at a time, and you&rsquo;re confident the structure won&rsquo;t change, C has several ways to control how much of the file you read in. You could just write a python module to handle this situation.