I'm still researching the topic and would love to ask for some points, good reads and similar advice. The requirements would be:
- to only have a single file as an index (so easily used in S3 for example). i want the library to only return IDs upon search so i don't want to store the actual data to save on index size. the index would be loaded into the memory of eg an AWS Lambda upon startup. (or kept on disk with EFS, whatever is better)
- support at least 1 million records with around 200-300ms response time.