I'm aware of the OpenAI Assistants API (still pretty early stage) but I'm wondering if there's any other options out there?
To be specific: I'm looking for an off the shelf solution where I can easily drop in a bunch of documents or a list of URLs and then ask search or summarization type questions on the dataset. I'm trying to avoid building it since it's a bit of distraction from my primary objective. Most of what I've found is pretty simple and DIY and still requires you to figure out document parsing and chunking. I just need something that works for existing web pages (with some non-static content).
My best solution is to download pages after a delay using a Chrome extension and load them into the OpenAI Assistants API but that's limited to returning 20 chunks and can be a bit limited on summarization.
I've been sort of building my own system in Go which is already a big improvement on the Python-based solutions, but it's nowhere near ready for this type of thing. But it uses Go-colly and go-readability for web scraping.