HACKER Q&A
📣 madhatter999

Way to extract relevant parts from a PDF based on a question?


Dear HN,

I am trying to do some semantic search in a given corpus of PDF documents based on a question as input. My goal is to find the relevant parts from the PDF that best answers the input question. I am interested in finding out concepts, frameworks, and methodologies that will help me with this task. If you have any pointers, I would greatly appreciate it!

  👤 liampulles Accepted Answer ✓
This is a key usecase for text embeddings. Essentially it is a process of converting sentences or paragraphs to vectors, where the closeness of vectors then represents a semantic similarity.

So you can convert all the paragraphs in your document into vectors, convert your question into a vector, and then find the e.g. 10 closest vectors, or all that fall under a certain maximum distance, etc.

You can store the embeddings in a vector database, to search across multiple documents.

👤 verdverm
LlamaIndex is my tool of choice right now

https://docs.llamaindex.ai/en/stable/

https://docs.llamaindex.ai/en/stable/examples/citation/pdf_p...

I'm using it with Qdrant and can get the text sections & locations that are tied to the answer & citation as well

👤 anoni2
Try: https://notebooklm.google/