Ask HN: Local RAG with private knowledge base
Looking for a free, local, open source RAG solution for running a reference library with 1000s of technical PDFs and word docs. Tried the Ollama + open webui, Ollama+Anything LLM with opensource models such as Llama3.2 etc. As expected the more documents we feed the lower the accuracy. Doing it for a bunch of senior citizens who still love geeking out.
> expected the more documents we feed the lower the accuracy
Not surprising!
The LLM itself is the least important bit as long as it’s serviceable.
Depending on your goal you need to have a specific RAG strategy.
How are you breaking up the documents? Are the documents consistently formatted to make breaking them up uniform? Do you need to do some preprocessing to make them uniform?
When you retrieve documents how many do you stuff into your prompt as context?
Do you stuff the same top N chunks from a single prompt or do you have a tailored prompt chain retrieving different resourced based on the prompt and desired output?