Chunking in RAG (with hands-on in LangChain and LlamaIndex) - RAG video series

[@ravindarmadishetty736]

Hi, when i am having 100k pdf documents and i store all the embedding into vector store without following any chunking. Now if i want to retrieve using prompt how can we augment relevant information on such an huge un-chunked vector? Please suggest what is the best way to handle this problem? Please help some references as well along with your inputs

[@AIBites]

is there any particular reason you skipped the chunking process? As the pre-processing and chunking operation is kinda one-time operation, I can think of re-doing the entire vector store with chunking. It may then be much easier to retrieve several times, for multiple queries, as and when needed What are your thoughts?