ColPali: Vision-Based RAG System For Complex Documents

[@manishsharma2211]

one thing what I like about this man is - he shows some background on each line / framework / library used to make people aware about all those nuances interactions b/w projects and researchers involved in it. love that

[@israelazarkovitch5852]

Colpali is an excellent technique for English documents. When you try to use non-English documents, the retrieval doesn't work well because colapli uses the paligemma model which is a relatively small model trained mostly on an English data set

[@engineerprompt]

good point, but I think you can finetune the vision model for other languages. Qwen is probably a good option there as well. Will see if there are any resources available and will share.

[@vardhan254]

qwen 2 VL is good for indic languages atleast from what i have tested

[@saranepalashok]

Excellent. Exactly what I was looking for. A "fine-tuning" episode of such a VBRAG pipeline would be a great followup episode.

[@engineerprompt]

good idea, will look into it.

[@frag_it]

Can you make an end to end project where instead of an index we throw the embedding to a vectorstore like chromadb or pinecone or something would be amazing

[@darpnpro]

Thank you for sharing this!

[@manishsharma2211]

EXCELLENT VIDEO - THANK YOU

[@kai_s1985]

Great work! Thanks! I wonder how it compares to vanilla RAG for text pdfs in terms of accuracy? Vanilla RAG suffers when the answer for the user question needs to be synthesized from different parts of the text. GraphRAG is good for those cases bit it is slow and expensive. Can this handle complex questions like those?

[@LTBLTBLTBLTB]

I tried to do this technique but with gemini-1.5-flash-exp-0827 and it works fine.

[@wdonno]

How do you "chunk" or parse sections out of longer documents? Or if we want to create a Knowledge graph? The ultimate analysis is done on an LLM, so we still have context length issues especially for local implementations. Can you extract the text itself for further processing?

[@shobhitbishop]

Will this work properly on pdf comprising detailed tabular information? And the hand drawn images?

[@BACA01]

Very good content, thank you.

[@engineerprompt]

Glad you liked it!

[@loicbaconnier9150]

Why do we need a large Vram GPU ? where for Colpali or VLM ?

[@mrrohitjadhav470]

SAME QUESTION FROME ME

[@IdPreferNot1]

Cool find in Claudette

[@saeeds851]

Would this approach work well with summarization with qwen2 vl 7b locally for technical papers with diagrams. Thank you.

[@engineerprompt]

Yes, checkout the localgpt-vision project that implements and end to end vision based RAG. kzbin.info/www/bejne/j4HWZZh9edV8j5Y

[@Yes-lm9dq]

Do you think one could use this and convert a pdf into a text file which can be used to generate a knowledge graph using Microsoft's GraphRAG?

[@tecnom7133]

Thanks

[@tecnom7133]

may be if you pass an Image URL instead of the Image bytes you will consume less input tokens so less Cost?

[@absar66]

many thanks for this great video…I have a set of scanned pages saved as pdf. will this work?..thanks..

[@engineerprompt]

Yes, I think this approach will work on scanned pages as well.

[@diego.castronuovo]

What if the information to answer a question is in two consecutive pages, and only the first one is retrieved because the second only contains the continuation of the first. This is a real problem.

[@engineerprompt]

You can ask to retrieve multiple images/pages or can append the neighboring pages to your context. To make all this simple, I have put together an OSS, video coming soon: github.com/PromtEngineer/localGPT-Vision

[@amortalbeing]

thanks

[@goran-ai]

What is the best way to contact you for consulting with our dev company?

[@RedCloudServices]

I wonder if a VBRAG could perform math calculations extracted from an image table? 🤔 I suppose if the results are accurate they could then be passed to another agent capable of calculations on the result?

[@engineerprompt]

math might be a little hard but I think its worth trying.

[@user-uk9ls]

This does not run on local Nivida RTX 4x 16 RAM GPU ?

[@engineerprompt]

I think that will be able to run the pipeline.

[@neatpaul]

But this works only for pdf, what about docx, pptx, epub files? I want to work multimodal on those files too.

[@fra4897]

it works with whatever that can be converted to image, so everything

[@ayanshproplayer5559]

Offline work???

[@engineerprompt]

if you watch the video, you will know the answer :)

[@kareemyoussef2304]

None of these solutions are open source..even in your other videos. I think the video you have that uses marker is the only one