Multi-modal RAG: Chat with Docs containing Images

[@engineerprompt]

If you want to learn RAG Beyond Basics, checkout this course: prompt-s-site.thinkific.com/courses/rag

[@jfbaro2]

Does it cover how to minimize (or even eliminate) hallucinations, and that the result would ALWAYS consider the content added into the RAG "database"?

[@rubencabrera8519]

This is the best AI channel out there, PERIOD. Thanks for sharing your knowledge

[@aerotheory]

Keep going with this approach, it is something I have been struggling with.

[@waju3234]

Me too. For my case, the answer is normally hidden behind the data, context and the images.

[@ilaydelrey3122]

a nice open source and self hosted version would be great

[@AI-Teamone]

Such an insightful information, Eagerly waiting for more multimodel approches.

[@tasfiulhedayet]

We need more videos on this topic

[@Techn0man1ac]

What about make same, but using LLAMA3 or less local LLM?

[@b.lem.2499]

Thanks, is there a video of the same project, but with langchain instead of llama index?

[@ScottzPlaylists]

Need to do it all in open source. No API Keys.

[@BarryMarkGee]

Out of interest what is the application called that you used to illustrate the flows? (2:53 in the video) thanks.

[@engineerprompt]

I am using mermaid code for this.

[@BarryMarkGee]

@@engineerprompt thanks. Great video btw 👍🏻

[@legendchdou9578]

Very nice video but if you can do it with open source embedding model it would be very cool. thank you for the video

[@AyishaAshraf-s2f]

Use case is to extract the relevant text information along with images available in the file using generative ai, When any prompt is given then relevant text information and image should display as response.

[@ai-touch9]

I appreciate your effort. Pl create one to fine tune the model for efficient retrieval if possible, with lang chain.

[@ArdeniusYT]

Hi your videos are very helpful thank you

[@engineerprompt]

Glad you like them!

[@vinayakaholla]

Can you pls dive deeper into why qdrant was used and other vector dbs limitations to store both text and image embeddings, thx

[@engineerprompt]

will see if I can create a video on it.

[@RolandoLopezNieto]

Lots of good info, thanks

[@avinashnair5064]

can you make it using comeplete open source models?

[@Makkar-b3v]

Great stuff.

[@RedCloudServices]

do you think all of this is now replaced with Gemini ?

[@BACA01]

Thanks your videos are very helpful. I have several Gigs of pdf ebooks that i would like to process with RAG. What do you think what approach would be the best, this or a graphrag. In my case i'm looking only for local models as the costs would be very high. What if to convert all pdf pages into images first and then process them with local model like phi 3 vision and then process it with Graphrag, would it work out?

[@mohsenghafari7652]

it's great job! Thanks

[@engineerprompt]

thanks :)

[@codelucky]

Is it better than GraphRAG? How does the output quality compare to it?

[@engineerprompt]

You could potentially create a graphRAG on top of it.

[@JNET_Reloaded]

wheres the code used?

[@ignaciopincheira23]

It is essential to conduct a thorough preprocessing of the documents before entering them into the RAG. This involves extracting the text, tables, and images, and processing the latter through a vision module. Additionally, it is crucial to maintain content coherence by ensuring that references to tables and images are correctly preserved in the text. Only after this processing should the documents be entered into a LLM.

[@engineerprompt]

agree!

[@jtjames79]

That's a lot of work. Can an AI do this?

[@engineerprompt]

@@jtjames79 Yup :)

[@erdi749]

I think that is the major point of ColPali. Regardless of its content, each PDF page is an taken as an image. Thus, there is no need for OCR, Layout etc. For sure, it has some limitations(i.e for complex queries, multiple pages will be retrieved with high scores, this may quickly overwhelm the context window of downstream generation task ) but based on my experience, ColPali based RAG(pick a vison LM say Qwen-vl) works great.

[@amanharis1845]

Can we do this method using Langchain ?

[@engineerprompt]

Yes, will be creating a video on it.

[@cristiantironi296]

What if the user query contain text + image?

[@engineerprompt]

You can you a VLM to generate description of the images and send that as part of the text query

[@cristiantironi296]

@@engineerprompt yeah as i was expected, but what if i pass an image that VLM doesn't understand, for example personal image not available online, i should first fine tune the VLM on my images then do what u said right?

[@garfield584]

Thanks

[@RickySupriyadi]

I except image generation will be have another kind of breed... image gen based on image understanding based on facts

[@redbaron3555]

This approach is not good enough to add value. The pictures and text needs to be referenced and linked in both vector stores to create better similarities.

[@engineerprompt]

watch my latest video :)

[@arifmp3284]

U have any work?

[@Know_Ur_World]

Which video ​@@engineerprompt