How to build Multimodal Retrieval-Augmented Generation (RAG) with Gemini

[@GoogleDevelopers]

Check out all the AI videos at Google I/O 2024 → goo.gle/io24-ai-yt

[@gangababu2063]

IO2024_Multimodal_RAG_Demo.ipynb can't find this notebook

[@PeterLappo]

Pretty useless video without sample code.

[@jprak123asd]

I wanted to extend my heartfelt thanks for the excellent session on how Retrieval-Augmented Generation (RAG) can be used to train Large Language Models (LLMs) to build expert systems in the retail, software, automotive, and other sectors. Your explanation was incredibly clear and insightful, making a complex topic easily understandable. I truly felt like Dr. Watson listening to Sherlock Holmes unravel the mysteries of the universe, marveling at the clarity and depth of the information presented. Your efforts in breaking down the concepts and applications of RAG in such a straightforward manner have left me feeling both enlightened and excited about the potential this technology holds for our industry. Thank you once again for your time and for sharing your expertise. I look forward to exploring and implementing these innovative solutions in our own projects

[@sarvariabhinav]

WHERE IS THE SAMPLE CODE??????? This is very frustrating to showcase but not share code.

[@diegomoralessepulved]

@googledevelopers I second this comment.. could you please share that notebook?

[@noetic4681]

agree

[@thyagarajesh184]

Impressive technology. Look forward to using it for my project.

[@charlesbabbage6786]

Could'nt find the exact notebook used here.

[@dumbol8126]

will there be an opensource version of this, or atleast a paper

[@zuowang5185]

How do you handle terabytes of enterprise data, just do embedding groups? Should you generate sub questions first? How do you handle large amount of users?

[@homeandr1]

Hello Jeff, could be that there is a mistake 24:00 in a for loop instead of “for i, s in enumerate(texts + table_summaries + image_summaries)” should be “for i, s in enumerate(text_summaries + table_summaries + image_summaries)”

[@nestorbao2108]

Why do you use multimodal embedding model if you summarize images and ground them into text?

[@mariaescobar8003]

When I use RAG, Am I sharing my data with the model/company? or is it private with an extracost?

[@vichupayyan]

Rag is an architecture i believe. with out without it - whatever happening to the data same applies

[@hitmusicworldwide]

Not necessarily. You can keep the data local. You only use the LLM for it's ability to summarize and generate responses as well as queries

[@mohamedkarim-p7j]

Thank for sharing👍

[@hasszhao]

where is this notebook in the cookbook repo?

[@d.d.z.]

Same question

[@shubhamsharma5631]

33:18

[@Chitragar]

I have a notebook in Kaggle named Multimodal RAG Gemini - should help, YT removing links for some reason.

[@d.d.z.]

@@Chitragar thank you

[@cullenharris1837]

​@@shubhamsharma5631 I challenge you to find it. That is simply a link to the general github which is convoluted , not the exact notebook which is difficult to find.

[@TL735]

Nice, but why don't you develop a simple drag-and-drop RAG? e.g. I add a drive folder link and Google generates a RAG chat based on its content.

[@evanrfraser]

Fantastic. Thank You!

[@yadav-r]

Can I use the fine tuned Gemini RAG model via API from a mobile app?

[@nagpalvikas]

Is "unstructured" the best choice here for parsing PDF? Any better alternatives?

[@ai_asymmetric]

Llamaparse

[@You_Only_LiveOnce]

langchain would be a good choice

[@nagarathnabheggade8410]

This example briefs about text and PDF, do we have any for video how de we use RAG, Vector store for Video can anyone give some reference

[@descarded]

im not sure if there are existing libraries to do that, maybe check docs. although here's my intuitive approach. video is basically series of images with some history/context attached to previous and subsequent frames. so if you keep that history across frames intact by either providing previous frames as input, or keep a local vector of it all, you can make it work. not sure if its the best approach, but i m open for discussion

[@RiccardoCarlessoGoogle]

Is there a link to the python notebook? I'd love to play with it!

[@IndianLeopard7]

Wat about Copyright and Ethical issues? How much do u guys charge for using ur model? And as per IBM and Oracle embeddings are nothing new so why use urs?

[@ammarfasih3866]

where is the notebook? Can someone please share the link?

[@julianayue402]

Can you please provide the source code? It would be great help!! Thank you!

[@ajanieniola9172]

Wonderful

[@SonuChaudhary]

Where is the code link?

[@pra8495]

github link please

[@shubhamsharma5631]

33:18

[@kaushikdas5115]

@@shubhamsharma5631 can we run the code without subscription?

[@oldmansgoldenwords]

You can get blue driver and get all error codes and example

[@adithiyag4616]

Please share the colab link

[@gnanasenthil654321]

Yes please do

[@shubhamsharma5631]

33:18

[@ai_asymmetric]

dense embeddings are never enough for RAG system

[@SB-md2km]

Ok but someone could literally look any of this up online or look for it in a manual, etc. w/out using AI...

[@dr.p.srinivasaragavanperum2911]

Happy

[@dr.p.srinivasaragavanperum2911]

🎉

[@dr.p.srinivasaragavanperum2911]

❤

[@user-xx3mr6vx9u]

Haha we just need your browsing history

[@KitchenAIdev]

Hmm... interestin...

[@fast-path]

🥺

[@JH-bb8in]

This shows how garbage Langchain is as a library. Extremely verbose and intransparent.

[@imai-pg3cz]

Is there any framework better than Langchain?

[@gokusaiyan1128]

can you tell me more about it please :)

[@Inceptionxg]

After Muaadh Rilwan's post on LinkedIn

[@ohmatokita5990]

So if I'm using the 2nd way, what's the name of the multidality-modal would be?