“I want to give ChatGPT 10x more docs” - RAG Explained

[@bryandotmee]

This is sooo important, the pdf size limits has been a blocker for analysis of docs that are over a few pages

[@Zhaku69]

This video likely just solved an issue I was trying to fix for my company. Love these more teacher vibe type videos thanks so much!!

[@RalucaMMihai]

In his community he does lots of lectures...

[@RalucaMMihai]

In his community he does lots of lectures. I recommend you check it out

[@02167631]

You are such a gifted teacher. The way you explain things in a way that makes such a foreign concept easy to understand is magical.

[@zaneperrydotcom]

I agree; well done. Don't change a thing. This is the level of helpful explanation we are looking for. Well done!

[@malcolmvanhilten125]

Excellent, i think Vector shift should link to this vid from their website. A great tutorial for getting to know the product and especially its capabilities. More of these please Igor

[@JonasMunnich]

You are so good at explaining more advanced concepts in simple terms - Really useful! Pls make more of these kinds of videos!

[@EriCraftCreations]

I am so grateful that you explained what RAG is in an essy way to understand. ❤

[@aiadvantage]

This topic has been on my list for the KZbin for a while. Glad you enjoyed the explainer

[@pandipatipavan3804]

HI​@@aiadvantagebrother can you please make a complete video on how to earn or make money using AI tools for free as soon as possible for free 🙏 I am from India and I really need money since I lost my Job due to Recession brother. I think it could be more helpful if you could me a way by making a complete video on How to use AI to make money easily with Indian Rupees and what are the websites that will provide the real money in a faster manner. I will be eagerly waiting for your kind response and Thank You so much brother 🙏

[@danthompson8309]

Thank you for the video, very helpful and clarifying. Yes, please more of these technical primer and proof of concept videos . Thank you.

[@slackerpope]

I'm a student and writer. This video is exactly what I needed to get started with RAG. The last example was critical. Showing us how to hook up the knowledge base was huge. That step was not intuitive. I'll be referring to this video several times in the next month. Also, yes please, more education videos. THANK YOU!

[@tar-yy3ub]

Honestly don't care that this was sponsored, this was useful content

[@aiadvantage]

That means a lot. I only do sponsored vids that I would want to make anyway :)

[@kirlyukvv]

@@aiadvantage Respect

[@nicolaikalb]

Great video! Exactly what I was looking for. Keep on going with your wonderful style. Greetings from Cyprus.

[@hope42]

Back when me and Al Gore invented the internet, I remember hearing about my first fun memory of automation, like this story: Around the early 1980s, a group of programmers at MIT connected their Coke vending machine to the internet. This allowed them to check the status of the machine remotely, particularly to see if there were cold sodas available or if it needed to be refilled. This is often cited as one of the earliest examples of an internet-connected appliance, a precursor to what we now call the Internet of Things (IoT).

[@ekot0419]

Wow. I cant agree more for what you said in this video. 8 minutes into it. Thank you. Yeah, I burned myself out trying to understand what agents is about.

[@Hall]

❤ These educational videos are very valuable. You are a good teacher. The first half especially upgraded my AI approach. Thank you!

[@dannyquiroz5777]

Nice video I wasn’t aware of this tool, could be useful for many use cases

[@curtcooper5465]

Very much thankful for this man.

[@RaxLakhani]

You really did a good job in helping me understand the various terms and functions around LLM queries Thanks

[@SomogyiCosmin]

WoW. Thank you Igor for sharing this knowledge. I think this kind of videos are very usefull because of the real live examples.

[@CM-zl2jw]

Brilliant video- the Zombies have arrived and we need to be prepared!! 😂😂. Thanks for the chuckle!! much appreciated! Thank you. I like the “completionist Igor”. Others have said it… you are a gifted and highly talented sharer of knowledge. Love learning how to 🐟 🙏

[@fcmelo1]

It was exactly what I was expecting. Thank you.

[@maniacatthecontrols]

Yes, I prefer educational tutorials like this. Thank you.

[@kamelirzouni]

Thank you very much for this impressive work. This is one of my favorite videos you have produced.😍

[@aiadvantage]

Alright noted! Thanks for the kind and generous comment

[@Bmutch966]

Please do more educational videos :)

[@Mr_Chiro_]

Great explanation of everything. Even with my advanced knowledge this was a very good explanation.

[@aiadvantage]

Love to hear that! The definitions in this space are getting fuzzy at best so more discussions on topics like this really are necessary (at all levels). I feel like no two people could agree on the definition of "AI" or "Agents" let alone "AGI".

[@jamessukka]

Amazing Tutorial Igor. 👏👏👏. I have been having a question since long if we can create a knowledge base with pdf that contains images as well. If possible plz direct me where I can learn about that.

[@ivanjosegomes9599]

Great stuff, great video format, great teacher!

[@sandiegojoey1]

I haven't finished the video yet, so sorry if you covered this. One of the things I want to be able to do is to have an LLM read a document and when I ask questions about the document, I need the LLM to cite the source of its answers by highlighting the specific passages that support its answer. Is this possible yet?

[@CM-zl2jw]

Should be easy if you give it a prompt… just say what you said here to the LLM.

[@kamelirzouni]

Thank you!

[@GiveMeTheMic22]

Amazing thumbnail is this midjourney?

[@EddyRoAguilar]

So great content! Please do more like it, but as you said be conservative with sponsorship so it can be credible. 💯

[@LuisMaaan]

Is it possible to add some kind of database with the goal of storing histories of unique chat sessions of users with VectorShift? to build some kind of customer support bot that is able to store past conversations and pull them as context too

[@697sona]

Nice demonstration, I have a question about chunks, how would I know the number of chunks that would be required to answer a query beforehand? as you put 2 for chunks before you gave query about chainsaws

[@aiadvantage]

All depends on the purpose of the chatbot. If the goal is to answer specific queries then 2 chunks is what you want. If there is many documents and the answers should pull together a lot of varied info then you will need more. All depends on the goal and knowledge base (and how its structured)

[@697sona]

@@aiadvantage I understand thanks, but what if there was another mention of chainsaw and you were unaware and chose 2 chunks, in that case which 2 chunks will be used and which will be left out?

[@thequantumsword]

Thanks so much for this. Could anyone help me understand why investing in Vectorshift is superior to just building a custom GPT in ChatGPT Plus? I'm asking in the context of writing an entire academic textbook.

[@aiadvantage]

GPTs always require user interaction. You can't just let a GPT run in the background as opposed to the Vectorshift pipeline which you can deploy and it just runs 24/7 and waits for the trigger to happen. Also GPTs can't be added to a website as a standalone chatbot and you don't get to branch out the interactions like you can here. Hope that makes sense

[@alexanderahamilton6123]

​@@aiadvantage Also a writer here - so my understanding then, is that if I only need to use this as a tool to retrieve and utilize information for myself for my own project or projects, I can just upload files to a ChatGPT interface as I have done previously, and there's no additional advantage to my building a knowledge base. Or is there...? Thanks much - great work.

[@writetopardeep]

Brilliant video man

[@Mr_Chiro_]

I would also promote Flowise for building RAG backends

[@SupahNin10dohp]

Do you know what this increases the file size or context length too? I have PDFs that even Google Gemini with its 1 million token limit won't even read, or when it does it gives a short answer.

[@Bmutch966]

Awesome video! Thanks!

[@africanqueen007]

what happens to the documents uploaded...does vectorshift preserve confidentiality?

[@slackerpope]

Does using RAG force the LLM to only reference your context data to formulate its reply? I'd like to eliminate hallucinations so I can be confident that the output is actually relevant. Example: I'd like to feed a pdf of a play into RAG and know that the LLM is only referring to my document while formulating answers and not hallucinating material that is similar. Any information on how to keep the LLM from hallucinating while I'm trying to chat with my documents would be greatly appreciated.

[@rapidAIWebDev]

RAG-base implementations hallucinate because LLMs often treat certain words as being identical to each other even when they are referring to different things. For example, GPT-3.5 Turbo surprisingly treats calcium and magnesium as if they are the same thing. (Demo: kzbin.info/www/bejne/gWW6mGmHr7WZr6ssi=kqlrJCfbYcT-j67x&t=412 Time: 06:52) GPT-4 treats Afonso and Alfonso as if they are the same, even though they are distinctly different names. (Demo: kzbin.info/www/bejne/gWW6mGmHr7WZr6ssi=9gizAFSk7OTEZabe&t=554 Time: 09:14) If you want to use RAG, your implementation must address this issue or you will still have hallucinations. In fact, this is the most important issue that your code needs to address. You could use something like RAGFix to resolve this and eliminate hallucinations.

[@waqqashanafi]

Doesn't Google Cloud suite already do this for free? (with their AI Chatbot builder tool)

[@RedTick2]

Great video! Very cool tool, however the price ramp seems very high. The free level of course is a great place to start, but very quickly I can see needing a bit more to practice and try out flows, but I get that that should not be free but 20 bucks a month is a lot considering at this stage of "play" this is not yet commercially viable and may never be. In other words, I would very much like the ability to have useful conversations with documents, or in my case obsidian files/DB's and I am willing to pay, but I can't justify 20 bucks a month. That's on top of my other subscriptions to make all this work. Is this just me or does this feel pricy for a hobby application? I understand it's reasonable for commercial applications.

[@rapidAIWebDev]

Even with RAG, ChatGPT hallucinates up to 23% of the time. The reason is that LLMs often treat certain words as being identical to each other even when they are referring to different things. For example, GPT-3.5 Turbo surprisingly treats calcium and magnesium as if they are the same thing. (Demo: kzbin.info/www/bejne/gWW6mGmHr7WZr6ssi=kqlrJCfbYcT-j67x&t=412 Time: 06:52) GPT-4 treats Afonso and Alfonso as if they are the same, even though they are distinctly different names. (Demo: kzbin.info/www/bejne/gWW6mGmHr7WZr6ssi=9gizAFSk7OTEZabe&t=554 Time: 09:14) If you want to use RAG, your implementation must address this issue or you will still have hallucinations. In fact, this is the most important issue that your code needs to address.

[@kirlyukvv]

And how to solve it?

[@rapidAIWebDev]

@@kirlyukvv @14:04 the video uses PubMed IDs to teach how Noun-Phrase routes work. The video then applies this to a real-world RAG problem to show how to alter the context to remove conflicting noun-phrase routes. That's the methodology for solving it. And if you want it fully automated, you can use a service like RAGFix that takes care of it for you.

[@yassinbadermedia]

Danke Brudi

[@plaverbach]

Just tried to find out if I should cancel my Chat GPT and get a Co-pilot Pro. I could not find anything updated on KZbin! IDK if this is an opportunity to you

[@aiadvantage]

copilot is worse in every way except that its safe to use for corporations so I would stick with ChatGPT

[@plaverbach]

And integration with MS Office. My friends have been telling me it is crazy!!

[@kellymaxwell8468]

so will this help with games how will this help with games   We need an AI agent's ai can reason code program script map. So games break it down and do art assets do long term planing. Better reason so it can do a game rather than write it out. Or be able to put those ideas into    REALITY. And maybe being able to remember and search the ent conversation needed for role playing and making games.

[@collabcomm9007]

Dude how do you make those thumbnails please tell me

[@aiadvantage]

We have a very detailed step by step guide in our community 😁 Might have to make it public soon because so many people keep asking haha

[@learnwhispering]

I'd rather learn to have have wild apex dinosaurs 🦎🦎crawl on me (and i have)... but learning this means one can survive the AI event horizion that will obliterate the middle class. 💕💕 thank you for the vids

[@CM-zl2jw]

🤔 interesting comment.

[@angloland4539]

♥️

[@UchechukwuOnuoha-cw4cl]

I think 🤔 that Iam the only Nigerian ( African) in this channel ❤

[@tedbozarth8300]

The AI industry should align on one key priority / to stop using the word agentic 😂

[@hope42]

I just asked Gemini Pro 1.5 Experimental 2 million context window ... Are you a vector DB? It said kinda ... then said diagram you with Mermaid code. graph LR A[Your Input] --> B(Tokenization); B --> C(Embedding Creation); C --> D{Similarity Search}; D -- Closest Embeddings --> E(Retrieve Information); E --> F{Response Generation}; F --> G[My Output]