End To End LLM Langchain Project using Pinecone Vector Database

In this video we will be creeating an end to end LLM Project with vector.Pinecone makes it easy to provide long-term memory for high-performance AI applications. It’s a managed, cloud-native vector database with a simple API and no infrastructure hassles. Pinecone serves fresh, filtered query results with low latency at the scale of billions of vectors. database
github: github.com/krishnaik06/Comple...
Timestamp
00:00:00 Introduction To LLM Project
00:01:56 Setting up LLM Project
00:06:18 LLM Project Architecture
00:09:15 LLM Project Read the Data
00:16:05 Convert Data Into chunk And create vectors
00:21:40 Creating,Initializing,Inserting In Vectordb
00:27:58 Retrieving Results From VectorDB Similarity Search
----------------------------------------------------------------------------------------------------
Start Contributing in Open Source Projects
The-Grand-Complete-Data-Science-Materials
github.com/krishnaik06/The-Gr...
------------------------------------------------------------------------------------------------------
Complete Langchain Playlist:
• Amazing Langchain Seri...
--------------------------------------------------------------------------------------------------------
Support me by joining membership so that I can upload these kind of videos
/ @krishnaik06
------------------------------------------------------------------------------------------------------------------------------
►Data Science Projects:
• Now you Can Crack Any ...
►Learn In One Tutorials
Statistics in 6 hours: • Complete Statistics Fo...
Machine Learning In 6 Hours: • Complete Machine Learn...
Deep Learning 5 hours : • Deep Learning Indepth ...
►Learn In a Week Playlist
Statistics: • Live Day 1- Introducti...
Machine Learning : • Announcing 7 Days Live...
Deep Learning: • 5 Days Live Deep Learn...
NLP : • Announcing NLP Live co...
►Detailed Playlist:
Python Detailed Playlist: • Complete Road Map To B...
Python playlit in Hindi: • Tutorial 1- Python Ove...
Stats For Data Science In Hindi : • Starter Roadmap For Le...
Machine Learning In English : • Complete Road Map To B...
Machine Learning In Hindi : • Introduction To Machin...
Complete Deep Learning: • Why Deep Learning Is B...

[@krishnaik06]

Check out other Langchain Projects below kzbin.info/www/bejne/aoCUo6ZtoteBetE&pp=gAQBiAQB

[@saikatdas4654]

Sir U r amazing. Sir, if possible then make videos on poison attacks on text data like trigger attacks or backdoor attacks.

[@Nothing_but_NavEEEn_Official]

First like and Comment So Grateful for your guidance and contribution to the open-source community at the very need of Hour @KrishNaik Sir 🙏🏻❤️ Love from Bangalore ✨

[@vivekanand1122]

Amaizing Knowledge - Thanks Krish for your guidance !!

[@dastajan6789]

Informative video like ever

[@deekshad4774]

Thank you for such an amazing video!

[@prasadagalave9762]

Thank-you dada ❤

[@nooraftab6101]

@krishnaik06: Thanks for the valuable resources. Couple of changes as I was running the code: 1) the model davinici-002 is deprecated. Instead OpenAI advises to use 'gpt-3.5-turbo-instruct'. 2) Pinecone initialization is also changed. For anyone following and coding from here, it was two breaking changes for me.

[@rishabhranjan7687]

Hey there, I'm stuck at pinecode initialization. Can you please tell me how you proceeded?

[@darrendacosta6341]

@@rishabhranjan7687 hey have you been able to get it done?? if yes plz help .

[@karangupta_DE]

​@@rishabhranjan7687 the code in the video works with pinecode-client==2.2.1 But if you have 4.1.1, just search for langchain pinecode documentation and follow it step by step, it will resolve the issue.

[@krishj8011]

Amazing Tutorial...

[@linuxmanju]

19:12 you need to return doc from the function not docs ( which is the raw input before chunks )?. Nice video and tutorials, thanks

[@zainulabideen_1]

Yes I also noticed same thing

[@umeshnath8818]

Awesome bro!

[@inayatph]

thanku sir

[@60pluscrazy]

Excellent Krish 🙏 🙏 🎉🎉

[@sangeethag1966]

def chunk_data(docs,chunk_size=800,chunk_overlap=50): text_splitter=RecursiveCharacterTextSplitter(chunk_size=chunk_size,chunk_overlap=chunk_overlap) doc=text_splitter.split_documents(docs) return docs In this part you are not using chunked_data ..need to return - return doc...Is it right?

[@machinelearning3518]

yes have you checked len(doc)

[@krishnaprasadsheshadri6206]

The similarity_search function is not defined? Any ideas how to define it cause I dont get how it works without the function being defined

[@MansiKale-mr5ng]

This is outstanding please share me some videos related to this Build one Q&A bot using open Al and Vector DB provide a couple of docs which should be stored in a vector db and when the questions asked to LLM model the LLM model should retrieve the answer from the vectors stored

[@ShubhajitDutta-ve8bb]

Please consider to build an application based on this LLM model

[@quengelbeard]

Hey Krish, great video! Do you know if it's possible to automatically create a pinecone db index from code? So that you don't have to create them manually

[@user-kx5ll7gj3u]

Hi Krish , Bigg thanks for all your contributions, I just have a doubt How to retrieve all docs that has been injected to elastic db?

[@user-ho6iw4cg1c]

@krishnaik06, can we do the same with csv's having some kind of predictions from another ML model? Also, can we do some kind of weather report generation from IMD reports - I did try this, but it was not responding accurately. How to ensure the precision?

[@tamilselvan3525]

26:28. We should give the chunks as an input to the function 'from_documents' but not 'doc'

[@RaghavSoni-qv5bd]

AttributeError: type object 'Pinecone' has no attribute 'from_documents' how can i resolve this error

[@deepaliparochey2285]

@@RaghavSoni-qv5bd you got the solution ??

[@RaghavSoni-qv5bd]

@@deepaliparochey2285 yes I got the solution of it

[@user5398-v7x]

Hey Krish that was a great video indeed just to let you know some of the functions related to PineCone Vector db are deprecated had to do a bit of googling from my side to connect to the database.

[@autobase16]

🎉🎉🎉🎉

[@soumyaranjansahu4262]

Hey Krish, Could you please create a video on the setilup of pgvector

[@SameerAli-nm8xn]

Sir please try to apply dockers with it, because it will be really helpful.

[@dhruvsanan4738]

Hey!! Would you like to collaborate to make a llm app? I've already done this tutorial but I am thinking of making a full fledged app.

[@tanmayjagtap78]

Can anyone explain what is the difference between FAISS and Pinecone? In multiple PDF video he used FAISS but here Pinecone. Is there any specific way to use them?

[@Stacks-ce2k]

Sir I have a error of NoneType object is not callable in retrieve query vectore.similarity search line how to overcome with it

[@user-ho6iw4cg1c]

Do we have to create vector db all time? Cannot we store the data permanently?

[@anjalisingh1588]

I have one query - what if I want to connect my local database to a vector database and from that local database, the user can ask the question from the table !! Is this possible ?? if possible then how ??

[@rishiraj2548]

👍🙏

[@GunjanGrunge]

Sir, lets say I want to deploy this chatbot trained on one pdf which has 2000 pages, can I remove the docuemnt part and only query from the db so that the load time and the embedding cost can be minimzed in production level chat bot, do let me know if you dint get the question, I will try to explain it in a better way.

[@user-oz9fd8oc9j]

What is difference between loader vs parser in langchain?

[@niravkhanpara5039]

How its able to extract the text from the PDF? Is it using Tesseract or something else?

[@user-me1ry6lg6d]

how can i make an LLM model produce output that can be fomatted in the browser for display , like with html tags in it , is that possibel ?

[@AIdevel]

If I have many files in the same directory how to upload them ? thank you for your amazing vidoes

[@ashokreddyz]

Openai apikey which we nned to use free tier or subscription, I'm using pro version is 4 apikey error is unauthorized

[@muaadhnazly7022]

Do I need to have a paid account of Open AI to use the OpenAI Embeddings? Thank You

[@ShivamPatel-yg3kd]

Actually inside chunk_data you are returning the wrong variable

[@oliverkaunds]

I did not understand one thing. Does the Vector DB talk to the LLM? We store data from PDF in Vector DB to make it searchable fast. But where is the part where we query Vector DB and send to LLM I could not see any LLM call may be I missed?

[@shivanisingh8980]

Same question

[@shivanisingh8980]

I think we use llm as open embedding model...

[@mitejmadan8672]

Sir i dont have a credit card. Is it possible to create a project without using the openai api and just by open sourced models from huggingface and langchain. I am quite new in this llm field.

[@RISHABH-VERMA]

Use Gemini-Pro's API instead of OpenAI.

[@lakshmims7590]

How run this directly on visual studio by git hub repository

[@studytips7034]

can you please solve this error AttributeError: type object 'Pinecone' has no attribute 'from_documents

[@AkshayKumar-nh4fv]

same error i'm also getting, it seems like they have removed this feature. Now only server less services can create index

[@abhi9029]

I am stuck on "You exceeded your current quota, please check your plan and billing details". Anyone help !

[@mailsuresh9]

I am stuck on the same @krish Naik. Please help

[@mailsuresh9]

Actually, added $5 to open ai and waited for 20 mins and tried and it worked. Just created a new API key for clarity.

[@narnavar]

If you are using old account for accessing chat gpt ..create new account and key. 5 dollars can be used only till 3 months from the date of opening

[@sefercanapaydn9347]

in chunk_data function you returned "docs" instead of "doc" isn't it wrong?

[@shivanisingh8980]

Below you can see docs=doc

[@lokeshsharma668]

hi krish, when i run this code i am getting error like Pinecone has no attribute from_documents please help me to resolve this issue

[@zayanahmad5706]

Hi! Guyz when I run the code I got some errors. Please help me

[@SurajKumar-uf6it]

Sir can we do this same for csv files. Please answer this question

[@thomasferreira306]

Hi, just want to tell you thay i created a similar project 8 month ago, everything was working but pinecone has migrate to a new version now "init" is no longer a top-level attribute of the pinecone package. I've made lot of reasearch but i don't find any solution to make it working again did you have a idea ?

[@pranavbajpai5702]

Yes needed help on this if anyone has a solution to this

[@hamza_mehmood]

@krishnaik06 sir please can i have the link of that video which you are talking about at the end of this video??

[@vaishnokmr]

Disclaimer: if you dont have pinecone api, you can use chromaDB, Sqlite or multiple db but you have to knowledge about schema and tables.

[@dhruvsanan4738]

Hey!! Would you like to collaborate to make a llm app? I've already done this tutorial but I am thinking of making a full fledged app.

[@stefano94103]

Unfortunately 90% of the developer videos are useless because they only build the backend. This is useless to show your manager or a client. Videos that show how to connect a backend to the users front end would be 1,000 times more helpful

[@gudiatoka]

Right 👍

[@ARkhan-xw8ud]

You can check code with harry he has made with frontend

[@saurabhsinghjat]

Bhai hr cheez thali me paros ke nhi di jaati. Fr kaahe ke developer. Thoda research kro sb mil jaayega. Is information age me bhi specific information ke liye ro rhe h hm to bantadhar h.

[@shivanisingh8980]

​@@ARkhan-xw8udcan you plz share the link

[@Vasu-Katravath]

Then you build the backend, Krish Naik will try to make the front end videos if you feel backend is easier.

[@hetdave8379]

I don't want to use model API key. Is there any way to integrate LLM model which is stored in local machine and query the SQL database?

[@krishnaik06]

Use llama2 i have already created a video

[@user-yk7qb4ib3e]

hi sir in gihub source code of test.ipynb not avialabel with full code it has only import libirary

[@krishnaik06]

Updated the code

[@programwithpradhan]

Is there any open source embeddings we can use instead of Pinecone

[@manasviemmadi8072]

pinecone is vector database that stores the embeddings, it doesn't offer embeddings itself. there are many open source embeddings online. you can check the huggingface MTEB leaderboard for embedding models.

[@TheLeads]

Error: init is no longer a top-level attribute of the pinecone package.

[@m.rr.c.1570]

Why mine is taking so much time to print the answers

[@saransomasundharam2617]

hello krish, I was tried this code facing some issue with Pinecone.from_documents. there is no library called from_documents in pinecone.

[@dheerendrasinghbhadauria9798]

I am also getting same error

[@tharunps8048]

17:50 shouldn't we return "doc" instead of "docs" ?

[@krishnaik06]

Yes i guess that is a simple mistake i did :).Gald you found it

[@pk231]

I too noticed and thats why the Chunks have not been created and the size remains 58 I guess. I am not sure though.

[@rajvimultani871]

Can you pls provide next video link for UI part

[@rahulpawar8594]

@krishnaik06 : Thank for the detailed video. I need your help, I am trying to replicate the same with my list of documents. I am able to create vector DB, but when I am querying, something like, Give me the list of employee for which data is uploaded, I am not getting correct answer. Can you please help, how can we solve such type of scenario. your help is appreciated. Thank you

[@abhisheksinghmahra446]

i dont want to use openai api kindly make it with googlepalm api krish..i have to submit my college project tomorrow just let me know how can i do embedding with the help of palm api key i dont have openai subscription

[@kiranshenvi2626]

Hello Krish, Kindly request if you can share link to colab file if possible.

[@amit2k5]

The pinecone initialization step described in this video is not working. Any updated video or code please?

[@deepaliparochey2285]

did you get the solution ?

[@shubhamkumar1305]

How can we overcome that without using openai key?

[@CodeVeda]

use gooogle's like .... GoogleGenerativeAI(model="models/text-bison-001",google_api_key=api_key, temperature=0.2) It works

[@DebjyotiSaha-lh8lg]

I am getting an error in the pinecone initiation process. Considering all the process you said I am getting error as pinecone has not attribute init. Can you please help me on how to proceed furhter

[@syedmuzammilahmed6872]

Pinecone has now changed its framework. Some elements like "init," "from documents" etc. are now removed. So, the procedure used in this tutorial is now outdated.

[@dijendersaini9056]

it should be Pinecone instead of pinecode in pinecode.init

[@kyoli9847]

​@@syedmuzammilahmed6872 So how do we load a document object into pinecone now then? Since Pinecone.from_documents(docs, embeddings) no longer work

[@deepaliparochey2285]

@@kyoli9847 Anyone found the solution for this ?

[@piyush_nimbokar_07]

Pinecone instance is not being created and how to do in newer version in v3.2.2 please tell

[@deepaliparochey2285]

Did you get the solution ?? if yes please help

[@piyush_nimbokar_07]

@@deepaliparochey2285 use older version of langchian and pinecone it will work

[@RaghavSoni-qv5bd]

AttributeError: type object 'Pinecone' has no attribute 'from_documents'

[@shinycaroline3722]

Yeah same issue, something wrong with langchain and pinecone once they have upgraded the version

[@VikashKumar-ty6uy]

Getting this error while using pinecone vector db, can someone plz help me to resolve this pinecone.init( api_key="a4e02d-d1db-4d66-aaf9-fec1a868", environment="gcp-starter" ) index_name="langchainvector" AttributeError Traceback (most recent call last) Cell In[33], line 2 1 ## Vector Search DB In Pinecone ----> 2 pinecone.init( 3 api_key="a4e0266d-d1db-4d66-aaf9-fe1be7c1a868", 4 environment="gcp-starter" 5 ) 6 index_name="langchainvector" File e:\Gen AI Project\Langchain_using_vectorDB\vectordbvenv\lib\site-packages\pinecone\deprecation_warnings.py:38, in init(*args, **kwargs) AttributeError: init is no longer a top-level attribute of the pinecone package. Please create an instance of the Pinecone class instead.

[@harshadapatke885]

Pinecone has removed init() in latest update. please use pinecone-client==2.2.4

[@shubhamkumar1305]

Have you anyone give RateLimitError , 429 error?

[@mailsuresh9]

Yup. Stuck on the same on open AI 429 error.

[@mailsuresh9]

Actually, added $5 to open ai and waited for 20 mins and tried and it worked. Just created a new API key for clarity.

[@parantikaghosh1182]

pinecone is modified, this code does not work now, needs to be updated

[@nitinsiwach1989]

This video is outdated. The pinecone API has changed significantly and this tutorial wont work

[@ayushmishra5861]

Did you find the updated solution?

[@gowthamdora6146]

You need to return doc in chunk function but you are returning docs which is same as original documents,so you are not giving the text chunks for embeddings but whole documents.

[@krishnaik06]

Ha bhai its a minor mistake u can update the code :)

[@mohammadriyaz5586]

Please use free api's

[@siddappakurabar9015]

Sir how to get openai key sir

[@user-hs5ty7gz7j]

Can someone please help me, PyPDFDirectoryLoader is not working, Below is the error, tried my best to resolve not happening Traceback (most recent call last): File "C:\LLMAppPinecone\test.py", line 4, in from langchain_community.document_loaders.pebblo import PebbloSafeLoader File "C:\Users\jchristian2\AppData\Local\anaconda3\Lib\site-packages\langchain_community\document_loaders\__init__.py", line 163, in from langchain_community.document_loaders.pebblo import PebbloSafeLoader File "C:\Users\jchristian2\AppData\Local\anaconda3\Lib\site-packages\langchain_community\document_loaders\pebblo.py", line 5, in import pwd ModuleNotFoundError: No module named 'pwd'