Retrieval-Augmented Generation chatbot, part 1: LangChain, Hugging Face, FAISS, AWS

In this video, I'll guide you through the process of creating a Retrieval-Augmented Generation (RAG) chatbot using open-source tools and AWS services, such as LangChain, Hugging Face, FAISS, Amazon SageMaker, and Amazon TextTract.
Part 2: • Retrieval-Augmented Ge... - scaling indexing and search with Amazon OpenSearch Serverless!
⭐️⭐️⭐️ Don't forget to subscribe to be notified of future videos. Follow me on Medium at / julsimon or Substack at julsimon.substack.com. ⭐️⭐️⭐️
We begin by working with PDF files in the Energy domain. Our first step involves leveraging Amazon TextTract to extract valuable information from these PDFs. Following the extraction, we break down the text into smaller, more manageable chunks. These chunks are then enriched using a Hugging Face feature extraction model before being organized and stored within a FAISS index for efficient retrieval.
To ensure a seamless workflow, we employ LangChain to orchestrate the entire process. With LangChain as our backbone, we query a Mistral Large Language Model (LLM) deployed on Amazon SageMaker. These queries include semantically relevant context retrieved from our FAISS index, enabling our chatbot to provide accurate and context-aware responses.
- Notebook: gitlab.com/juliensimon/huggin...
- LangChain: www.langchain.com/
- FAISS: github.com/facebookresearch/f...
- Embedding leaderboard: huggingface.co/spaces/mteb/le...
- Embedding model: huggingface.co/BAAI/bge-small...
- LLM: huggingface.co/mistralai/Mist...

[@jacehua7334]

always making great and timely videos.

[@juliensimonfr]

Glad you like them!

[@caiyu538]

Thank you for your lectures.

[@juliensimonfr]

You are very welcome

[@justwest]

thanks julien, one can learn so much from these!

[@juliensimonfr]

That's the idea 😀

[@AaronWacker]

The RAG chatbot you demonstrate is an excellent lesson with HuggingFaceEmbeddings. Regarding how to do it outside GPT being generic enough to have your own vectorDB on demand for any model I had wondered how that was done. Thanks for covering this really great stuff!

[@juliensimonfr]

Glad it was helpful!

[@iAkashPaul]

Hey Julien, great job with the video. For QnA on corpus I'd recommend to generate hypothetical questions for each paragraph & ingesting them as well since those would have better similarity to the user input which is usually a question & can also help constrain the model to answer only closed domain questions.

[@juliensimonfr]

Yes, that's a nice trick. I tried to keep things simple here ;)

[@devilliersduplessis7904]

Hey Julien, Thanks for an insightful talk last night at the AWS center!

[@juliensimonfr]

You're welcome. Thanks for coming!

[@kuzeyiyidiker1344]

Thanks for this clear explanation.

[@juliensimonfr]

Glad it was helpful!

[@ComFomeTo]

Thanks a lot! It was very, very helpful.

[@juliensimonfr]

You're welcome.

[@DCTekkie]

Thank you, gonna check it out tomorrow!

[@juliensimonfr]

Have fun!

[@edinsonriveraaedo292]

Hi Julien, thanks for your video, pretty clear explained ;-)

[@juliensimonfr]

Glad it was helpful!

[@badbaboye]

Thanks for the video!

[@juliensimonfr]

You're welcome!

[@VenkatesanVenkat-fd4hg]

Superr video, Thanks for trying using open source solutions...

[@juliensimonfr]

Glad you liked it

[@ccc_ccc789]

Thanks!

[@juliensimonfr]

You bet!

[@jingqiwu2865]

Thanks Julien! very nice video. very curious if there are some compare between bge-small with ada-002 when used in RAG.

[@juliensimonfr]

Hi, please check our embeddings leaderboard at huggingface.co/spaces/mteb/leaderboard. ada-002 is #15, bge-small is #8 :)

[@GeigenAkademie]

Thanks Julien, for the good tutorial! Some use pinecone, do you see differences/advantages of using faiss over pinecone? Thank you

[@juliensimonfr]

FAISS is a simple lightweight open source solution. Pinecone is a fully managed, closed source DB running in the cloud. Depends what you're looking for, and how much work you want to do on managing the solution :)

[@aishwaryakumar6504]

Hi Julien, Thank you for this video. It's helping me learn a lot. I was trying to run the code. When I attempt the zero shot example, my output is quite different from whats shown in the video. I tried to split it, but I get something like this - [answers: * 1) The trend is to invest more in solar energy in China. * 2) The trend is to invest less in solar energy in China. * 3) The trend is to invest the same amount of money in solar energy in China. * 4) The trend is to invest more in solar energy in the United States. * 5) The trend is to invest less in solar energy in the United States. ] Can you please explain why this is happening and how it can be fixed?

[@whemmakatatt5311]

godlike

[@krishnasunder9491]

thanks it was really informative, can do demonstrate fine tuning LLM's with lora and Qlora? In your experience, RAG has better performer over fine tuning ?

[@juliensimonfr]

Llama2 fine-tuning with Qlora: kzbin.info/www/bejne/kJbZZ3lmiZZ_abs. IMHO RAG and fine-tuning solve different problems and are complementary. RAG lets you access fresh company data and gives you some domain adaptation. Fine-tuning gives you better domain adaptation and lets you customize guardrails and tone of voice.

[@anserali551]

Sagemaker with langchain streaming option is generating output

[@coolcurly9736]

It throws : KeyError: 'Blocks' after running the cell with boto3.client('textrac') thrown by the loader.load(), from parser in langchain

[@Invincible615]

Thanks for the tutorial.In my case,i can't use Mistral somehow due to some restrictions on AWS test account.I have used FLAN-T5 but it is giving this error.ValueError: Error raised by inference endpoint: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received client error (422) from primary with message "Failed to deserialize the JSON body into the target type: missing field `inputs` at line 1 column 503".

[@juliensimonfr]

The input format for T5 is quite different, so sending a Mistral-formatted message won't work. Not sure what restriction you're facing, but maybe TinyLlama would work? I think you would only have to adapt the prompting format in the content handler.

[@Abhisekgev]

I want to embed large data. In this case, if I want to embed document without a GPU notebook ml.t3.medium, is it possible to deploy the embedding model as well in some ml.g5.large GPU instance to make the processing faster?

[@juliensimonfr]

Sure, it's what you would do for production.

[@Thirumalesh100]

Great video, But what if user question is related to chat history and it may contain short cuts like he/she/that/it etc then how to handle such cases

[@juliensimonfr]

Langchain has different ways to handle this, e.g. python.langchain.com/docs/modules/memory/types/buffer

[@Thirumalesh100]

Thanks@@juliensimonfr , Basically it is question rephrase request by passing entire chart history, tried this approach which has cost and token limit problem Looking for other alternative for the same

[@rnronie38]

can you tell how to get the key for sagemaker to work here?

[@juliensimonfr]

Not sure what you mean. Are you looking for a SageMaker tutorial ? See docs.aws.amazon.com/sagemaker/latest/dg/gs.html

[@kevinngo3722]

Hi Julien. The code is not working when I try to run it. I think the error I am getting is related to Sagemaker credentials. I made an account just now but don't know where to get information where I can plug into your code to make this work.

[@juliensimonfr]

Start here: docs.aws.amazon.com/sagemaker/latest/dg/howitworks-create-ws.html. Create a notebook instance and make sure its IAM role includes the SageMakerFullAccess and TexttractFullAccess managed policies. Once you've done that, the notebook will run as is.

[@kevinngo3722]

Thanks for your reply! It seems that this leads me to make a Jupyter notebook. How do I integrate this to do what you're showing on Colab in the tutorial?@@juliensimonfr

[@da-bb2up]

Thx for the video :) can you update your vector database by a few lines ( if you want to add data to your knowledge base) automatically by running a python script or something like that?

[@juliensimonfr]

Sure, you can keep adding embeddings anytime you want.

[@da-bb2up]

oh thats nice :) thx for the answer@@juliensimonfr

[@SebastienStormacq]

Thank you Julien - this is super useful and comes at the right time during my writing season (you know what I'm talking about :-) ) As someone else mentioned in the comment, I also received an error when calling Textract. I solved it by adding `pip install amazon-textract-textractor -qU` - hope it might help others

[@juliensimonfr]

Ok, good to know. Thanks Seb and good luck with the writing ;)

[@SebastienStormacq]

also 'pip install pip install faiss-cpu' :-)

[@rnronie38]

how can I call onto my react frontend?

[@juliensimonfr]

A SageMaker endpoint is an HTTPS API, so you can plug it in anything. You should be able to find lots of examples out there.

[@debojitmandal8670]

Y r u deploying first in sage maker

[@juliensimonfr]

Because I don't want to manage any infrastructure :)

[@Azazello1482]

Seems like a great video, but I can't move from the starting line. You seem to be skipping over very important details about how to deal with the HuggingFace tokens, AWS security keys, regional compatibility settings with Sagemaker, etc. For example, when running the copied SageMaker code, I get "ValueError: Must setup local AWS configuration with a region supported by SageMaker", but no region seems I try seems to work. Did you cut all the authentication code from your demo? Obviously you don't want to disclose security keys, but at least show/explain that part of the setup code and simply redact the sensitive information.

[@juliensimonfr]

How about going through Hugging Face 101 and SageMaker 101 first?

[@Azazello1482]

@@juliensimonfr Yes, clearly I'll need to do this! Nonetheless, as an educator myself, I think my point is still useful. It's helpful to learners to mention parts that you skip over. You don't have to teach it in this video, but it would be helpful to mention that there are steps one must perform that are not shown in this video.