Create Custom Tools for Chatbots in LangChain - LangChain #8

Agents are one of the most powerful and fascinating approaches to using Large Language Models (LLMs). The explosion of interest in LLMs has led to agents becoming incredibly prevalent in AI-powered use cases.
Using agents allows us to give LLMs access to tools. These tools present an essentially infinite number of possibilities. With tools, LLMs can search the web, do math, run code, and much more.
The LangChain library provides a substantial selection of prebuilt tools. However, in many real-world projects, we'll often find that there are no tools that quite fit our requirements. Meaning we must modify existing tools or build entirely new ones.
In this video, we will explore how to build custom tools for agents in LangChain.
📌 Code Notebook:
github.com/pinecone-io/exampl...
🌲 Pinecone article:
www.pinecone.io/learn/langcha...
👋🏼 NLP + LLM Consulting:
aurelio.ai
🎙️ Support me on Patreon:
/ jamesbriggs
👾 Discord:
/ discord
00:00 LangChain agents and tools
01:46 What are LLM tools
03:12 Code notebook setup and prerequisites
05:58 Building a simple LangChain calculator tool
05:50 Initialize the conversational agent
10:08 Updating agent prompts
12:14 Building tools with multiple parameters
15:40 Helping ChatGPT understand images
23:05 What else can LangChain agents do
#artificialintelligence #langchain #openai #nlp

[@alperakbash]

Perfect video, thank you so much. Anyone making use of langchain should watch this video to see the opportunities and how easy and simple it is to implement.

[@rhiteshkumarsingh4401]

Such a freaking coincidence!! I was working on making a custom tool myself and was getting stuck and here you are with a video dedicated to it. Its my lucky day! Thanks a lot!!

[@Alain.Robert]

If you're trying to build something with LangChain, James is always one step ahead and showing us the way.

[@jamesbriggs]

haha good timing, good to hear it helps

[@nesdi6653]

same

[@jidun9478]

Thank you for all the insight. I have watched quite a few of your appearances across the web dealing the AI memory spectrum etc and just wanted to say thanks for your work in this matter 😃

[@mysticaltech]

Super interesting, and now a lot of videos dive that deep! Thanks James, really appreciate it.

[@subodh.r4835]

AMAZING work. Can't wait to build something incredible with this.

[@jamesbriggs]

awesome, you should share it with us in discord or twitter when you do :)

[@emanahmed4394]

really fascinating to see how AI agents can be integrated with custom tools for language models. Your Jupyter

[@code-grammardude5974]

This is SOOO cool. Can't wait to get going with this. Integrating this even with home automation APIs would be epic

[@0ptim]

Thank you so much. Your videos are invaluable! Please more content about LangChain, agents, tools, and everything around this topic.

[@jamesbriggs]

100% will do

[@init5god]

This is amazing James! Thanks for sharing so much. This open a very big door for me and the exploration I'm currently working on using AI models

[@jamesbriggs]

that's awesome, glad to hear!

[@nicolasstegmann5855]

Thanks a lot for your work James. It has been really useful for me.

[@mikemansour1166]

really fascinating to see how AI agents can be integrated with custom tools for language models. Your Jupyter Notebook tutorial was super helpful and I appreciate the time and effort you've put into creating this content. As a beginner, I'm finding it a bit challenging to deploy these AI chatbots, especially when it comes to integrating them with a GUI, like React or other tools. I would love to see a video on that topic, as I believe it would be beneficial for many of us who are new to this field. Thanks again for the amazing content, and I look forward to learning more from your future videos!

[@easzyprogramming]

streamlit

[@mikemansour1166]

@@easzyprogramming Thanks for the hint

[@robxmccarthy]

Appreciate your work - you're a great teacher.

[@hiranga]

Legend! Great stuff James🙌🏾

[@Kevgas]

Awesome! Thanks for sharing!

[@dusanbosnjakovic6588]

Great videos! I would love to see a video on different agent types. This area is not covered that well out there but that seems to be the biggest difference between different awesome systems that are coming out like baby GPT and hugging GPT. I'd love to know how to plug & play them and what exactly is the difference between them.

[@Mankind5490]

Thank you! very easy to follow code

[@user-df4wr8li3e]

Great videos!

[@crisgath3512]

I love your content and have learned so much from your work. And those shirts...!! I love the collection. 🏅

[@jamesbriggs]

thanks man!

[@ArchAid1]

I LOVE this! 🤩

[@Evox402]

Hey James, thank you very much for your videos :) They really helped me to get a basic understanding in the ever evolving GPT jungle, keep it up! But I also have a Video request: Could you explain how these Tools/Agents work behind the scenes? Like: - What is the final prompt that is issued to the GPT (with available tools, formatting etc.) - What is the answer from the GPT? - How is the answer parsed to identify that a tool should be used - How are the tool parameters extracted from the GPT answer and given to the actual Python code - How is the answer given back? - Maybe, how could you make it so certain Agents talk to each other before returning to the main GPT prompt? It looks like the GPT is instructed to indicate tool usage by using json format? But I'm not sure. Maybe an in-depth explanation would help clear the fog for me (and possibly some others) Thanks! :)

[@midnightmoves7976]

Nice, thanks

[@shamaldesilva9533]

Dude your reading our minds 🤖 🤯

[@memesofproduction27]

Got that early access to GPT-5 ESPplugin

[@chocky_1874]

thanks

[@AI_Financier]

Thanks for the info, on image captioning, why are we plugging it into the LLM, what does the LLM bring on?

[@AdrienSales]

Very kind intro to tools, now I know how I'll release mine : would there be like a pypi repo for langhcain tool ?

[@lordmelbury7174]

This is one of the most useful videos I have seen on AI. *I had an error with latest version of LangChain, but this solved it: pip install --force-reinstall -v langchain==v0.0.147 One thing that I am thinking is to be able to accept lightning payments per search, eg micropayments of Satoshis per search to make a public facing site.

[@magick93]

Regarding tools with multiple inputs, if not enough inputs are initially provided, can the chat agent ask the human to supply more info?

[@PazLeBon]

im not sure ive actually seen a single tool yet that has actually made me more productive, i just write faster

[@Cbon-xh3ry]

Hey James very similar to what we’ve seen in the past as far as I can tell using tools. Quick question to you, how does langchain works internally with the new prompt and the tool it has, does it make it so when it asks ChatGPT that one returns a completion that says it can’t do it and that’s what langchain looks for to know it needs to use tools in its list ?

[@olivert.7177]

The format the response should have and the tools with its descriptions are in the original prompt that is given to the LLM. The LLM then return the instruction to use a tool and the parameters as the output. Langchain only checks the response of the LLM if it needs to forward it to a tool.

[@peterball8900]

Can we stack agent on top of each other just like agent as a tool ? Please suggest

[@RedCloudServices]

James can you make a video with custom chart with x or y axis as inputs?

[@Bubbalubagus]

But how is the LLM "using" the tool, how does it break its text output to switch to using a python function and then switch back to text?

[@ScottzPlaylists]

Video Idea: I would like to see a video on how to classify files from the filenames ( video title from youtube) in an input folder, and put them into a different folders subfolders where the name of the subfolders is the classification list to sort them into. (ex. AI, AI LLMs, AI Art, AI Tools, Health, FPGA, Crypto, etc) I'll use this to classify videos I've downloaded, into categories that I'll watch later by category. = A similar thing would be to classify screenshots from videos I watch into a similar categories of subfolders like above example, but the text to classify would have to come from OCR of the Images because the filename is not descriptive.

[@chuanjiang6931]

Is there a way to let agent return what _run function returns?

[@6ptech-vy3uc]

Can you create some videos using Apichain and Chainpal to interact with external APIs?

[@kevon217]

very cool. orangutans are apes though, not monkeys. any ideas on implementing some sort of double checking to verify a model’s prediction and confidence in this type of application?

[@JonathanYankovich]

Tangentially related; pre/post prompts use tokens and tokens are precious in 2048 context length. Are people talking about how to make per-prompts more compact? Could you do a video on how to write compact pre-prompts or if this is even important? In your example, you say “Unfortunately, assistant is terrible at maths.” Could tokens be saved if this was written as “assistant must not do math, use tools instead” or “assistant always uses tools to do math”?

[@ijhake]

It's not Langchain related, but what tools are you using for drawing your illustrations?

[@jamesbriggs]

excalidraw - it's really good and I think they still have a free tier

[@RatafakRatafak]

Why don’t we have to edit the sys_msg for image captioning task, if we had had to do it for math calculations?

[@josephturbin9087]

Probably because the LLM knew it didn’t have that ability. With the math example the LLM was confident that it could do the math by itself. With image captioning, it knows it needs help.

[@RatafakRatafak]

@@josephturbin9087 How do I know in general which task the llm is confident about and which not?

[@josephturbin9087]

Great question that I’m only guessing at an answer for. My guess is that each model has it’s strengths and weaknesses and you’d need to consult the model documentation or videos online to find out what they are. Or maybe you experiment with the models and come to the conclusion yourself that it needs a tool to handle the task because the answer it’s giving is wrong. Kind of like the math example. It spit out the wrong answer so clearly it needs help from a tool and you might need to remind it via the system message.

[@jamesbriggs]

yeah what Joseph said - it's less confident in it's ability to caption images, so it's easier to avoid hallucinations with that tool Best way to figure it out is to test, it's very hard (if not impossible) to predict how an LLM is going to behave, so testing is the way to go

[@bolivianprince7326]

How to set Scrapy as a tool instead of the default SerpApi?

[@DavidRodriguez-sh9cq]

Is it possible to create an agent with an input? Let's say a pandas dataframe? I know that there is already a pandas dataframe agent, but it does not allow to modify its prompt.

[@buksa7257]

i actually have problems letting the agent NOT use tools when it's not necessary. He just always wants to use tools for everything. I dont have that initial prompt with "assistant is a large language model trained by open ai..." Where can i find and copy this text?

[@exclusivemedia3889]

Join the $Psyop coin community and be part of a movement that's pushing the boundaries of psychology and blockchain technology. Together, we can drive meaningful change in the mental health landscape. #JoinTheMovement #PsyopCoinCommunity"

[@RatafakRatafak]

So what this is basically doing is expecting alignment between tool’s description and user input for the chatbot.

[@jamesbriggs]

yeah that's a good way of thinking about it

[@alii4334]

the name `chat-conversation-react-description` confused me a lot as React developer

[@jamesbriggs]

ha yeah I didn't think about that 😅

[@chuanjiang6931]

If you change 'radius' to 'diameter', it still got 49.07mm.

[@texasfossilguy]

youd have to change the formula, its just creating a parameter named Radius, you could call it Flubflub and itd still process it the same. Remove the "2*" radius to change the formula to be for diameter.

[@frankbee9831]

Human: How can I distract from my baldness? AI: Put on a shirt with a low neckline so that your chest hair is clearly visible and accentuate your masculine beard growth.

[@jamesbriggs]

I will say - I did start wearing the shirts around the time chat-gpt came out

[@frankbee9831]

@@jamesbriggs I was just trolling around. It looks great! And you are a very handsome guy .. and apparently very smart too :) Thanks for the good content!

[@stanTrX]

James, it started good but then gets too complicated and difficult to follow

[@StephenRayner]

You interested in collaborating?

[@dfadir]

Poor LLMs, we have to gaslight them so they use the tools haha Forgive us Machine God

[@jamesbriggs]

Pray GPT-5 have mercy on us 🙏

[@dfadir]

@@jamesbriggs 🙏

[@ScottzPlaylists]

I have a gread Idea you could code. Here's my idea for "Offline AI DVR for KZbin" Have an input folder of Files to Classify by the filenames that are the title of the youtube video called Input_Folder Have a Output_Folders directory, where the Subfolder Names under Output_Folders are the Classes to train a neural net, to specify where the files should be copied to. I already have this setup and lots of youtube videos already manually moved into the Subfolders. This could be used to train the classifier. once trained I'd like to run the classifier on each filename in Input_Folder, the classifier would tell me the folder name to copy it to (I would call it Folder_Destination). Python would copy the file to \{Output_Folders}\{Folder_Destination. This would be an awsome little AI to organize files when you already have started the organizing. I use a nice little tool called "WinX KZbin Downloader" to get the files I've put in a public folder named like "12"(the Date I started a new download list) as I browse KZbin, I "Save to Playlist" and put it in "12" to put on my HD Later. WinX even puts them in a folder called "12 in this example. Then when I'm ready to Download and organize I copy the URL of the Playlist, paste it into Winx and it gets all the video files and puts them in "12". Then there's the long process of organizing them into categories on my HD like described above, so I can view Videos by catagory at my leisure. ( I used to be Offline most of the time, there was a method to my madness to get lots of videos for watching later). See all my Playlist names, and you can see I have a lot of interests. I keep the best videos by category on youtube playlists. It would be so nice to have the program I described and it would be the main component in a larger program later that would be like an Offline AI DVR for KZbin. Thanks for reading this long post, hope you whip up the code as a better programmer than me!