I always considered asking LLM to spill out the python code directly with using whatever functions you want it to use works much more native, robust and reliable, than taking this detour with JSON-here-is-how-you-call-the-tool thing. The only small downside is to think in advance on how to run the LLM generated code in a safe sandbox, so it won't do any crazy shit. But there's a plethora of options for that

great description

When I run the following script: from haystack.document_stores import PineconeDocumentStore document_store = PineconeDocumentStore( api_key='**************************************, index='myindex', similarity="cosine", embedding_dim=768 ) It gave me an error like this: ApiException: (400) Request failed. You've reach the max pod-based indexes allowed How to fix that? EDIT: I have only one index, and even if I delete it, I still get the same error.

Thanks James. Can the assistant ingest scnned pdfs (require OCR) as well as other file extensions like .doc, .pptx etc?

You are awesome ❤

9:54 I don't get where the 9.8 million comparisons in every attention layer comes from. Is every patch compared to every patch? That would be 256 patches => 65536 comparisons.

Congrats on the son!

nice video! I've been implementing a version of this myself.. this could be quite useful

Is this the samething as gemini 1.5 deep research ?

Retrieval Augmented Generation (RAG) has dominated the discussion around making GenAI applications useful since ChatGPT’s advent exploded the AI hype. In recent evaluations, GraphRAG demonstrated its ability to answer “global questions” that address the entire dataset, a task where naive RAG approaches often fail. "We need an alternative retrieval method to demonstrate its ability to answer 'global questions' that address the entire dataset, a task where naive RAG approaches often fail in modern AI Applications"
 Welcome to Graph RAG…! GraphRAG, Outperforms traditional RAG ( Retrieval-Augmented Generation ) for Query Focused Summarization This book is for programmers, researchers and developers who are interested in LLMs techniques and advancement for Generative AI specifically the recent GraphRAG: Open-source research of Knowledge Graph to support human sense-making, improving the accuracy of data discovery, solving RAG pain points, and to enhance LLMs ( Large Language Models ) ***Including Case-study of PDF AI Chatbot using Python 
Available On Amazon www.amazon.com/dp/B0DJB2N5T3

great video - thanks!

Thanks! Google Search and citations are great features! :)

📌 Code: github.com/aurelio-labs/cookbook/blob/main/gen-ai/google-ai/gemini-2/web-search.ipynb

it was very clear thank you man you are the best

*Overview of Google's Gemini 2 Model and its Multimodal Capabilities* * *0:00** Introduction:* The video discusses Google's new Gemini 2 model, highlighting its potential to be a strong competitor to OpenAI's models. * *0:41** Focus on Agents:* Gemini 2 is designed with a focus on agent use cases, emphasizing its ability to produce structured output effectively. * *1:53** Running the Code:* The presenter demonstrates running the code locally and provides instructions for running it in Google Colab. * *3:08** Describing Images:* * The model accurately describes underwater scenes, identifying various fish and coral species. * It successfully identifies a camouflaged fish that was difficult to spot. * The model sometimes misidentifies objects, particularly corals. * *9:29** Image Bounding Boxes:* * The model generates bounding boxes around objects in images, providing labels and coordinates. * A frequency penalty is used to prevent the model from repeating itself excessively. * The model is able to label various fish species, even those partially obscured. * The presenter noted improvement in the model's accuracy over a short period, indicating it might be under development. * The model struggles with identifying corals accurately. * It was able to identify a hidden fish in a shipwreck's turret. * *21:06** Example 2:* A large fish in the center of the image is correctly identified as a "painted sweetlips" whereas it was not identified in a previous run. * *23:29** Example 3:* A complex scene with many fish. * It's able to identify fish in the distance that are not clearly visible. * Identifies a black fish with a distinctive tail as a "Nazo Lituratus" - which may be related. * *26:52** Example 4:* * Able to identify the scene as a shipwreck. * Correctly identifies a cylindrical object as a likely gun barrel. * Unable to determine a specific location for the shipwreck. * *29:09** Image-to-Text:* The model can describe images and identify objects without explicit instructions to draw bounding boxes. * *30:50** Google Gemini vs OpenAI GPTs:* The presenter expresses optimism about Gemini 2's potential as an agent and its ability to compete with OpenAI's models in real-world use cases. I used gemini-1.5-pro-exp-0827 on rocketrecap dot com to summarize the transcript. Cost (if I didn't use the free tier): $0.03 Input tokens: 20215 Output tokens: 547

I don't find this model really impressive for object detecion. Florence2 can already do a similar job and it is under 1B param model. For real world case, I would not trust prompt engineering to get my results. Rather I would prefer to fine tune the model. It is also a nightmare when google does some tweaking on the model as you were experiencing. I'm also experimenting grounding with Qwen 2VL as they have <|object_ref_start|>, <|object_ref_end|>, <|box_start|>,<|box_end|> tokens specifically for object detection. Thanks James for the update 🙏

I need to rethink my AI workflows-this model offers many new opportunities.

Thanks for the Demo. An interesting model for sure, but anything non open source is not suitable for enterprise use. Not now, not ever, especially since even models that are even tagged as "Appropriate for Enterprise" go through a lot of changes and have their instructions changed while being live. It's an absolute nightmare to work with.

I've been saying this for months. Building LLM-centric agents is extremely limiting and dramatically reduces reliability. You don't need o1 to build good agents. GPT-4 level does a good job when you break The problem down into smaller pieces and only use LLMs in places where deterministic code can't do the job.

Amazing!! Thanks James! Always explaining it really clear!

Such an important topic in whole of computer science still only 30k views

Hi James, thanks for beautifully explaining the idea of ViT. May I request a similar video on SLIP as well? I would really appreciate it. Thanks in advance.

It would be good to have some examples to test or see the real life results, like when you did the 75% discount example, I could see how we can semantically block ANY discount to be proposed by the llm, but will it work if we want to allow max 10% discount instead?

Thank you for such a good explanation

James, how does the semantic router behave for long user queries? For example, the user's request to `check the weather` is in the middle of the long query, so it can miss the function calling because the score is pretty high. But a lower score will hurt short queries.

Loved this video! Would be great to have more theoretical videos like this one.

Great video. I agree with James Briggs. I recently submitted a RAG Idea to my AI Idea Evaluation Team. The idea to do RAG on our massive library of PDF technical documents available in the public domain. We did the Retrieval and the Augmentation of our LLM but when I started testing and prompting (asking questions) to our LLM (Chat GPT) the answers were lots to be desired and I was somewhat disappointed. I learned a lot that and realized that perhaps we were not really chunking the PDF documents into a Vector Database and my Prompts had to somewhat juice the LLM. After several prompts the answers got better and better. I was almost loosing interest in the whole effort. Then I remember James Briggs and still pursing RAG.

Great video! However, Doesn't the latest version (0.0.72) use RouteLayer instead of SemanticRouter?

Best video I have seen on embeddings

Not safe for work and safe for work if you have been wondering

thanks <3

such a cool idea - and it does exactly what it sounds like it needs to do. will be very useful for tool use and llm routing

Would the semantic router work if the questions are written in different languages?

I'm doing something very similar in Golang, manually handling the creation of the system prompt. The hardest thing is to get the models to simply obey the output format! For instance, a bot that fills out forms; acting as the secretary at the front desk of an office. It's instructed to use SAY, SET {fieldName}, SAVE; and its job is to interrogate the user about a list of form fields. So, the system prompt has injected into it: []{fieldName, fieldFormatExample, fieldCurrentValue} ... and it interrogates you in whatever language you are speaking. The chat history accumulates existing values. In the end, I have a Go program that loads up a big xml file that contains all the assets such as prompts, html templates, config. The main thing is getting back responses like this: SAY We are setting your full name to Robert Jason Fielding SET FirstName Robert SET MiddleName Jason SET LastName Fielding SAVE SAY What is your phone number? ie... filling out form fields in bulk; one round of conversation; easily switching languages. I don't use LangChain when I am using Python; but it seems like those libraries are handling a bunch of miscellaneous stuff to read and write random data sources. I wish there was a way to make the output format completely mandatory; literally impossible for it to insert conversational junk that breaks the parser.

awesome!

I am struggling to understand how the search would work if we needed to handle some kind of structured data. Anyone having thoughts? I would appreciate your assistance

In building agents, which is preferred; langgraph or crewai ?

RAGAS must go. I dont really understand why such weak method promoted so much.

Thank you for the video! The notebook seems to have been removed from the repository. Is it still available?

If you want to go deep into the rabbit hole of Neuro Symbolic AI there is some interesting recordings here www.youtube.com/@asimmunawar/videos

Very Helpful, subscribed !!

The font at the beginning is hard to read

Too much talk and very few illustrations

Love your videos and wlakthroughs - really easy to understand!! thank you james

By the way, do check out Max Tegmark's recent paper "The Geometry of Concepts: Sparse Autoencoder Feature Structure" - points in the same direction

I've loved embeddings for a long time, so very powerful for semantic AND analogical search. But their power relies on the quality of the embedding model. We've been looking at SAE's (mechanistic interpretability of LLMs) as embedders. All concepts in the embeddings are unique (sparse_, and decomposed, ie Interpretable embeddings from an LLM. Goodfire intend to make this into a service, do take a look at their Goodfire SDK, specifically the Latent Explorer. Their SDK is new this week for the ApartLabs Hackathon tomorrow.

Just curious if there is a way to pass required param option to the function schema definition and fail the function call if not provided.

Not a great description of what's going on

Great RAG walkthrough! With Web3NS, you can combine this tech with YourName.Web3 to create decentralized AI agents that integrate seamlessly with Web3 ecosystems. 🚀

Are you suggesting the image encoder never have seen any cassette player during it's training and still it has formed an understanding of it in the latency space? I would disagree to this.