Try out the code here github.com/pinecone-io/examples/blob/master/learn/search/semantic-search/openai-embed-v3/openai-embed-v3.ipynb Testing the 256-d embeddings here! kzbin.info/www/bejne/mIicZGSnfbGJZq8 (they're very good)
@leonardotato30678 ай бұрын
"Hi James, I've been a fan of your videos for a long time and they never cease to impress me. Your dedication is evident in every piece of content. As the Director of a technology consulting firm in Spain, where we're venturing into creating specialized chatbots for information search, your insights have been invaluable. Keep up the fantastic work!"
@jamesbriggs8 ай бұрын
Hey Leonardo, that’s really awesome to hear, thanks and good luck with the venture!
@IvarDaigon8 ай бұрын
Side note: The models with "ada" in them were named after Ada Lovelace so they should be pronounced the way her name is pronounced by most people in the UK at the time. Open AI started with model codename Ada Lovelace (GPT-1), then Charles Babbage (GPT-2), then Leonardo Davinici (GPT-3) but they stopped when they got to GPT4 or the letter E. They could have kept going with Einstein, Faraday, Galileo and then if they got stuck they could have just asked GPT-4 to create a list of codenames for them. My guess is that the embedding-3 models are based on GPT-3.5 which would be good balance between speed and language abilities but who really knows for sure because they keep changing the naming convention almost every time they release a new model.
@ernestosantiesteban63338 ай бұрын
Great video! You always give us recent information. Thank you for your work.
@jamesbriggs8 ай бұрын
You’re welcome, thanks for watching :)
@jp007388 ай бұрын
Yo James, thanks for the deep dive! One thing I felt missing was showing the models performing on bigger dimensions as well. I bet this would generate some awesome responses. Guess openai wants to do that "apple marketing move" to compare models, but to be honest less than 1k are still a bit drunk answer style maybe? 😂
@haneulkim49026 ай бұрын
Great video as always. One question, how it the embedding model trained? Are the embeddings simply extracted from chatGPT4 or are they trained differently from the beginning (pre-training stage)?
@jawdridi8 ай бұрын
why the similarity search scores for a given query with the large and small embeddings are lower than the old OPENAI embedding?
@jamesbriggs8 ай бұрын
They just have a different range, actually Ada 002 was weird because the range was so small all the time - these models seem to have a larger range that tends to show lower similarity scores, you see the same in many recent open source models and Cohere’s embedding models
@jawdridi8 ай бұрын
thank you for the clarification. I love your videos. Keep it up @@jamesbriggs
@dariuszsemba8 ай бұрын
I think "the revolution" in embedding models went unnoticed for a reason :) It's hard to pinpoint any disrupting method/technology which would make embeddings much more useful whereas ChatGPT had its moment thanks to RLHF method. Actually, vector databases gained all this attention a few months after ChatGPT's gigantic success - to me that proves how vector databases and embedding models are just about the ongoing hype.
@micbab-vg2mu8 ай бұрын
Thank you for the update _ I was not aware about a new embedding model.
@jamesbriggs8 ай бұрын
They released them last night :)
@micbab-vg2mu8 ай бұрын
Great - ada model is quite old:) I will test new models during the weekend. @@jamesbriggs
@vladif2518 ай бұрын
Great overview. Thanks James
@jamesbriggs8 ай бұрын
You’re welcome!
@josephbeau-reder8138 ай бұрын
Thank you for the video man! I am quite surprised about the "qualitative" analysis of each model at the end (especially the "compare LLaMaA/GPT4" question) : you indeed check if the model has understood the question (it needs to compare LLaMA and GPT4) but isn't it even more important to check if it the informations provided are correct (and well sourced) ? Because the answer can indeed compare LLaMa and GPT4 but base this comparaison on hallucination (and wrong source).
@MastersWithHarshith8 ай бұрын
Can you do a comparison video between the new embeddings VS Cohere's embeddings?
@mortezalayegh25878 ай бұрын
Thanks for the update. Great Content. 👍👍👍👍
@avi72788 ай бұрын
Great James, thanks I was looking for this. Do you have any videos about indexing and running RAG on entire codebases / projects ?
@luisguillermopardo77928 ай бұрын
omg that's impresive. Greetings from Colombia
@jamesbriggs8 ай бұрын
Hey there 👋🏼
@luciolrv8 ай бұрын
A PINECONE Question: Will pinecone charge us more if we use the 3-large model since the vector dimension is larger? If so will it be more expensive just for upsert or also for retrieval?
@LaylaBitar-z7z8 ай бұрын
Hey James! Thanks for the video. Wondering, can you do a video converting TruLens?!
@jamesbriggs8 ай бұрын
Would love to they’re great! Hopefully soon
@snarfer2938 ай бұрын
It would be better to test semantic retrieval instead of these queries that can also be done with standard word-to-word matching. Ideally, using words that are not in the target text. Also, are there public data sets with good numerical judgments so we can use ndcg to evaluate retrieval rankings?
@jamesbriggs8 ай бұрын
Yeah I’ll test the new models and a few others with good benchmarks soon - this was a quick first look after I heard about the release :)
@snarfer2938 ай бұрын
Is there something wrong with the text extraction? Output at 11:14 has spaces in "vuln erabilities", "organi zations", "sys tem", etc which would create issues in tokenization, and same with 11:53 having "TheseclosedproductLLMsareheavilyfine-tunedtoalignwithuman" as a single string.
@jamesbriggs8 ай бұрын
Yep, but real world data is messy - so I like to test with this and see how they perform
@GeobotPY8 ай бұрын
Thak you! Do you have any videos where you make some quantative analysis on how to evaluate LLMs and RAG? For instance I looked at RAGAS - seems interesting, but I find evaluation of RAG quite difficult to quantify. Keep up the good work!
@jamesbriggs8 ай бұрын
Planning to do this soon - I do have a video on evaluation metrics for retrieval though - doesn’t focus on RAG but still very relevant (any new video I do on ragas will likely incl these too) kzbin.info/www/bejne/eHWchZ6seth-ra8
@IvarDaigon8 ай бұрын
The only way to really evaluate if a model is good for your use case is to start with the largest one that reliably does the job that you want it to do, then create a series of unit tests/benchmarks against that model and then go down the list of models (or embedding dimensions) and run the same tests against each model until the tests break. Unfortunally synthetic benchmarks OR user reviews will not give you the answers you need because natural language is highly nuanced so it depends entirely on the specific use case that you are interested in.
@Lucky9_98 ай бұрын
@@IvarDaigonwow thank you for this pro level template! Filling this away for future reference!
@eyemazed8 ай бұрын
is it just me or does the new "non lazy" gpt4 turbo model take much longer to respond to an API call? (testing with RAG, 20-30k input tokens)
@JohnMcclaned8 ай бұрын
9:30 - why don't you do them in parallel? doing them one by one will always take longer
@jamesbriggs8 ай бұрын
Just writing quick simple code - but yeah I should’ve
@fintech13788 ай бұрын
can we use this for image and video embedding too?multimodal embedding for multimodal RAG
@jamesbriggs8 ай бұрын
No just text, I’m actually surprised they didn’t release multimodal embedding models
@OccamsPlasmaGun8 ай бұрын
Maybe the embedding model stores "Llama 2" at the origin (0.0, 0.0, 0.0, ...) to screw up cosine similarity search.
@jamesbriggs8 ай бұрын
Always replaces competitive model names with outdated model names lol
@JohnMcclaned8 ай бұрын
between this and the new serverless pinecone, get excited.. christmas came late
@jamesbriggs8 ай бұрын
It’s pretty good timing
@dr.mikeybee8 ай бұрын
Mistral 7b scores better. MTEB for Mistral is 66.63. And Mistral isn't the top performer. Why would anyone pay for embeddings?
@jamesbriggs8 ай бұрын
I'm all for OSS embedding models - most of my recent videos focus on them - but the reality is openai api is easy, cheap, and gives you top-tier embedding performance (even if not quite number 1) so it's popular That being said, the 256-dim Ada 002 performance, if true, is pretty impressive
@lpls8 ай бұрын
You pay either way. Running a model ain't free.
@the-us-runner8 ай бұрын
Why? To get embeddings that work across dozens of languages.
@kazwat8 ай бұрын
@@lplsyes it is
@lpls8 ай бұрын
You have to run it somewhere, right? CPU, memory, power...
@AnthonyZboralski8 ай бұрын
Your videos would be much better without your overuse of stock footage