5. OpenAI Embeddings API - Searching Financial Documents

[@parttimelarry]

Like the video? Support my content by checking out Interactive Brokers using the link below: www.interactivebrokers.com/mkt/?src=ptlPY1&url=%2Fen%2Findex.php%3Ff%3D1338 I will be starting a spinoff channel on AI in music, art, and gaming in 2023. Subscribe at: youtube.com/@parttimeai Buy Me a Coffee: buymeacoffee.com/parttimelarry Notebook: colab.research.google.com/drive/1tttDqgnWL9yJtmlOFXJqA-BjQ1Pyfpax?usp=sharing This notebook is based on the embeddings examples in the OpenAI Cookbook: github.com/openai/openai-cookbook

[@jonconnor6697]

say vector one more time :)

[@SwingingInTheHood]

Not being an AI guy, I've been trying to understand embeddings and search vectors for the past two weeks. This is the first video I have found that explains it so that even I now get it. Whew! Thank you Brother!

[@hishamafzal1999]

Can you explain me bro?

[@erikstillman7336]

Thanks!

[@alanfeng99]

Extremely well presented of how embeddings work!! I "kind of" knowing how it works, but after watching this video, I can say I "know" how it works! Thanks.

[@robertcormia7970]

This is outstanding, his clarity and enthusiasm is amazing, real passion for this topic!

[@SumedhKadoo]

Thanks for sharing, you are a good teacher. Can you please share how much did this cost in terms of API usage?

[@PatrickAlphaC]

Dank. Love your content dude.

[@parttimelarry]

Likewise man, been awesome to see you kill it over the years!

[@Guy-Scott]

I appreciate the work you have put into this series. I like how you start small, with a few examples and build from there. Great job.

[@PhilMcCarty]

This is a really great tutorial, thank you.

[@parttimelarry]

Thanks so much! Really appreciate it.

[@deepster5150]

How do you buy a drink and get that logo you have?

[@PhilMcCarty]

@@deepster5150 On my screen to the right of "Share" there's a "Thanks" button. It's probably a KZbin red or whatever the paid thing is feature.

[@soesevenonesix]

Another great Video Larry! Thanks for sharing!

[@wmchoi1474]

Thanks for this amazing tutorial. Great video dude

[@Raja-jo5dm]

Nice one larry easy to understand : )

[@blender_wiki]

Your contents are surpassing the simple high economic value and make the world a better place. 🙏🏾🙏🏽🙏🏼🙏🏻🙏🙏🏿

[@rengavasan6245]

This is the best explanation Ive come across about embeddings anywhere on the internet. Great work Larry. Subscribed :)

[@jay-d-x]

Excellent and simple. Thanks Larry.

[@davidw8668]

Your recent video quality again has come in higher than expected - despite high priors, sir😁 damn good teaching and funny. Thanks man!

[@persas1683]

Your tutorial help me alot. Thank you 🍾

[@iamaguest2]

very much appreciated for the simple introduction. Very good at explaination of something that is not easy to understand, great job

[@mariusirgens5555]

The espresso-milk -> latte example blew my mind 😄

[@sivalokesh3997]

I searched for subscribe button then

[@Jberv]

Man! I just watched your video like its 5 minute length, very detailed. It took me so long to find this kind of sample/explaination. Thanks

[@parttimelarry]

Thanks, this video wasn't very popular when I first put it out, but I think people appreciate it now :)

[@karansajnani1135]

This is by far, the best video explaining embeddings! Thank you

[@kenchang3456]

Thanks, Larry for a great video and explanation. Being an old former developer I am so fascinated with these LLMs, and your explanation has helped me organize how I will do a POC. I really appreciate you sharing your knowledge and enthusiasm.

[@nedyalkokarabadzhakov5405]

quality content deserve subscription. Thank you.

[@rhemagenesis]

You came with the HEAT 🔥 🔥 🔥

[@private_0]

Amazing video! Loving the series so far. Thank you for doing it!

[@ThomasODuffy]

Great video and explanation - thank you!

[@seandiamond7983]

Bro you're the fkn man. I would have never known about this. I just created a search field in my Django website and it will be getting a massive upgrade now lol thanks again bro.

[@lqvision]

very patient explaination!

[@munoncode]

thank you so much for this tutorial.

[@aecs71]

awesome video, was struggling to understand embeddings for a long time,, this video is superb

[@carlhlazo5907]

Awesome tutorial. Using familiar concepts made it easy to follow. I was finally able to understand much better what these concepts are and how they all relate.

[@thomastudu2977]

This is exactly what I needed man, Thanks !!!

[@ratral]

Thanks for the video; it is pretty straightforward and well-developed. 👍

[@shanthu7803]

Happy New Year! Great content learning a lot from you. Thank you so much for detailed explanations step by step guidance. Wish you all the best!

[@rishabhpanwar8971]

Superb content. Thanks!

[@laura43400]

One of the best explanations out there!

[@bakari007]

Great Video ! Super interesting how this transformations works words to vectors ! I'm wondering how the architecture of the classifications works ! Thank you for the great quality of the video and the super interesting topic !! Don't stop !! 🥰

[@testadrome]

Awesome video! Keep them coming

[@jessexing3456]

amazing video! how smooth it is!

[@deep2mixer]

Awesome. Disrupter R' Us. This is some high quality material.

[@TomanswerAi]

Great video nicely explained

[@businessdissection130]

After weeks of lurking around, god showed me this video! Never going to regret this. This is exactly what I wanted. And I would say this was even better than the documentation on openAI, since it always gave me an error!

[@parttimelarry]

Thanks for the comments, glad this was helpful!

[@HazemAzim]

Very nice and soft explanation 👌 .

[@Naijarianman]

These videos are awesome, thanks a lot

[@manuthvann7560]

thanks for sharing such informative content , I've learned a lot from you, and Kepep doing a great job

[@hb_dan2928]

thank you, really good and concise explanation

[@iamnoone7415]

I find your tutorials on openai to be very informative and easy to follow. The instruction explained the concepts well and used examples to help illustrate the topics. Even for someone who does not have a coding background, I find the tutorials very engaging. However, I have a question about how to overcome the rate limit error.

[@michaelwang6831]

hi, have you ever been able to fix the rate limit error? I am facing the same issue..

[@jskweres2]

@@michaelwang6831 I have the same issue. Advice on fixing it?

[@frutenfratenfriends]

This is impressive brother thanks

[@serhatkoroglu6708]

you telling very clear thank you!

[@MikeBtraveling]

"The inflation is too damn hight", was great.

[@processusAI]

very well done video. Fit exactly what I wanted to find out

[@psychurch]

Thanks man, super clear explanations! Subscribed and excited to see what you’ve got coming up for music and ai ❤

[@GonzaloRMDT]

Great job!

[@marshall3900]

Greate video!

[@charyog7]

thank you very much, it was very clear !

[@patriciodiaz2377]

Thanks so much! 🎉

[@breakoutgaffe4027]

When it comes to Python tutorials Larry you are the 🐐

[@undrash]

Amazing content, thanks

[@omkarajagunde4175]

Wow awesome man

[@FRANKWHITE1996]

thanks for sharing

[@lastemperor1347]

Amazing!

[@avbendre]

you are amazing!

[@Nashadelicable]

8:36 "this in the Matrix is how a hot dog looks like" XD

[@utuberay007]

Awesome..

[@BenjaminRay]

Very cool!

[@Malte.Research]

Awesome, thanks !

[@sachin-b8c4m]

very helpful thank you

[@camuelcoleridge479]

@8:36 Part time Oracle is exposing the matrix 😂

[@neelcansharma]

Incredible video, you're a natural teacher. Do you know how embeddings is calculating these numerical vectors? Is it comparing to other items in their dataset?

[@charlieevert7666]

Bad ass, thanks!!

[@xiangyu326]

This is super.

[@guillaumeberrido]

Excellent! Next step, would be to store our data in a Vector DB like pinecone instead of a csv...

[@user-wr4yl7tx3w]

Would be interesting to see how such word embedding is even computed in the first place. possibly with a toy example.

[@amanmeghrajani1]

great vid

[@kovidkasi6117]

Where you are creating embeddedings for text data, is it just embeddedings of each word in a text combined together or are you taking sentence embeddings?

[@VeeraForBharat]

For the last part of video, you just sounded like Jerome Powell. Great video to understand Embeddings API and how its useful for Cognitive Search. It looks like the Search term and every other sentence in the knowledge base has to go thru the Cosine Similarity check. Doing this on a small document is computationally ok but how does ChatGPT and others do this on Petabytes of data?

[@MrYomantanepali]

earned my subscription

[@HazemAzim]

One question is the effectiveness of embeddings varies with the size of the text? Embedding a word or phrase or sentence versus embedding a sizeable paragraph? Would this affect the accuracy ?

[@chenpaul99]

Excellent demo! When using embeddings for training, how do I feedback the graded results to improve the model?

[@michaeldebellis4202]

When you do the dot product, doesn’t one vector need to be a column vector rather than two row vectors as in your example? My linear algebra is rusty so perhaps I’m missing something but as I recall you can only multiply (dot product) an M x N matrix with an N x M matrix(?)

[@michaelwang6831]

Question - for the Microsoft earning calls example, how do you segment the whole speech into different paragraphs as a unit to run through the AI algorithm? What is the logic? or you simply segment them based on comma and period punctuation?

[@lucas.souza2022]

I would like to know that too, did you come to any conclusions?

[@alexchanchan9359]

Thank you very much. Can I ask why in the notebook, the earnings_df has not used ".apply(eval).apply(np.array)"?

[@TommyInGame]

This is so huge. Currently creating a “digital employee” product to limit human capital required for SMEs and startups, and this is a game changer. Would you be open to be a potential beta tester ? I’d love your feedback👍 great vid as always

[@Konstantin19877]

I wonder if embedding needs to be saved in one column and single record. So one huge chunk of data, where i can do vector search? When vector search is done it will return a swntence exactly how it was stored in db or it will compile a new based in similarities?

[@orbitinggeek4000]

Thanks for the amazing video - so am I correct in understanding that even after that embeddings are generated, we still need the OpenAI API for inferences ?

[@PLAYWW]

thanks man

[@lucawurker4911]

How did you prepare the csv file? (How did you code the "rules" for new rows, and how did you get rid of the names of the speakers?

[@forbiddenera]

This is pretty powerful maybe, I did the automated support case this weekend from knowing nothing of embeddings to full responses with gpt in just a few hours (and chatgpt didn't even help on this one)

[@nattyzaddy6555]

Does the cosine_similarity api use openai credits? If it does, is there a similar library that doesn't use credits?

[@krissn8111]

Man this damn interesting

[@SumitKumar-yq1og]

Sup man, I tried to generate the fox example you showed. But, its throwing RateLimitError, even though I created a new API token and the acount is new. Please Let me know.

[@nathanburley]

Question: do you know of a service which would allow me to upload pdfs to a directory, automatically get embeddings for the contents of each file as they're added, then allow semantic search for the embeddings? Preferably via API? Also, I love this video. It was super helpful in helping me get my head around embeddings as a concept.

[@strkn25]

What would be your use case? I'm developing a product that's very similar to what you described.

[@nathanburley]

@@strkn25 technical service AI partner agent. Complex scientific instrumentation. I don't see anything around which can do this right now. I have ideas on how to trim the search based on the specific machines in question.

[@mudassiraqeel-x3g]

would you prefer this method, over fine-tuning gpt models ?

[@arpankumarlahiri4180]

Any way to sort this type of data, If my understanding is correct for large corpus , calculating distance for every sentence with respect to search keyword will grow linearly as the corpus scale?

[@talal-az]

Great content and video. Extremely valuable. One question though as I’m in the area of research. Is there a way to get the total similarity index score for a certain word? For example, let’s say I’m interested in doing an analysis on all the fed speeches and its relationship with bond markets by searching the term “high inflation” in all speeches and getting an overall similarity index total score for every speech that I can later use for regression purposes

[@azuredominus5739]

You would just calculate the embeddeding for the each document then calculate the embedding for your word. Then rank each document based on the cosine similarities

[@fxsignal1830]

can I ask what are exactly your regression purposes? thank you very much

[@wangjueliang]

Thank you for another amazing video. Is this the answer to improving the intranet search results (built on SharePoint Online)?

[@yamani3882]

Is there a similar library that we can use to locally vectorize the strings without having to constantly make openai calls?

[@parttimelarry]

For sure. Check out HuggingFace, will get to local and open source libraries in more detail after the OpenAI series.

[@yamani3882]

@@parttimelarry Awesome, thank you!

[@Jaybearno]

How do you incorporate document structure into embeddings? I.e when I query relative documents for “chapter 3”, it returns all chunks contained in that section, regardless of text content

[@myessmith9577]

Could you do a video showing how you could build an API to pull out all the numbers in the financial statements (balance sheet, income statement and statement of cash flows) into excel or google sheets? Also would it be possible for have a program do this for every 10k published on the SEC website when they are published ?

[@akshadawalunj7018]

I have a use case: If graph/csv file is used as input. Using embedding we should get their relevancy as output. Can we implement this in nodejs?