5. OpenAI Embeddings API - Searching Financial Documents

[@parttimelarry]

Like the video? Support my content by checking out Interactive Brokers using the link below: www.interactivebrokers.com/mkt/?src=ptlPY1&url=%2Fen%2Findex.php%3Ff%3D1338 I will be starting a spinoff channel on AI in music, art, and gaming in 2023. Subscribe at: youtube.com/@parttimeai Buy Me a Coffee: buymeacoffee.com/parttimelarry Notebook: colab.research.google.com/drive/1tttDqgnWL9yJtmlOFXJqA-BjQ1Pyfpax?usp=sharing This notebook is based on the embeddings examples in the OpenAI Cookbook: github.com/openai/openai-cookbook

[@jonconnor6697]

say vector one more time :)

[@SwingingInTheHood]

Not being an AI guy, I've been trying to understand embeddings and search vectors for the past two weeks. This is the first video I have found that explains it so that even I now get it. Whew! Thank you Brother!

[@hishamafzal1999]

Can you explain me bro?

[@wmchoi1474]

Thanks for this amazing tutorial. Great video dude

[@erikstillman7336]

Thanks!

[@alanfeng99]

Extremely well presented of how embeddings work!! I "kind of" knowing how it works, but after watching this video, I can say I "know" how it works! Thanks.

[@robertcormia7970]

This is outstanding, his clarity and enthusiasm is amazing, real passion for this topic!

[@soesevenonesix]

Another great Video Larry! Thanks for sharing!

[@Raja-jo5dm]

Nice one larry easy to understand : )

[@Guy-Scott]

I appreciate the work you have put into this series. I like how you start small, with a few examples and build from there. Great job.

[@nedyalkokarabadzhakov5405]

quality content deserve subscription. Thank you.

[@mariusirgens5555]

The espresso-milk -> latte example blew my mind 😄

[@sivalokesh3997]

I searched for subscribe button then

[@persas1683]

Your tutorial help me alot. Thank you 🍾

[@jay-d-x]

Excellent and simple. Thanks Larry.

[@PatrickAlphaC]

Dank. Love your content dude.

[@parttimelarry]

Likewise man, been awesome to see you kill it over the years!

[@karansajnani1135]

This is by far, the best video explaining embeddings! Thank you

[@seandiamond7983]

Bro you're the fkn man. I would have never known about this. I just created a search field in my Django website and it will be getting a massive upgrade now lol thanks again bro.

[@deep2mixer]

Awesome. Disrupter R' Us. This is some high quality material.

[@rengavasan6245]

This is the best explanation Ive come across about embeddings anywhere on the internet. Great work Larry. Subscribed :)

[@iamaguest2]

very much appreciated for the simple introduction. Very good at explaination of something that is not easy to understand, great job

[@lqvision]

very patient explaination!

[@aecs71]

awesome video, was struggling to understand embeddings for a long time,, this video is superb

[@PhilMcCarty]

This is a really great tutorial, thank you.

[@parttimelarry]

Thanks so much! Really appreciate it.

[@deepster5150]

How do you buy a drink and get that logo you have?

[@PhilMcCarty]

@@deepster5150 On my screen to the right of "Share" there's a "Thanks" button. It's probably a KZbin red or whatever the paid thing is feature.

[@kenchang3456]

Thanks, Larry for a great video and explanation. Being an old former developer I am so fascinated with these LLMs, and your explanation has helped me organize how I will do a POC. I really appreciate you sharing your knowledge and enthusiasm.

[@Jberv]

Man! I just watched your video like its 5 minute length, very detailed. It took me so long to find this kind of sample/explaination. Thanks

[@parttimelarry]

Thanks, this video wasn't very popular when I first put it out, but I think people appreciate it now :)

[@SumedhKadoo]

Thanks for sharing, you are a good teacher. Can you please share how much did this cost in terms of API usage?

[@private_0]

Amazing video! Loving the series so far. Thank you for doing it!

[@munoncode]

thank you so much for this tutorial.

[@jessexing3456]

amazing video! how smooth it is!

[@TomanswerAi]

Great video nicely explained

[@blender_wiki]

Your contents are surpassing the simple high economic value and make the world a better place. 🙏🏾🙏🏽🙏🏼🙏🏻🙏🙏🏿

[@MikeBtraveling]

"The inflation is too damn hight", was great.

[@rhemagenesis]

You came with the HEAT 🔥 🔥 🔥

[@ThomasODuffy]

Great video and explanation - thank you!

[@thomastudu2977]

This is exactly what I needed man, Thanks !!!

[@ratral]

Thanks for the video; it is pretty straightforward and well-developed. 👍

[@davidw8668]

Your recent video quality again has come in higher than expected - despite high priors, sir😁 damn good teaching and funny. Thanks man!

[@rishabhpanwar8971]

Superb content. Thanks!

[@HazemAzim]

Very nice and soft explanation 👌 .

[@carlhlazo5907]

Awesome tutorial. Using familiar concepts made it easy to follow. I was finally able to understand much better what these concepts are and how they all relate.

[@shanthu7803]

Happy New Year! Great content learning a lot from you. Thank you so much for detailed explanations step by step guidance. Wish you all the best!

[@testadrome]

Awesome video! Keep them coming

[@laura43400]

One of the best explanations out there!

[@hb_dan2928]

thank you, really good and concise explanation

[@manuthvann7560]

thanks for sharing such informative content , I've learned a lot from you, and Kepep doing a great job

[@breakoutgaffe4027]

When it comes to Python tutorials Larry you are the 🐐

[@frutenfratenfriends]

This is impressive brother thanks

[@businessdissection130]

After weeks of lurking around, god showed me this video! Never going to regret this. This is exactly what I wanted. And I would say this was even better than the documentation on openAI, since it always gave me an error!

[@parttimelarry]

Thanks for the comments, glad this was helpful!

[@processusAI]

very well done video. Fit exactly what I wanted to find out

[@Naijarianman]

These videos are awesome, thanks a lot

[@bakari007]

Great Video ! Super interesting how this transformations works words to vectors ! I'm wondering how the architecture of the classifications works ! Thank you for the great quality of the video and the super interesting topic !! Don't stop !! 🥰

[@serhatkoroglu6708]

you telling very clear thank you!

[@marshall3900]

Greate video!

[@patriciodiaz2377]

Thanks so much! 🎉

[@GonzaloRMDT]

Great job!

[@charyog7]

thank you very much, it was very clear !

[@iamnoone7415]

I find your tutorials on openai to be very informative and easy to follow. The instruction explained the concepts well and used examples to help illustrate the topics. Even for someone who does not have a coding background, I find the tutorials very engaging. However, I have a question about how to overcome the rate limit error.

[@michaelwang6831]

hi, have you ever been able to fix the rate limit error? I am facing the same issue..

[@jskweres2]

@@michaelwang6831 I have the same issue. Advice on fixing it?

[@undrash]

Amazing content, thanks

[@omkarajagunde4175]

Wow awesome man

[@psychurch]

Thanks man, super clear explanations! Subscribed and excited to see what you’ve got coming up for music and ai ❤

[@avbendre]

you are amazing!

[@Nashadelicable]

8:36 "this in the Matrix is how a hot dog looks like" XD

[@FRANKWHITE1996]

thanks for sharing

[@sachin-b8c4m]

very helpful thank you

[@lastemperor1347]

Amazing!

[@utuberay007]

Awesome..

[@user-wr4yl7tx3w]

Would be interesting to see how such word embedding is even computed in the first place. possibly with a toy example.

[@camuelcoleridge479]

@8:36 Part time Oracle is exposing the matrix 😂

[@guillaumeberrido]

Excellent! Next step, would be to store our data in a Vector DB like pinecone instead of a csv...

[@amanmeghrajani1]

great vid

[@chenpaul99]

Excellent demo! When using embeddings for training, how do I feedback the graded results to improve the model?

[@BenjaminRay]

Very cool!

[@neelcansharma]

Incredible video, you're a natural teacher. Do you know how embeddings is calculating these numerical vectors? Is it comparing to other items in their dataset?

[@Malte.Research]

Awesome, thanks !

[@VeeraForBharat]

For the last part of video, you just sounded like Jerome Powell. Great video to understand Embeddings API and how its useful for Cognitive Search. It looks like the Search term and every other sentence in the knowledge base has to go thru the Cosine Similarity check. Doing this on a small document is computationally ok but how does ChatGPT and others do this on Petabytes of data?

[@xiangyu326]

This is super.

[@maverick456-33]

You are Genius

[@charlieevert7666]

Bad ass, thanks!!

[@MrYomantanepali]

earned my subscription

[@forbiddenera]

This is pretty powerful maybe, I did the automated support case this weekend from knowing nothing of embeddings to full responses with gpt in just a few hours (and chatgpt didn't even help on this one)

[@michaelwang6831]

Question - for the Microsoft earning calls example, how do you segment the whole speech into different paragraphs as a unit to run through the AI algorithm? What is the logic? or you simply segment them based on comma and period punctuation?

[@lucas.souza2022]

I would like to know that too, did you come to any conclusions?

[@HazemAzim]

One question is the effectiveness of embeddings varies with the size of the text? Embedding a word or phrase or sentence versus embedding a sizeable paragraph? Would this affect the accuracy ?

[@TommyInGame]

This is so huge. Currently creating a “digital employee” product to limit human capital required for SMEs and startups, and this is a game changer. Would you be open to be a potential beta tester ? I’d love your feedback👍 great vid as always

[@cricinfo261]

Hi can someone explain how to make this search fast on 1000s of documents when we need to calculate cosine similarity on every line of those documents for each query?

[@yamani3882]

Is there a similar library that we can use to locally vectorize the strings without having to constantly make openai calls?

[@parttimelarry]

For sure. Check out HuggingFace, will get to local and open source libraries in more detail after the OpenAI series.

[@yamani3882]

@@parttimelarry Awesome, thank you!

[@talal-az]

Great content and video. Extremely valuable. One question though as I’m in the area of research. Is there a way to get the total similarity index score for a certain word? For example, let’s say I’m interested in doing an analysis on all the fed speeches and its relationship with bond markets by searching the term “high inflation” in all speeches and getting an overall similarity index total score for every speech that I can later use for regression purposes

[@azuredominus5739]

You would just calculate the embeddeding for the each document then calculate the embedding for your word. Then rank each document based on the cosine similarities

[@fxsignal1830]

can I ask what are exactly your regression purposes? thank you very much

[@PLAYWW]

thanks man

[@tatendaabeta8121]

Okay, so lets say i have a pdf file which contains some text, do i have to convert every sentence there to csv file. Anyone with an Idea ??

[@michaeldebellis4202]

When you do the dot product, doesn’t one vector need to be a column vector rather than two row vectors as in your example? My linear algebra is rusty so perhaps I’m missing something but as I recall you can only multiply (dot product) an M x N matrix with an N x M matrix(?)

[@kovidkasi6117]

Where you are creating embeddedings for text data, is it just embeddedings of each word in a text combined together or are you taking sentence embeddings?

[@erik1639]

How would you do this kind of search on an excel or csv document?

[@Konstantin19877]

I wonder if embedding needs to be saved in one column and single record. So one huge chunk of data, where i can do vector search? When vector search is done it will return a swntence exactly how it was stored in db or it will compile a new based in similarities?

[@nathanburley]

Question: do you know of a service which would allow me to upload pdfs to a directory, automatically get embeddings for the contents of each file as they're added, then allow semantic search for the embeddings? Preferably via API? Also, I love this video. It was super helpful in helping me get my head around embeddings as a concept.

[@strkn25]

What would be your use case? I'm developing a product that's very similar to what you described.

[@nathanburley]

@@strkn25 technical service AI partner agent. Complex scientific instrumentation. I don't see anything around which can do this right now. I have ideas on how to trim the search based on the specific machines in question.

[@SumitKumar-yq1og]

Sup man, I tried to generate the fox example you showed. But, its throwing RateLimitError, even though I created a new API token and the acount is new. Please Let me know.

[@myessmith9577]

Could you do a video showing how you could build an API to pull out all the numbers in the financial statements (balance sheet, income statement and statement of cash flows) into excel or google sheets? Also would it be possible for have a program do this for every 10k published on the SEC website when they are published ?

[@joy2000cyber]

How to get a certain word’s embedding in the context of a sentence using openai? Say I have a sentence “the thief takes a walk on the river bank after robbing a bank in downtown”, I feed the sentence to openai api, but I want to get the embedding of each word bank in the sentence. From your video, the api just feedback one vector for the entire sentence. Thanks!

[@alexchanchan9359]

Thank you very much. Can I ask why in the notebook, the earnings_df has not used ".apply(eval).apply(np.array)"?

[@mudassiraqeel-x3g]

would you prefer this method, over fine-tuning gpt models ?

[@lucawurker4911]

How did you prepare the csv file? (How did you code the "rules" for new rows, and how did you get rid of the names of the speakers?

[@krissn8111]

Man this damn interesting

[@arpankumarlahiri4180]

Any way to sort this type of data, If my understanding is correct for large corpus , calculating distance for every sentence with respect to search keyword will grow linearly as the corpus scale?