NER Powered Semantic Search in Python

[@manuthvann7560]

thanks, sir for sharing, it's such informative content, but there is a question "hmm, instead of using pinecone can we use the elastic search ? and why not apply "removing stopwords step and keyword extraction before working on the NER engine? Looking forward to hearing back from you, thanks "

[@andy111007]

Hi James, With everyone using langchain for NER extraction, are these methods still valid?

[@willsmithorg]

Would be good if you could add a section where you show the query performance WITHOUT the NER filtering so we can understand the benefits that the additional NER complexity brings.

[@MayurLaxmanrao]

@James Briggs First of all I would like thank you very much for your wonderful content. Is it possible to use the Llama-2 model for NER search for legal documents instead of "dslim/bert-base-NER"?

[@DenisBougrimov]

Would this work if you make a typo in the named entity of your query? Or would i have to use fuzzy matching on the query first?

[@ylazerson]

Pure awesomeness!

[@jamesbriggs]

thanks!

[@alivecoding4995]

Does it include fuzzy matching for entity names? It seemed so to me, because we query by embedding similarity.

[@jamesbriggs]

it's similar yes, like fuzzy matching based on semantic similary

[@ariramkilowan8051]

Great video, again. Would have been nice to see what the benefit of NER in the search actually was. i.e. would vanilla semanitc search have yielded similar results anyway. Time for me to run some experiments :)

[@jamesbriggs]

Sometimes yes sometimes no, I think it’s very use-case dependant, in this case we saw some better results when testing it but I didn’t show that in the video oops 😬

[@jimmynguyen3386]

Great video! Do you see a benefit of using OpenAI’s embedding vs. sentence transformer?

[@jamesbriggs]

Amount of text you can embed into a single embedding is far more with OpenAI vs. sentence-transformers. Performance for shorter chunks of text is comparable to a average fine-tuned sentence transformer, and if you fine-tune the sentence transformer well you might even get better performance - but this is just shorted chunks of text (sentence - to - paragraph length)

[@dongli8477]

wonderful video and even better explanation. Even for a non-coder like me, I can understand it. Great mentoring skills!

[@jamesbriggs]

that's awesome to hear, glad it makes sense!

[@Kmysiak1]

Once again great video! I would love to see how we can fine-tune a transformer NER model for our own custom entities. Thanks!

[@jamesbriggs]

yes I'd like to cover this soon

[@athikunte]

@@jamesbriggs did you manage to do this yet?

[@shaheerzaman620]

Can NER based semantic search be done without pinecone?

[@jamesbriggs]

Yeah as long as you can do filtering it’s entirely possible

[@dikshyakasaju7541]

Great video!👍🏻 It's weird however I saw you or someone who looked like you working at Tribal.

[@jamesbriggs]

Haha, yeah that was me, say hi if you see me again 😁

[@dikshyakasaju7541]

@@jamesbriggs I have always used SpaCy for NER related stuffs but I think I should play around with HF ''NER' models as well.👍

[@jamesbriggs]

Definitely, there's a ton of NER models on HF

[@objectobjectobject4707]

amazing content,probably gonna apply for positions at Weaviate :)

[@jamesbriggs]

look here first www.pinecone.io/careers/ ;)

[@objectobjectobject4707]

@@jamesbriggs ah this is great,sorry somehow did think that you work at Weaviate :D. Thanks for the link