Graph ML: Build Knowledge Graphs using Generative AI and LLMs

Knowledge Graphs
Generative AI,
LLMs
Graph DBs
Neo4j
Cypher
Fine Tune LLMs
Graph ML
Node Embeddings
Graph Features
Langchain
Chatbots
Gradio
Code: github.com/neo...
#datascience #machinelearning #deeplearning #datanalytics #predictiveanalytics
#artificialintelligence #generativeai #largelanguagemodels #naturallanguageprocessing
#computervision #transformers #embedding #graphml #graphdatascience
#datavisualization #businessintelligence #montecarlosimulation #simulation #optimization
#python #aws #azure #gcp

[@SridharKumarKannam]

If you found this content useful, pleases consider sharing it with others who might benefit. Your support is greatly appreciated :)

[@ScottzPlaylists]

Will KG's be the basis for the semantic web they say is coming❓ = An embeding representation would be a very large download with so many vectors per token❗ = A token representation where each token is a number, would be more compressed than plain text, but not very useful since the calculation is so easy❗ = But a Knowledge Graph (KG) would be the best❗ A Tokenized KG would be more compressed. (If we can standardize on the best Tokens to numbers standard) 🔥

[@SridharKumarKannam]

These things are still evolving and lets see how useful they can become in real-world use cases. Thanks for your support.

[@Shivam-bi5uo]

how can we make use of any other model, how about openai?

[@SridharKumarKannam]

you can use any LLMs, openAI, Ollma, HuggingFace, etc, all works fine, the quality of results varies of-course...

[@mindgraphai]

thankyou for the help sir...i had a wanted to ask you that i'm working on a similar project where my input is unstructured contract files like Non disclosure agreements ,loan agreements ,etc ... how am i suppose to create a input file for the same ...

[@SridharKumarKannam]

The challenge is not with input, you can just process all those docs and create text docs. You have to experiment with LLM prompts to be able to extract all the entities and relations you are interested in.

[@andydataguy]

This series is exactly what I was looking for! Thank you 🙌🏾

[@SridharKumarKannam]

thank you very much.

[@qingdong801]

Thank you for your impressive video!

[@imanechatoui4308]

Thank you for your efforts. I have a question: How can we extract entities from PDF files?

[@SridharKumarKannam]

you can read pdf files using langchain or llama-index, use any other tool to convert to text first..

[@mulderbm]

Thanks for sharing this use case is something i have been looking at for a while now.

[@SridharKumarKannam]

Thank you very much for your support :)

[@vishaldesai28]

Please add reference for notebook code

[@SridharKumarKannam]

done, the repo link added :)

[@romakhajiev9535]

THank you so much. Do I have to pay for the google's model? Is there any way that I can use for free?

[@SridharKumarKannam]

You can use any LLM model. When you create a new GCP account, you will get 300$ free credit for first 3 months. You can use free LLMs from HugggingFace and OpenAI (10$ free credit) also.

[@rayhon1014]

Have u tried your prompt over openai and see if vertexai is better? What makes you to go after few prompt approach vs simply specify your entity relations as context and have the LLM create the full graph for you in one shot instead?

[@SridharKumarKannam]

I've not compared for this task but OpenAI should give similar results. As we wanted the output to be in a very specific format, we need to provide the schema and an example, as this is not a generic Q&A. LLM results are better with one task at a time. The results generally poor if we ask it to do multiple tasks in a single prompt.

[@kevinkate4500]

Sir, Can't we use any other open source llm model instead of text-bison?

[@SridharKumarKannam]

you can use any models, try the free ones from Hugging Face. As you know the larger models perform better on these tasks as we are using generic models.

[@pavellegkodymov4295]

Great, very helpful and detailed step by step instructions, thanks a lot, really appreciate!

[@SridharKumarKannam]

thank you :)

[@alimahmoudmansour9681]

Thank you it is very helpful

[@SridharKumarKannam]

Thank you very much :)

[@AshSeddeek]

Sent you LinkedIn

[@SridharKumarKannam]

Thanks.

[@anusornchaikaew2567]

Thank you very much

[@SridharKumarKannam]

Thanks :)

[@leobeeson1]

Very good hands on! Thank you. Three questions: 1. Why run `clean_text()` inside the for loop for every prompt? Why not run it once only before the for loop? 2. Why have the LLM generate the unique ID? Wouldn't it be better to have a proper UUID generator for every entity instance (person, skill, company, etc.) and inject into the hydrated template or replace it ex post from the LLM's response? 3. If the LLM is creating the "unique" IDs (e.g. `skill1`, `skill2`, etc.), how do you normalize ID's across multiple resume's? Wouldn't you risk having some id collisions across resumes? e.g. resume_1 -> `{"skill_label": "SQL", "skill_id": "skill1"}` vs resume_2 -> `{"skill_label": "Mongo", "skill_id": "skill1"}`? Thanks again!

[@SridharKumarKannam]

Thanks for your very valid comments. You are right on all 3 points. Agree its not a production ready pipeline, I was just experimenting and thought useful to share the idea . Its far from perfect :)