Local GraphRAG with LLaMa 3.1 - LangChain, Ollama & Neo4j

[@garic4]

This was EXACTLY what I've been looking for. Thank you for the great walkthrough, you just got yourself a new subscriber. Cheers

[@eloiseidlitz7062]

Smart video ! It could be a good way to access easily relationships between stakeholders in any kind of business.

[@mgmanzanillo]

Could you also create sample code for instances where multiple documents are stored in a directory in either .txt or a compressed format like Parquet? How would it work then? Would you transform each document separately and combine them as a graph? Would like to get your insight. Thank you for the video!

[@miguelprctba]

Congratulations, great approach! I was wondering if you could consider using a different embedding method instead of OpenAI's, as it requires payment. What do you think?

[@codingcrashcourses8533]

Ollama also offers an embedding model and to be honest, I think you should probably use it. You can use the same approach to use an env variable to use the ollama embeddings class like I did for the model

[@CptSnunu-ii4um]

How much VRAM and RAM do you think it takes to run the 70b model at minimum?

[@mgmanzanillo]

Has any tried this code for a large amounts of documents? How does it scale?

[@SilviaRicci13]

I try this approach with Ollama but It seesm when execute the row "llm_transformer.convert_to_graph_documents" it run for more time and I need to stop the run..can help me? What is it the problem? The graph DB neo4j connection?

[@codingcrashcourses8533]

@@SilviaRicci13 any errors?

[@SilviaRicci13]

@@codingcrashcourses8533 No errors!

[@SilviaRicci13]

@@codingcrashcourses8533 After 118 minutes with ollama end with this error: ValidationError: 2 validation errors for Node id none is not an allowed value (type=type_error.none.not_allowed) type none is not an allowed value (type=type_error.none.not_allowed). Error is generated from the line: graph_documents = llm_transformer.convert_to_graph_documents(documents)

[@moellerseo]

Nice video. 🎉 Thanks

[@PassingTheDog]

Running this error over and over in the "graph_documents = llm_transformer.convert_to_graph_documents(documents)" cell: 731 parsed_json = self.json_repair.loads(raw_schema) 732 for rel in parsed_json: 733 # Nodes need to be deduplicated using a set --> 734 nodes_set.add((rel["head"], rel["head_type"])) 735 nodes_set.add((rel["tail"], rel["tail_type"])) 737 source_node = Node(id=rel["head"], type=rel["head_type"]) TypeError: string indices must be integers

[@codingcrashcourses8533]

Hmhmh, some people seem to get this unfortunately. Did you use the 8B model? That could explain it, since it might create empty documents or so.

[@Wingly113]

@@codingcrashcourses8533 getting the same error as well

[@codingcrashcourses8533]

@@Wingly113 also with 8b? I did not run in any errors with 70b of gpt4-small yet

[@PassingTheDog]

@@codingcrashcourses8533 Yup, 8b. I can't handle 70b tho...

[@sidharthgarg2348]

Llama 3.1 8B can also handle it, instead of ollama use llama.cpp and download llama 3.1 8B.gguf model and customise prompt template and you are good to go.i For reference you can use this repo github.com/s3dhanth/GraphRAG-with-Hermes-2.5-Pro-LLM-using-neo4j-database

[@i2c_jason]

Are the graph visualizations mostly used for development, to double check what the system is 'thinking' and look for valuable clustering?

[@codingcrashcourses8533]

The visualization will probably used for development, the LLM won´t use it (a vision model won´t be able to deal with it, if you think of that)

[@i2c_jason]

@@codingcrashcourses8533 I may have missed it from the video, but is Neo4j open source / free to use always? I looked at their site and was confused. Or are you using free-tier for the demo? Would love to use this for development. Thanks!

[@codingcrashcourses8533]

@@i2c_jason It´s free to use with Docker, you can also download a Desktop version. If you want to use it for your enterprise app, it might be different

[@alishaw9999]

@@codingcrashcourses8533 can you demonstrate it with desktop version of neo4j

[@Transforming-AI]

You mentioned you received a key error at one point. I am getting that key error as well. Please share how you worked around the key error. Following your notebook, my key error is, KeyError: 'head'.

[@codingcrashcourses8533]

@@Transforming-AI had the same issue a few times. I think the 8k Model produced this, bit not always. Or did you try the 70b or 405b Model?

[@Transforming-AI]

@@codingcrashcourses8533 I have the 8b installed, will try the 70b. Thanks.

[@saadsafi6073]

@@codingcrashcourses8533 had the same, seems to be caused by the double quotes in dummytext.txt, if you replace them (say, with single quotes) then the json will not be broken. key error was to do with missing 'head' which is one of the node keys.

[@mingjenchen2975]

@@codingcrashcourses8533 I tried it with llama3.1:70b and it took 13 minutes on my Mac Studio.

[@carlospareja7951]

@@codingcrashcourses8533 I'm also running into this issue when running with the 8B model.

[@kevli6373]

Thank u for great video. Is it possible to still use ollama via langchain? I got a error when i use "if llmtype=="olllama" -- error message : Received unsupported message type for Ollama. (my_path) : LangChainDeprecationWarning: The method `BaseChatModel.__call__` was deprecated in langchain-core 0.1.7 and will be removed in 0.3.0. Use invoke instead. warn_deprecated(

[@codingcrashcourses8533]

Did you change something in the codebase? Which line exactly produces this error? I have used the invoke method, to perform calls to the LLMs, so this kind of warning should not appear

[@Mohidthevloger]

Great video! One thing I would be interested to know that were you be able to compare the results between the two approaches Graph or Vector search? Which one is retrieving the most relevant results?

[@codingcrashcourses8533]

It depends on the question :). You probably want to use a Hybrid approach

[@maxlgemeinderat9202]

Interesting! Is Neo4j language dependent? So is it capable of finding relations of multilingual data or does Neo4j not care about the language and only the LLM to create the knowledge graph should be multilingual?

[@codingcrashcourses8533]

the LLM is responsible for creating the entites and documents, you will just save them in the required format in the database. Neo4J has got many libraries with different languages. The format of the documents matter.

[@oluwaseunakinropo6318]

Hii, amazing content as usual man, i am trying to implement this noe4j graph on my pc after downloading your code, i went to neo4jaura and downloaded a .txt file containing the database credentials but whenever i try it from your code i get this error message below, please help " Could not connect to Neo4j database. Please ensure that the url is correct" it is exactly the same thing i downloaded from their website

[@truliapro7112]

How can I use this for my code project files. I need to convert whole php project with python.

[@codingcrashcourses8533]

The code is in the description, it´s free, do whatever you want with it :).

[@sylap]

Thank you for this video! 👏 I've been eager to see someone cover this topic. I have a couple of questions I hope you can help me with. In this approach, is there a way to rerank the retrieved documents? Did I understand correctly that you first query the graph database, and based on the retrieved entities and relationships, it then retrieves pieces of documents from the vector store?

[@codingcrashcourses8533]

Regarding reranking: I don´t know to be honest. I could not think of any metric to do that. Regarding retrieval: No! I used a hybrid approach there, but they are independent approaches. The LLM will use docs from both approaches.

[@sylap]

@@codingcrashcourses8533 Honestly, not sure if reranking is even needed for Graph based RAGs, as they are very accurate. 😄 Thanks! 👊

[@mahoanghai3364]

Cool. Thank a lot

[@Sundaysun9633]

Cool 🤙

[@tuba-inxs]

Is this really an RDF Knowledge Grap, or just a Neo4J property graph. That doesn't look like a SPARQL query in your code.

[@codingcrashcourses8533]

Hm I am not a Graph Expert to be honest. I had go through some tutorials myself to learn how to construct a query first, since I mainly worked with the RAG part so far. How would it look like in your opinion?

[@DePhpBug]

I just finished learning the ChromaDB , can the GraphRAG works with ChromaDB instead of Neo4J?

[@codingcrashcourses8533]

No, chroma is not able to store graphs

[@Gabriel-ch7sx]

What about Surrealdb?

[@giantworks1366]

Nice. Can I only use llm ollama3.1 without openAi? Thanks

[@codingcrashcourses8533]

@@giantworks1366 yes, i wrote the Code in a way that you can switch from one to another via the llm_type variable. But the 8b param models seems sometimes to have issues to create the docs in the required way

[@federicosalvati2454]

where is the .env file in the repository?

[@codingcrashcourses8533]

@@federicosalvati2454 its not there. You need to use your own api Keys and Passwords of course

[@federicosalvati2454]

@@codingcrashcourses8533 thank you so much I am approaching coding with python and RAGs for the first time and all of this helps a lot

[@rgolanng]

look like llama cant generate low resource language like Vietnamese

[@codingcrashcourses8533]

no, but to honest most models have issues with that. You probably have to create the english documents first and then try to translate it

[@rgolanng]

@@codingcrashcourses8533 when i first tried it since release, it failed to generate low resource language, but i just tested in Poe, it seems to work well now. Btw thank you, your content really help.

[@rgolanng]

@@codingcrashcourses8533 I really appreciate your content. When i first test with togetherAI, it only responded with English, it seems work well now when i tested again

[@tawakalitolamide578]

Thank you for this video, I have been looking for a way to use local graphrag until I see this. Thank you so much. But I encountered an issue while using the neo4j browser. It keeps saying "Cannot load from URL 'file:///import/test_container.csv': Couldn't load the external resource at: file:///import/test_container.csv ()" I have tried all means to resolve this issue but the error keep persisting. Please I need your help.

[@codingcrashcourses8533]

You got this issue when running my code? I don´t use a test_container.csv file

[@tawakalitolamide578]

@@codingcrashcourses8533 No, I did not get the issue while running your code. I already solved it. Thank you

[@kevli6373]

for entity in entities.names: response = graph.query( """CALL db.index.fulltext.queryNodes('entity', $query, {limit:2}) YIELD node,score CALL { WITH node MATCH (node)-[r:!MENTIONS]->(neighbor) RETURN node.id + ' - ' + type(r) + ' -> ' + neighbor.id AS output UNION ALL WITH node MATCH (node)

[@codingcrashcourses8533]

@@kevli6373 did you try openai models too? Or just the small 8b llama model?

[@kevli6373]

@@codingcrashcourses8533 I used only openai model. i got a message : ClientError: {code: Neo.ClientError.Procedure.ProcedureCallFailed} {message: Failed to invoke procedure db.index.fulltext.queryNodes: Caused by: java.lang.IllegalArgumentException: There is no such fulltext schema index: entity} Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings... 😂

[@kevli6373]

@@codingcrashcourses8533 thank you reply. i used only openai model. i didnt try llama yet.

[@kevli6373]

@@codingcrashcourses8533 I used only OpenAI gpt 4o mini not llama. In your case, Does it work well for both ?

[@codingcrashcourses8533]

@@kevli6373 I had issues with Llama 8B. Everything else worked fine :/. The LLM seems not be able to create/extract entities