Indeed, I showcased how to use strict_json (LLM) to build a Knowledge Graph. We also can use KGs to provide context for LLMs, so it works both ways.

@@johntanchongmin 'We also can use KGs to provide context for LLMs' Can you please send me any video/blog link where I can read more about this

@@rajathslr You can refer to the paper I covered in this video: arxiv.org/html/2306.08302v3 Can look at an example KG-RAG workflow here: www.nebula-graph.io/posts/KG_RAG_Bridging_Gap_Knowledge_and_Creativity_of_LLM_based_Agents

@@johntanchongmin Thanks for your time, you really rock!!

I have considered yaml before. I don't think the llms are ready for it yet. I may change my mind in the future and do a yaml version of strictjson haha.

I have yet to use TaskWeaver. For CrewAI, I think it is one of the better agentic frameworks out that, but it can also be too verbose since it is conversational-based. LangGraph tries to do what TensorFlow did instead of native Python. It feels unnatural to use, I recommend not using it.

Hope you enjoy the exploration process, come join the discord group for more intriguing conversations!

Hey there, you can try it out. Website and webapp, if they are part of Python programming, could be done too. I think the baseline is whether the LLM used is robust enough to generate code for your particular application. For cases that are too complex involving multiple files, it is best to design each file separately as the context length of ChatDev is the total sum of all the content of each file you need it to generate and more.

I think most mature is LangChain, simply because it is around longer. AutoGen is better than LangChain in getting Agents to talk to one another in a group setting. Most promising one so far I've seen is Crew.ai. TaskGen is relatively new, but it will eventually be the better option as I plan to incorporate a lot more things about memory and continual learning.

It uses StrictJSON, which parses the LLM output into a specified json format. Check it out here: github.com/tanchongmin/strictjson

Do refer to the most updated Tutorial for more information before you use it for your use case: kzbin.info/www/bejne/Z3-Qq6F6idGbhMU

Tomorrow is TaskGen Ask Me Anything session, join my discord group for link!

@@johntanchongmin I joined the discord and saw that! Really happy for that, I spent my entire Sunday catching up on your projects through your great videos. Unfortunately I'm in CET so it will be 4am for me, but I'll catch up on it after! Thanks for sharing your work so enthusiastically 😎

Hope you like it :)

Hi @@johntanchongmin Having this error ... An exception occurred: "Broad Trip Plan" not in json string output. You must use "###{key}###" to enclose the {key}. Current invalid json format: {... When using local llm. What is the likely cause. Look fwd to your kind guidance. Cheers

@@ginisksam Local LLMs are likely not as good at instruction following. I'll be releasing a patch in a few days. Meanwhile you can add this line to your system prompt: "Begin your response with {{ and end with }}"

@@ginisksam Could you also show me the invalid json format?

@@johntanchongmin Here is snippet running cell below from strictjson_AMA...ipynb res2 = strict_json(system_prompt = '''Given the triplet list, write some code to display this as a Knowledge Graph''', user_prompt = triplet_list, output_format = {'Code': 'Code in Python to display knowledge graph from triplet list'}, llm=llm) Error: An exception occurred: list index out of range Current invalid json format: { "'###Code###': " "import networkx as nx " "from networkx.drawing.nx_agraph import write_dot " "from itertools import permutations " " " "triplets = [{ " "'obj1': 'LLM', 'relationship': 'provides', 'obj2': 'Base Functionalities'}, " "'obj1': 'Tutorial.ipynb', 'relationship': 'refers to', 'obj2': 'Base Functionalities'}, " "'obj1': 'int, float, str, dict, list', 'relationship': 'supports', 'obj2': 'Base Functionalities'}, " "'obj1': 'Dict[], List[], Enum[]', 'relationship': 'supported by', 'obj2': 'Base Functionalities'}, " "'obj1': 'bool type forcing', 'relationship': 'handled by', 'obj2': 'Base Functionalities'}, " "'obj1': 'LLM-based error correction', 'relationship': 'used for', 'obj2': 'Base Functionalities'}, " "'obj1': 'Function (renamed from strict_function)', 'relationship': 'available in', 'obj2': 'Base Functionalities'}, " "'obj1': 'OpenAI JSON Mode', 'relationship': 'enabled by', 'obj2': 'Base Functionalities'}, " "'obj1': 'llm variable', 'relationship': 'exposed for', 'obj2': 'strict_json, Function'} " "] " " " "# Create a graph " "G = nx.DiGraph() " " " "# Add nodes and edges from triplets " "for obj1, rel, obj2 in triplets: " " G.add_edge(obj1, obj2, relationship=rel) " ... Thanks for your continous guidance.

Glad it helps. I am actively pursuing my idea of multiple abstraction spaces, and KG can be one of them. The rest of how we store memory will depend on what kind of memory - semantic facts, episodic memory and so on. These can be stored in various ways like traditional databases, or even in video/image format.

@@johntanchongmin Thanks for sharing your research. I will particularly follow closely your work on context-dependent embeddings. That's an exciting angle to explore in depth.

Haha what footage?

That said, the spacetime patches is an interesting idea, and if it works out, could form the basis of a lot of innovation in the video prediction domain.

Brian has open sourced most of them on the TaskGen repo!

CLIP's loss function has also been described as InfoNCE loss, a common loss term for contrastive learning. See builtin.com/machine-learning/contrastive-learning for details. It is essentially Cross Entropy over cosine similarity terms, which is what is done in CLIP.