Glad you enjoyed it @tanguero2k7 … we did too! We’ll keep going to wherever exploration of the LLM Edge takes us!

The most classic pattern we've seen from our community!

Thanks!

I would build some metadata, in this case, like a summary/outline and use that to generate contextual augments.

It can be online, yes.

We're planning to cover computer use in an upcoming event soon - probably nov 13. Stay tuned!

Caching is offered by Anthropic's endpoint by default - and is being taken advantage of under the hood here.

Can you be more specific about what the flaw is? Also, why do you choose the word "exist" rather than "inferred from?" --Here's what appears to be true from the documentation: -- "To calculate this a set of claims from the generated answer is first identified. Then each one of these claims are cross checked with given context to determine if it can be inferred from given context or not." Three steps to the calculation: 1. Break generated answer into statements 2. For each statement, verify if it can be inferred 3. Calculate Faithfulness! It seems that the condition "if (and only if) it can be inferred from the context" will keep the faithfulness calculation from going higher than 1.0

@AI-Makerspace you might be right, but at the point referenced in the video, it talks about the context not the generated answer. So a context like "Paris is a bustling French capital and center of culture and art" could contain 2-3 claims , but the answer to "what is the capital of France" may contain one claim "Paris is the Capital of France". The faithfulness would be 3/1 in that case if they were not related to the golden truth answer. I may be missing something! Great video though, thanks!

Ah I get it now, duh - the element of the formula "number of claims that can be inferred from the given context" I was reading as the number of claims that can be inferred from the context alone. It's really the number of claims in the generated answer which can be inferred from the given context.

"It's really the number of claims in the generated answer which can be inferred from the given context." Nice follow-up @Mark! Let's gooo! We find it helpful to look directly at the prompted examples in the src code here: github.com/explodinggradients/ragas/blob/7d051437a1a5d8e9ad5c42252bf1debf51679140/src/ragas/metrics/_faithfulness.py#L52 You can see how FaithfulnessStatements turn into SentencesSimplified with an example and in general via the instruction given in NLIStatementPrompt as "Your task is to judge the faithfulness of a series of statements based on a given context. For each statement you must return verdict as 1 if the statement can be directly inferred based on the context or 0 if the statement can not be directly inferred based on the context."

Can you expand on your request?

@@AI-Makerspace thank you for answering! I'm curious about the best practices for building a chat engine or chatbot that can interact in a continuous loop with a workflow. Currently, we are receiving one response at a time from the workflow, but I was wondering if we could enhance this by buffering the "chatmemory" and keep on with the conversation. Should this be achieved with a loop? I feel like I remember a llamaIndex or Langchain tool that kept the chat engine running, but I might be mistaken, maybe I was just re-querying. Also, how can I ensure other workflows share the same context? Additionally, is it possible to store interactions as vectorized semantic and episodic memories in a vector database, allowing the system to recall past conversations and in the future query from those memories and the RAG? and maybe do some type of reranking.

from llama_index import SimpleDirectoryReader, VectorStoreIndex from colorama import Fore, Style, init init(autoreset=True) def chat(): print(f"{Fore.CYAN}Loading documents...") index = VectorStoreIndex.from_documents( SimpleDirectoryReader("./data").load_data() ) chat_engine = index.as_chat_engine() print(f"{Fore.GREEN}Ready! Type 'quit' to exit ") while True: query = input(f"{Fore.GREEN}You: {Style.RESET_ALL}").strip() if query.lower() == 'quit': break if query: print(f"{Fore.BLUE}Assistant: {Style.RESET_ALL}{chat_engine.chat(query)} ") if __name__ == "__main__": try: chat() except Exception as e: print(f"{Fore.RED}Error: {e}")

Agreed! We typically stick with easy to consume toy-examples, however!

Definitely will check it out!

Sure thing, just pinned the slides and notebook in a comment!

Elasticsearch is a great tool for this!

Great insights and instincts @Sean! We'll keep the content recommendation in mind for sure! This is farthest we've gotten on Long-Context and Evaluation for the big window LLMs: kzbin.infoBrwhbjh3boU?si=V24z6pagQ0EQ8Ms1

The Ragas part of code in the notebook is not working. Could you fix it?

Hey Givanildo! The full event is here: kzbin.infoxmaG4al2A6E?si=bdHM0wzlll5XkXWJ To learn more about loss functions, check out this one! kzbin.infoiB8FWR9aD5Q?si=4oABKIf-DDNQQv1R

Yes, that is a decent way to approach that problem.

We don't currently have scholarships available @nazmuss! We are working to get our business model right and to grow our partnerships in the US so we can best serve our community members around the world in the long-term moving forward! In short, stay tuned!

"Ground Truth" can come from any of these sources! Of course, getting it straight from the people who perform whatever tasks you're automating is the right idea, but this can be very expensive. In the case of RAGAS the "Ground Truth" is represented by the output you get when you provide [question, retrieved context] pairs as input to a generator. That is, we are not actually using a RAG system, but passing "correct" [question, context] pairs as input. These are "correct" because they were synthetically generated and are known to be correct; see Synthetic Test Data Generation: docs.ragas.io/en/stable/concepts/testset_generation.html Note that Ground Truth is different than "Answer" because "Answer" actually uses the RAG application that you're building, while "Ground Truth" passes [question, context] pairs in direclty.

Thanks Andres!