Flowise Record Manager: Stop Duplicate Data Forever!

[@leonvanzyl]

Hi guys, one note on the None cleanup method. In the video 13:00 we only see 2 records. This was a bug. The None method will actually add the change to the Vector Store, but will NOT delete the previous record. It will also not remove the deleted document "Cat". This means that you will see 3 records in total (Dog, Dog 2 and Cat). Thanks to the Flowise team for pointing this out.

[@MaliRasko]

Legend!

[@leonvanzyl]

Thank you for the generous support! 💖

[@estevegraells]

¡Gracias!

[@leonvanzyl]

Thank you for the support ❤️

[@HassanAllaham]

Very useful, direct, and clear content... as usual .. Thank you from deep my heart ...With every video related to Flowise, I get more and more sure that Flowise is the right way to get the best results of LLMs .. KZbin asked me about this channel and my answer was: It is a life changer (+5-stars) ...I watched and keep watching many any many of channels ... this channel is one of the best I ever watched .. even my notification bell is on "ALL" but whenever I open my browser the first thing I do is to go here searching for any new video .. Believe it or not: One of the things I hate in KZbin that it does not permit us to add more than one like on each video. Sometimes I like to re-watch one or more of your videos. Every time I automatically hit the like button to find that I am mistakenly removing the old like. I hope that KZbin, one day, will permit us to add more than one like. Again thank you for the good useful content. Have a nice day. 🌹🌹🌹

[@leonvanzyl]

I wish I could give this comment more than 1 heart ❤️❤️. You have absolutely made my day!

[@iso-flow]

Man your flowise tutorials have been such a huge lifesaver. Thank you man!

[@leonvanzyl]

You're welcome 🤗

[@jofjofjof]

I learn a lot. Thanks Leon!

[@leonvanzyl]

Thank you for the support 🙏

[@Alvaro-cs7zs]

Another great video!! Leon, I don't know if you accept suggestions. But some of us have asked in the flowise discord channel for a video about how to manage when you have a lot of knowledge you want to include in the flowise flows. how to use different retrievals with different sources of information with just one agent. A video about that would be highly appreciate it!! Thanks Leon. Your work is amazing!

[@leonvanzyl]

You could just add more than one retrieval tool, one for each source?

[@nicolasgramnea1324]

Thank you for this. I was looking for a good solution to update file data. Your videos are always clear and easy to follow.

[@leonvanzyl]

You're welcome 🤗

[@JonnieMo]

Excellent tutorial! Very clear and thorough. Everything worked as described. thank you, Leon!

[@HemaSherbeny]

Please dont stop!!! i rearched all the internet to find an answer to (how to update vector db with new data) and never find a solution!! Really thanks Leon

[@leonvanzyl]

You're welcome 🤗

[@regman25]

Each of your videos is amazingly simple to understand. Thanks for your job Sir!

[@maniecronje]

Thank Leon always great to watch your videos, small thank you enjoy the coffee ☕️

[@leonvanzyl]

Thank you for the coffees Manie! ❤️

[@sdcharly]

Thanks!

[@leonvanzyl]

Thank you for the support ❤️

[@ThePandaGuitar]

Man your videos are so clear and well explained

[@MladenKirilov]

This was on my agenda to try today and you’ve posted 5min ago. Thanks for all the content that you make!

[@leonvanzyl]

Great minds think alike 😉

[@vskavo]

I was going to look how this works today too! Thanks Leon!!!

[@GilbertMizrahi]

Great video. Super helpful. Thanks for doing all these videos.

[@leonvanzyl]

You're welcome 🤗

[@JunaidAsghar-r5b]

Very helpful man, keep up the good work

[@leonvanzyl]

Thank you 🙏

[@sdcharly]

perfectly explained again @Leon !! loved it. not sure what to do with all the junk in the Pinecone already lol !!

[@leonvanzyl]

Clear the DB or use a new namespace 👍

[@ziayounasch]

Thanks a lot for such an informative video... Long awaited...

[@Obinna-ai]

Thanks for the video. It's quite insightful. Does it work with Document Stores? I have a document store linked to a Pinecone vector store, and a Postgres Record Manager linked to it, but the document store doesn't have the 'Additional Parameters' feature for adding metadata and stuff. How would I go about this?

[@WayneBruton]

Great video Leon, Thanks

[@leonvanzyl]

You're welcome

[@jimbob3823]

This was perfect thank you for explaining! Now I am stuck at how I apply "source" value to use incremental on multiple documents or say scraped content. Do you use the same method as the plain text input examples you demonstrated?

[@muhammadmursalin8915]

Excellent boss😊😊

[@RobertoFabrizi]

I don't understand how Record Managers works assuming that I have a pdf and add a paragraph somewhere in it. This newly added paragraph, depending on the text splitter and chunk_size/overlap, might have a waterfall effect of grouping the following (unchanged as far as their content/meaning) paragraphs in a different way, and therefore generate completely different hashes. How can Record Manager handle this?

[@estevegraells]

Amazing Leon. Just one question if you have time: as for the metadata for every file (I am uploading 300 PDFs) during the video you mention that we can use the "filename" as source, which it is a good idea. How could you use the filename into "source" node as parameter? Thanks in advance.

[@RaghvendraSingh21]

Hi Leon, thanks a lot..just checking this out. Can you also suggest some ways to use embedding model or tools that can ingest documents with graphs and pictures.

[@leonvanzyl]

You need to use the unstructured document loader for that. Will create a tutorial on it 👍

[@RaghvendraSingh21]

​@@leonvanzylgreat, that would be awesome!

[@mirensookdeo]

Another good tutorial Leon. Does this work with LlamaIndex as well or is it only for the LangChain components?

[@leonvanzyl]

It should work as long as the node has input for Record Manager 👍.

[@lucaciaraffo111]

Thank you very much really good and accurate explanation! Could you just take away one doubt for me? How would the Record Manager work with web pages? I tried hooking a scraping tool (cheerio) to load data from a web page. The first upsert loads the data, if I do it again it skips the data. It seems to rely on the URL. But what if i change the data within the page? For example if I update my price list on the page would there be a way for the record manager to understand that the data has changed so it has to delete the old data and add the new ones?Thanks a lot in advance!

[@leonvanzyl]

Thank you for the feedback! That is exactly what RM will do. When the website changes and you upsert again, RM will detect that the document has changed and therefore delete the old record and insert new doc.

[@lucaciaraffo111]

@@leonvanzyl Got it! Thank you Leon!

[@MsAntonio664]

OOHH!! THANKS!!! INCRIVEL.

[@siddhanthb682]

hello! great tutorial. can i get some guidance to connect a local postgres server with flowise instead of a cloud instance?

[@drtunmyatwin]

thanks for your video I am following what you teach in video and it works well but I just want to know that how to delete vector count. is this correct to delete index to delete vectors counts . What I want is just to delete vectors counts which are duplicated for testing not to delete index.

[@alexkimen654]

Thank you for the video. It's very useful. However, could you help me with a question: how can I use Flowise to take a large text file, split it into fragments, and then use AI to summarize/translate/abstract each fragment (I've understood up to this step), but I do not understand how to then combine all these summaries/abstracts into one text file and save it on disk. Thanks for your help.

[@ricardofernandez2286]

Hello Leon, I hope you are doing great!! Theoretically perfect! Thank you very much for the explanation!! However, I have a "fun" situation I'd like to share: I have configured everything in full cleanup mode, I have no errors, I execute the upsert and I get my 58 embeddings from 3 txt files added (shown in the result box as added, inserted into the vector database, chroma, and inserted in my record manager database, mysql, with the correct source metadata) So far, so good. Now I run the upsert again, no changes in the data or files, and I get 58 skipped, 58 deleted. And indeed all the records are deleted from everywhere. I would have expected only 58 skipped. Nothing to add, but all the files and data present in the upsert. I run the upsert again, 58 added. Run it again 58 skipped 58 deleted. And so on ... Your insights will be highly appreciated!!! Thank you very much!!

[@AliKibao]

Gems! Thank you.

[@leonvanzyl]

You're welcome 🤗

[@wilschoy78]

As always sir Leon very informative and useful. Hope if you can help me how to call an upsert method from external sources, currently I am working on moodle plugin development calling the html chatbot but my problem was the uploading part should be done inside moodle.Again thanks and keep uploading!!! God bless!!!

[@leonvanzyl]

Check out my API video.. you can call the API from outside of Flowise to upsert documents.

[@williamx0]

Is Flowise essentially a Visual Flow builder for the RunnableSequence method from langchain?

[@leonvanzyl]

Basically, yes. It uses Langchain with LCEL below the hood. It also implements LlamaIndex, so best of both worlds. Flowise used to be a Langchain prototyping platform, but it's since evolved into a full blown AI app builder.

[@cooperchien1]

I see that with one document update, Flowise record manager can still update the vector database with the new document content without any meta data configuration. I guess meta data is only important if 1. we update more than one document. 2. track what document must get updated This means to maintain the version of the data in the vector database, we also must keep track of all the meta data assigned to each document we upsert, right?

[@ramendersingh3072]

How to get the source key value from the body of the document? Say the input is a resume in pdf and I want to extract email address as the source key value.

[@sharannagarajan4089]

You are awesome

[@leonvanzyl]

No, YOU are awesome 😎

[@KratomSyndicate]

the chatbot needs to understand the current date, the records being added continuously can just have dates in which specials are active.

[@florentflote]

[@alexfratello1982]

these are great videos, I would love it if you could show me the possibility of creating a chatbot that also integrates within whatsapp with flowise, I would be grateful.

[@leonvanzyl]

I've been torn about this.. I do want to create a WhatsApp tutorial and have done this for clients. The problem is that it is very expensive and complex. Not just technically, you also need a registered company to even consider this process. I'll be creating the video for a very small amount of people who would realistically be in a position to follow along.

[@jimbob3823]

How about Telegram?