Feed Your OWN Documents to a Local Large Language Model!

[@krishnaprasadsomashiandan4660]

I went to bed yesterday night thinking about what other ways we can train the models, other than feeding the documents at the prompt - and woke up in the morning with this video being at the top of suggested videos in KZbin !

[@LeahleahB]

My radio in my car tells me what I'm thinking all the time! It's been listening to me for a long time I guess it knows me

[@bwcbiz]

I appreciate the practical hands-on approach to your AI videos. So many AI presentations are 90% hype (or more).

[@senseisecurityschool9337]

Agreed! And those that aren't hype require three years of careful study in the field before you can understand them. This video is perfect for just showing me how to accomplish what I want to get done.

[@rgsiiiya]

LOL, white text on blue background, watching the 70B model generate was like a total flashback to the bulletin board days on dialup/early isps.

[@KerboOnYT]

I've started playing around with a local LLM based on your other video. This was very helpful. Thanks!

[@RegalEase]

Dave, you've been reading my mind lately on topics I want to know more about. Thanks for making this!

[@MarkoVukovic0]

This is awesome, what a time to be alive! Much appreciated, thank you Dave!

[@javabeanz8549]

Thanks Dave! RAG now makes a lot more sense to me. This sounds like a way that would actually make AI LLMs useful to me.

[@thecompanioncube4211]

RAG is a godsend. And it’s necessary as new knowledge itself is not used for LLM training, and forget about your/your company’s specific knowledge

[@jonw0224]

Dave, this is great! I walked through everything this weekend, but my web-ui looked different. I didn't have a scan button under documents, but under workspace I had a "knowledge" tab where I would upload documents to my custom model. I think they must have just changed this functionality to make it easier to upload documents without copying to the backend because I could upload the documents from the web-ui and it automatically put them in an "upload" directory and used the document as a reference the same as your video. Overall, I think the change is a huge improvement. Thanks for putting this together, because I was able to use it to host my own AI at home using ollama and web-ui and build custom models using my own reference document. Keep up the great work!

[@tonysolar284]

Software updates and changes occur often due to the rapid development of AI tools, so every step in this tutorial could be obsolete with-in days, weeks, months...

[@annaczgli2983]

Thanks for the friendly introduction to this topic, Dave. Quite keen to try this stuff out now.

[@bobdemp8691]

Great demo. It shows that you had the knowledge to know when it gave a subpar answer. People going in without that knowledge won't have that ability though. It again shows that these are assistants that may need help, not replacements, as some are pitching LLMs as

[@MichaelCoIIins]

Absolutely awesome content you are pushing out on this Dave, thank you so much! Im a little worker bee at Apple with 0 programming skills and im using what im picking up from you to try and make our department a custom IA for some things. Thank you!

[@CristianAcosta-i9v]

Dave... greetings from Uruguay! I must say... you're the man! Thanks for this one!!!

[@MariaAileenMarina]

specswriter AI fixes this. Local large language model training

[@lmath56]

It is insane how fast AI is moving. Since you made the video Open WebUI have released an update for knowledge and document management, making it easier (you can now upload directly from the web interface)!! Thanks for making these videos, I love how clear and concise they are, as well as just entertaining!

[@davidboreham]

Dave: you can "bind mount" the documents directory into the container FS. Bind mount is docker-speak for "reflect a piece of the host FS into the container FS".

[@_pixelpicnic]

Love this - I'm hoping to set up my own in the next year! I'll need to watch this back and take notes! Cheers! I also loved the part when your blinds momentarily lost connection to wifi at 11:12.

[@terrysimons]

You can also map a local directory into a docker container at a specified path with the --volume flag.

[@jagannathdas5491]

So there are no limits to the size that can be uploaded? Drive is usually 10s or 100s GB.

[@erictayet]

For users who can't find the document tab in Workspace in V0.3.12, it is now called Knowledge and you can just upload your documents here even while in Docker. In Workspace, create your Model as per normal but choose the Knowledge Base that you've created and uploaded files into. Everything else works as normal.

[@maxdiamond55]

great stuff Dave, lots of things to try. Thanks for the direction.

[@Agililord]

@Dave's Garage i'm playing with this myself now. i'm having success adding docs via the Knowledge section for the workspace. then, when making a new model, it can reference that doc the same as when you scanned the folder. i did that because i am running the ui in a docker container. i know how to find and populate the volume mount, but i wasn't seeing a button to scan the directory. but the Knowledge section is another way to add the docs! great video!

[@Rawprawned]

Excellent, many thanks, great presentation, well done

[@andikunar7183]

Perfect explanation, thanks a lot! I did not know that open-webUI can do RAG, again thanks!

[@PITERPENN]

oh, just a straight and easy information. what a man, thanks

[@bwrscott1]

Thank you for the clear instruction and still bringing back memories of PDP11's.

[@nicennice]

Very useful and very clearly explained.

[@PauloEdson]

Thanks for being so clear and detailed in your presentation. Even someone like me without formal tech training can understand and follow the steps easily. 🙌 Liked. Subscribed. Waiting for more!

[@LambTjops]

You are an inspiration and legend!

[@josuevivas]

This is where I wanted AI to be two years ago. Great video. Once Sales directors, Project Managers and Product Owners realise that their company needs reliable and reviewed documentation to leverage LLMs, the nightmare of technical documentation will begin. I'm pro manuals and pristine technical documentation, but the majority of engineering teams NEVER document knowledge: I guess Knowledge Management will have a resurgence.

[@davidboreham]

AI can auto-generate the doc from the code.

[@terrorists-are-among-us]

They don't want to be replaced 😂

[@carelvanderpoel9953]

Thanx Dave. I think I finally found a way to get rid of all that zillions of user manuals of all kind of equipment (from kitchen to garden and everything in between). And still find an answer when I really need to use a manual.

[@TheZimberto]

Hey Dave, your vids are value packed - so much good info in under 20 minutes. I'd love to see a similar vid on AI image analysis - perhaps for use with security cameras, or with 3D printers to detect foul-ups etc.

[@MichaelTan]

Superb and honest, Dave. Rare.

[@skylineuk1485]

Great as ever Dave.

[@HaydonRyan]

Very informative and entertaining! Nice one!🎉

[@ScottGinn]

Love your clear explanation, and I might not have heard you correct on why you wanted to run Ollama locally rather than in a container... but you can just mount a volume to the docs rather than run locally. That would enable maximum flexibility.

[@manfredmuller9724]

So cool, you create an PDP-11 expert. I am old enough to remember this type of computer. The video was also very informative. Thank you.

[@caspersmith7112]

Dave I asked for this exact setup your talking about, dont know if you saw my comment but either way I REALLY appreciate this video.. It will help my create my own custom base of information for computer parts for gaming office and server ect.. SO GRACIOUS!!!

[@stevendonaldson1216]

Great video. Always great audio.

[@HaydonRyan]

Thankyou for running the other models. There’s a lack of people actually just running simple ollama comparisons from hardware.

[@frankkolmann4801]

I am both stunned and very very apprehensive. AI is beyond human.

[@kwazar6725]

Thanks Dave. This is what i was looking for.

[@RobertLindsley]

I have a Wix site with chat that I’d love to use my own LLM for. This is a terrific start to figuring all that out. Thanks Dave!

[@SoundFriendly]

Fantastic content Dave. Thank you!

[@richards6269]

SOLD, I'll install open webui . Thanks Dave

[@stevebarnes766]

Re: RAG - This is the first time anyone has shown me an actual USEFUL reason to use AI (aside from "Wow! It's really neat!) Thanks Dave!

[@photorealm]

Very useful information.

[@AngeloCarlotto]

Dude, thanks for your content. Just Great.

[@tutacat]

The retraining is called _finetuning._ If you don't need images and the OCR is good enough you can remove the images from documents, so it doesn't need to see the images.

[@neoxanthus]

I can tell you from experience, as this is my current field of expertise, that A100s are no longer a viable platform. The H100 is current and will be quickly replaced with the H200. This means that, on the second-hand market, you can pick up an A100 relatively inexpensively for all your home lab needs.

[@babybirdhome]

You had my hopes up that I might be able to make running the llama3.1 70b model viable at home, but so far those things are still going for $7,000 - $25,000 used. I guess I’m waiting until the H200s are widely deployed and/or getting long in the tooth, or until I win the lottery or something. Still, good to know that day is coming! I asked ChatGPT how the model’s speed would likely compare on my system with that vs. my current 3080 and the numbers are estimated to come in between 20-150x faster. I can hardly wait! That model takes ages to run natively on my i9, but the quality of the output is approaching functional for what I’d use it for, so I’m thinking something around there is probably near my bottom of the barrel target for the time being.

[@raduboboc]

Where is a good place to buy those A100s second hand? Is there a specific channel where we can find them? And how much do you estimate they will cost? As right now they are definitely outside the price possibilities of a homelab

[@adamsnook9542]

@@babybirdhome FWIW the 70B version of Llama3.1 ran on the system I have at home. I wouldn't say it was exactly fast, but it did work. Ryzen 9, 4080 Super and 32 giggs of RAM FWIW.

[@neoxanthus]

@@raduboboc A100 occasionally comes on eBay at a lower cost, depending on supply and demand. I consider them affordable for the home market used, considering that new they cost around 12k. However, you will have two other hurdles than the price you must overcome. An A100 does not use a standard PCIe power connector, even in the PCI form factor. It uses what is known as an EPS connector, which is the same as the CPU connector to the motherboard, and although you could jury rig something together from two PCI 8pin adaptors to EPS, it is not mainstream right now, so something would have to be custom made. Now, let us talk about cooling. These devices are meant to be in pressurized data centers with at least very high airflow and hot and cold aisles. This means that to use something like this at home, you would need to print a 3D shroud to interface a high CFM fan, and you may not want to hear that thing running. Also, keep in mind that the A100-PCIE-40GB is powered by the 7nm (TSMC) process node, which makes it relatively old compared to even consumer video cards like the RTX 4090 with 4nm (TSMC) and with more than twice the CUDA cores. I realize video memory is what you need in AI workloads, but there are confirmed reports of the 4090s after the market was modified to 48 GB of RAM for this exact reason.

[@magfal]

​@@adamsnook9542 At which level of accuracy did you run it? Q4?

[@mikesveganlife4359]

The tooling available today to make this so easy to do on your desktop computer is just amazing. The tech here is very dense and can be challenging to do without the tools available today.

[@tonyg_nerd]

Serious double take here. I worked with the PDP-11/34, recognized the reference immediately, and booted from the panel every day. 😊

[@JaimeWyant]

Excellent video, thanks Dave!

[@rnairn01]

Great info. Will start playing around with this info for sure.

[@tonyg_nerd]

Your pre-recorded summary following the RAG demo was about your Custom GPT. Good vid anyway. Thanks. 18:20

[@TroySkirchak]

I love your AI stuff. I have learned a lot about AI and how it works from your channel. THANK YOU!!!

[@marekjakimowicz]

That the informations i need. RAG is what i want. Thank you.

[@petergray2790]

Thanks Dave, I'm buying another mug...

[@doozowings4672]

Love being a sub, and love to like videos with this quality content. This is one of the most exciting projects I have got to play with in decades ... This AI stuff is the door to a whole new world of learning . LOVE IT !!!! I love that for now it is free....

[@Igbon5]

It took a while, but it turns out to be easy to add documents using web-ui in a docker container under windows. Just using the add documents + button. I don't know where they go, if anywhere but it does work. ChatGpt suggested it. It wasn't obvious on my screen layout, but it is there.

[@Ultimatebubs]

Thanks Dave, this is good to know!

[@Kauffy901]

Exactly what I needed, exactly when I needed it.

[@txkflier]

Fascinating! Thanks, Dave.. 👍

[@ArthurFlimbimlinson-x1r]

Very helpful. Thank you

[@RogerValor]

You could use docker and mount a physical drive/folder on the host as the doc folder or a subfolder of it, for which you probably need to adjust the Dockerfile setup, so you won't need to copy, if you have access to whatever runs the docker container

[@distantcomets]

Uh, wow! I had no idea this was possible. I'm about to build a gpt out of the manuals folder I keep for my music equipment. Mind blown.

[@oleleclos]

Thanks for this Dave, great inspiration for playing around with LLMs - and maybe even get something useful out of them :-)) One thought: If - as they claim - LLMs had already absorbed all the knowledge know to man, there should be nothing more to add… so this shows that’s just hype, like much of the rest ;-)

[@bogdantanasa1374]

thank you for the brief intro into setting up a custom GPT, online and local. but the openwebui+ollama combo as docker container is still fit, you just need to map your host folder to the openwebui container path, there's no need to COPY something INTO the container

[@nexxsys]

Create information Dave! The lastest version of OpenWeb UI has changed this a bit with Knowledge Collections. There is no scan button anymore under Admin Panel --> Settings --> Documents .

[@CristovamPeres]

So where is it?

[@RobCurrent]

Same, container or bare metal, seems to be gone.

[@eugrus]

Exactly when I needed it! 👍

[@ematta]

14:28 You can mount a local drive into docker easily with -v. It's dead simple to get cwd and have docker read it.

[@Leonardo_A1]

great Video Thanks a lot!! greetings from Germany

[@adrianolopes_sp]

Extremely interesting. Thank you

[@UserErrorGille]

I'd just use LM Studio. Has a UI, has built in RAG stuff, can easily turn on an API for local dev, etc. etc. etc.

[@4X6GP]

I downloaded Ollama and llama3.2 and asked it if it was running locally. No, it said, it could only run from a server, not locally. So then I pulled out my internet connection and continued to chat with it!

[@BBWahoo]

Hahah, nice!

[@nschul4]

NotebookLM also lets you upload documents into the context and chat with them. The interface is really great.

[@dennisfahey2379]

Fantastic Dave!

[@DanSnipe-k8o]

This is very interesting to me. I recently downloaded langchain to start playing with local documents.

[@Matt0x00]

If you really wanted to use your docker image you could mount the docs directory as a docker "volume" with docker run -v :

[@AIG-Development]

Great video, I did get Ollama. Very useful in work. However have had hickup in connecting the front end to the Ollama, which is required for RAG. Could we get a detailed process on that?

[@DaveEtchells]

Super interesting! A couple of practical questions: 1) How can you deal with constantly updating files? I’d like to just have my entire Obsidian directory stack accessible by RAG (including PDF attachments), but all the ongoing new documents and changes to existing ones would need to be ingested somehow. Can it do that incrementally, or would I just need to re-run the ingesting process? (Running overnight via a cron job would be fine, I’m on a Mac if that matters - and yeah, I know it’ll be slow on my M1 Pro/64gb) 2) Is there any way to have it provide a link, index or other reference to specific documents it found the answer on? My use case would be as much about access and retrieval as query and summarization. (3 - For unrelated bonus credit if any readers might have a solution: Any good way of integrating Google Docs with local Obsidian? I don’t want to use Obsidian’s publishing function, but would like to be able to integrate individual Google Docs into my Obsidian system.)

[@ah244895]

Going ro have to rewatch this...

[@HeavyD6600]

Thanks, Dave!

[@slappy0079]

I use Alpaca, would you be willing to make a video doing similar things on uncensored locally running LLMs? Great video! Thanks.

[@doublej42]

Great starter, thank you. The info on webui is out of date now. I had to create a knowledge . It did not do great but I'll just have to try various models.

[@jacobzindel987]

Amazingly good videos!

[@bluedunn374]

Now you’re talking! Nice work

[@DavidKing-wk1ws]

I had been using what would be considered more structured ai many years ago. Then it was called ultra hal. Today I no longer use that one I use a combination of semantic word net, conceptnet, xgboost, and distilli gpt2 for fluency. Using much less resources for a wearable pc.Still playing around with it, may be a little in the woods for many. Like Dave is pointing to you may need a much more powerful machine to get reasonable results. I am working with 32 and 64 gb machines for a wearable applications. Think gigabyte brix mini pc. While there are lower power consuming hardware, you half to strike a balance for the application you may be using it for and the results you may expect in my use context of a wearable pc. I will definitely be playing around with rag and my setup.

[@Rich-ld6mj]

Dave, great video! I have a question that I haven’t been able to find an answer to so far. I have about 60 eBooks, mainly in PDF and EPUB formats, along with some personal notes in text files. Since I started using ChatGPT, I’ve tried copying and pasting the text from one book into the engine, but this doesn’t work well since it can only handle a limited amount of text. Even when I try to slice it, it forgets the beginning as soon as I add more text. This is very frustrating, and I’m sure many users have experienced the same issue. The whole reason I wanted the AI engine to read all my books is to make it more useful for my research by extracting certain chapters or information that is most relevant to what I’m currently working on. I want to avoid manually reading all the books and taking hundreds of side notes and references every time I’m looking for something specific. Could you provide a video on how to do that? I was also wondering if there is a local GPT or AI engine that can simply add all my documents, scan them, and provide an intelligent way to analyze and output the results in a similar manner to ChatGPT. Please advise or let me know if you will be providing a video on this topic.

[@jahu33]

fundamental question, cause I'm confused - you've said that in RAG approach, when document is updated it's automatically pulled into the context, whereas IMO you still need to "scan it" again (create embeddings in vector db) so that updated chunks can be retrieved, am I right? And that's the challenge with RAG's, to keep vector embeddings db up-to-date (remove chunks that are not up-to-date etc.)

[@DytliefMoller]

Thanks @DavesGarage for making video ❤❤

[@davocc2405]

That's going to be fascinating when the concepts settle into a viable product for enterprise and SME clients; though there's going to be some potentially messy security implications though particularly if used in a legal field (so if a firm has confidential/sensitive documentation that is fed in or is operating with a Chinese Wall in place internally, even demonstrating the integrity of that would be awfully difficult). I'm keen for AI like this to be used as a starting point or an "ideas expander" rather than a be-all and end-all of expertise, to that end it'd be nice if it could respond with the actual documentation link references or index links - though it doesn't seem to be built that way at this stage which is fair I guess. I do love the notion of being able to chomp on company technical and archival documentation like this though - places that discarded or threw things out (purely for handling reasons) decided that on primitive estimations of retrieval and usage, best to err on the side of science fiction with that I think (unless disposal is required by legislation of course).

[@int_pro]

Do we have a way to feed video clips into a self hosted LLM yet? I have terabytes of recorded lectures and would love to be able to extract knowledge from these without needing to re-watch all the weeks, maybe months, of footage. 😅

[@RobCurrent]

There's a "sync dir" option now instead of "scan" for a folder, so you can keep your documents in place, I believe. However I do not know or understand the underlying mechanism yet.

[@georgeknerr]

OpenWebUI Docker Folder - you can start the container and map/mount an internal folder to a local

[@FlintStone-c3s]

I wonder if it can handle the my local copy of the Gutenberg Project?

[@ScottLahteine]

Scripting with Open Interpreter and Ollama in Python has been blowing my mind this week. The myriad of added value and possibilities this new toolbox provides are seemingly endless.

[@mgm-140-atacms]

in recent open-webui versions, there is no longer "docs" directory. the way you do RAG now is create "Knowledge" (look for a tab in "Workspace"), where you upload files via the open-webui web interface. during model creation, you then select this "knowledge" I've tried llama3.2 3B with ~1000 of podcast transcripts and I'm not impressed. my "system prompt" may be weak, but llama3.2, despite being very fast, is quite dumb and has poor reasoning. such a homelab "AI" solution is no competitor to chatgpt

[@adamsnook9542]

The numbers for the "tell me a story" test are quite interesting on the Big Box. FWIW my setup at home is a Ryzen 9 with a 4080 Super. It managed about half the performance of your box on the Llama 3.2 tests with 173.74 tokens/s on the short story and 168.20 on the long story. Your server trounced it on the 70B model though. The short story was a whopping 1.12 tokens/s and I really didn't have the patience to do the longer one.

[@ronm6585]

Thanks Dave.

[@sandryseibert]

I was able to install and run Ollama as you’ve explained on your previews video, however each time I reboot my machine I’m losing all my models. Couldn’t find a way to fix it

[@АлексейГриднев-и7р]

It's probably because you're running it in Docker which "resets" every time you restart it be default. Google "docker persistent changes"

[@JoePosillico]

Looks like the new version has a different way of working with documents. I see there is now an API to add the Documents, but they are not showing in the Knowledge section in 0.3.32

[@JoePosillico]

I figured it out, it wasn't very intuitive to me. First you go into workspace, then to the knowledge tab. From there you create a knowledge base. Once that is created you can add files to it. It can now be done exclusively in the UI, so thats nice.