I went to bed yesterday night thinking about what other ways we can train the models, other than feeding the documents at the prompt - and woke up in the morning with this video being at the top of suggested videos in KZbin !
@LeahleahBКүн бұрын
My radio in my car tells me what I'm thinking all the time! It's been listening to me for a long time I guess it knows me
@bwcbiz13 күн бұрын
I appreciate the practical hands-on approach to your AI videos. So many AI presentations are 90% hype (or more).
@senseisecurityschool933710 сағат бұрын
Agreed! And those that aren't hype require three years of careful study in the field before you can understand them. This video is perfect for just showing me how to accomplish what I want to get done.
@rgsiiiya17 күн бұрын
LOL, white text on blue background, watching the 70B model generate was like a total flashback to the bulletin board days on dialup/early isps.
@KerboOnYT17 күн бұрын
I've started playing around with a local LLM based on your other video. This was very helpful. Thanks!
@RegalEase17 күн бұрын
Dave, you've been reading my mind lately on topics I want to know more about. Thanks for making this!
@MarkoVukovic08 күн бұрын
This is awesome, what a time to be alive! Much appreciated, thank you Dave!
@javabeanz854917 күн бұрын
Thanks Dave! RAG now makes a lot more sense to me. This sounds like a way that would actually make AI LLMs useful to me.
@thecompanioncube42112 күн бұрын
RAG is a godsend. And it’s necessary as new knowledge itself is not used for LLM training, and forget about your/your company’s specific knowledge
@jonw022412 күн бұрын
Dave, this is great! I walked through everything this weekend, but my web-ui looked different. I didn't have a scan button under documents, but under workspace I had a "knowledge" tab where I would upload documents to my custom model. I think they must have just changed this functionality to make it easier to upload documents without copying to the backend because I could upload the documents from the web-ui and it automatically put them in an "upload" directory and used the document as a reference the same as your video. Overall, I think the change is a huge improvement. Thanks for putting this together, because I was able to use it to host my own AI at home using ollama and web-ui and build custom models using my own reference document. Keep up the great work!
@tonysolar2844 күн бұрын
Software updates and changes occur often due to the rapid development of AI tools, so every step in this tutorial could be obsolete with-in days, weeks, months...
@annaczgli298314 күн бұрын
Thanks for the friendly introduction to this topic, Dave. Quite keen to try this stuff out now.
@bobdemp869111 күн бұрын
Great demo. It shows that you had the knowledge to know when it gave a subpar answer. People going in without that knowledge won't have that ability though. It again shows that these are assistants that may need help, not replacements, as some are pitching LLMs as
@MichaelCoIIins17 күн бұрын
Absolutely awesome content you are pushing out on this Dave, thank you so much! Im a little worker bee at Apple with 0 programming skills and im using what im picking up from you to try and make our department a custom IA for some things. Thank you!
@CristianAcosta-i9v17 күн бұрын
Dave... greetings from Uruguay! I must say... you're the man! Thanks for this one!!!
@MariaAileenMarina15 күн бұрын
specswriter AI fixes this. Local large language model training
@lmath5610 күн бұрын
It is insane how fast AI is moving. Since you made the video Open WebUI have released an update for knowledge and document management, making it easier (you can now upload directly from the web interface)!! Thanks for making these videos, I love how clear and concise they are, as well as just entertaining!
@davidboreham11 күн бұрын
Dave: you can "bind mount" the documents directory into the container FS. Bind mount is docker-speak for "reflect a piece of the host FS into the container FS".
@_pixelpicnic8 күн бұрын
Love this - I'm hoping to set up my own in the next year! I'll need to watch this back and take notes! Cheers! I also loved the part when your blinds momentarily lost connection to wifi at 11:12.
@terrysimons15 күн бұрын
You can also map a local directory into a docker container at a specified path with the --volume flag.
@jagannathdas54914 күн бұрын
So there are no limits to the size that can be uploaded? Drive is usually 10s or 100s GB.
@erictayet3 күн бұрын
For users who can't find the document tab in Workspace in V0.3.12, it is now called Knowledge and you can just upload your documents here even while in Docker. In Workspace, create your Model as per normal but choose the Knowledge Base that you've created and uploaded files into. Everything else works as normal.
@maxdiamond5517 күн бұрын
great stuff Dave, lots of things to try. Thanks for the direction.
@Agililord16 күн бұрын
@Dave's Garage i'm playing with this myself now. i'm having success adding docs via the Knowledge section for the workspace. then, when making a new model, it can reference that doc the same as when you scanned the folder. i did that because i am running the ui in a docker container. i know how to find and populate the volume mount, but i wasn't seeing a button to scan the directory. but the Knowledge section is another way to add the docs! great video!
@Rawprawned11 күн бұрын
Excellent, many thanks, great presentation, well done
@andikunar718317 күн бұрын
Perfect explanation, thanks a lot! I did not know that open-webUI can do RAG, again thanks!
@PITERPENN11 күн бұрын
oh, just a straight and easy information. what a man, thanks
@bwrscott115 күн бұрын
Thank you for the clear instruction and still bringing back memories of PDP11's.
@nicennice10 күн бұрын
Very useful and very clearly explained.
@PauloEdson7 күн бұрын
Thanks for being so clear and detailed in your presentation. Even someone like me without formal tech training can understand and follow the steps easily. 🙌 Liked. Subscribed. Waiting for more!
@LambTjops10 күн бұрын
You are an inspiration and legend!
@josuevivas17 күн бұрын
This is where I wanted AI to be two years ago. Great video. Once Sales directors, Project Managers and Product Owners realise that their company needs reliable and reviewed documentation to leverage LLMs, the nightmare of technical documentation will begin. I'm pro manuals and pristine technical documentation, but the majority of engineering teams NEVER document knowledge: I guess Knowledge Management will have a resurgence.
@davidboreham11 күн бұрын
AI can auto-generate the doc from the code.
@terrorists-are-among-us7 күн бұрын
They don't want to be replaced 😂
@carelvanderpoel995313 күн бұрын
Thanx Dave. I think I finally found a way to get rid of all that zillions of user manuals of all kind of equipment (from kitchen to garden and everything in between). And still find an answer when I really need to use a manual.
@TheZimberto16 күн бұрын
Hey Dave, your vids are value packed - so much good info in under 20 minutes. I'd love to see a similar vid on AI image analysis - perhaps for use with security cameras, or with 3D printers to detect foul-ups etc.
@MichaelTan16 күн бұрын
Superb and honest, Dave. Rare.
@skylineuk148510 күн бұрын
Great as ever Dave.
@HaydonRyan14 күн бұрын
Very informative and entertaining! Nice one!🎉
@ScottGinn11 күн бұрын
Love your clear explanation, and I might not have heard you correct on why you wanted to run Ollama locally rather than in a container... but you can just mount a volume to the docs rather than run locally. That would enable maximum flexibility.
@manfredmuller972414 күн бұрын
So cool, you create an PDP-11 expert. I am old enough to remember this type of computer. The video was also very informative. Thank you.
@caspersmith711217 күн бұрын
Dave I asked for this exact setup your talking about, dont know if you saw my comment but either way I REALLY appreciate this video.. It will help my create my own custom base of information for computer parts for gaming office and server ect.. SO GRACIOUS!!!
@stevendonaldson121613 күн бұрын
Great video. Always great audio.
@HaydonRyan14 күн бұрын
Thankyou for running the other models. There’s a lack of people actually just running simple ollama comparisons from hardware.
@frankkolmann480116 күн бұрын
I am both stunned and very very apprehensive. AI is beyond human.
@kwazar672517 күн бұрын
Thanks Dave. This is what i was looking for.
@RobertLindsley15 күн бұрын
I have a Wix site with chat that I’d love to use my own LLM for. This is a terrific start to figuring all that out. Thanks Dave!
@SoundFriendly17 күн бұрын
Fantastic content Dave. Thank you!
@richards626916 күн бұрын
SOLD, I'll install open webui . Thanks Dave
@stevebarnes76611 күн бұрын
Re: RAG - This is the first time anyone has shown me an actual USEFUL reason to use AI (aside from "Wow! It's really neat!) Thanks Dave!
@photorealm8 күн бұрын
Very useful information.
@AngeloCarlotto9 күн бұрын
Dude, thanks for your content. Just Great.
@tutacat14 сағат бұрын
The retraining is called _finetuning._ If you don't need images and the OCR is good enough you can remove the images from documents, so it doesn't need to see the images.
@neoxanthus17 күн бұрын
I can tell you from experience, as this is my current field of expertise, that A100s are no longer a viable platform. The H100 is current and will be quickly replaced with the H200. This means that, on the second-hand market, you can pick up an A100 relatively inexpensively for all your home lab needs.
@babybirdhome17 күн бұрын
You had my hopes up that I might be able to make running the llama3.1 70b model viable at home, but so far those things are still going for $7,000 - $25,000 used. I guess I’m waiting until the H200s are widely deployed and/or getting long in the tooth, or until I win the lottery or something. Still, good to know that day is coming! I asked ChatGPT how the model’s speed would likely compare on my system with that vs. my current 3080 and the numbers are estimated to come in between 20-150x faster. I can hardly wait! That model takes ages to run natively on my i9, but the quality of the output is approaching functional for what I’d use it for, so I’m thinking something around there is probably near my bottom of the barrel target for the time being.
@raduboboc16 күн бұрын
Where is a good place to buy those A100s second hand? Is there a specific channel where we can find them? And how much do you estimate they will cost? As right now they are definitely outside the price possibilities of a homelab
@adamsnook954216 күн бұрын
@@babybirdhome FWIW the 70B version of Llama3.1 ran on the system I have at home. I wouldn't say it was exactly fast, but it did work. Ryzen 9, 4080 Super and 32 giggs of RAM FWIW.
@neoxanthus15 күн бұрын
@@raduboboc A100 occasionally comes on eBay at a lower cost, depending on supply and demand. I consider them affordable for the home market used, considering that new they cost around 12k. However, you will have two other hurdles than the price you must overcome. An A100 does not use a standard PCIe power connector, even in the PCI form factor. It uses what is known as an EPS connector, which is the same as the CPU connector to the motherboard, and although you could jury rig something together from two PCI 8pin adaptors to EPS, it is not mainstream right now, so something would have to be custom made. Now, let us talk about cooling. These devices are meant to be in pressurized data centers with at least very high airflow and hot and cold aisles. This means that to use something like this at home, you would need to print a 3D shroud to interface a high CFM fan, and you may not want to hear that thing running. Also, keep in mind that the A100-PCIE-40GB is powered by the 7nm (TSMC) process node, which makes it relatively old compared to even consumer video cards like the RTX 4090 with 4nm (TSMC) and with more than twice the CUDA cores. I realize video memory is what you need in AI workloads, but there are confirmed reports of the 4090s after the market was modified to 48 GB of RAM for this exact reason.
@magfal15 күн бұрын
@@adamsnook9542 At which level of accuracy did you run it? Q4?
@mikesveganlife435917 күн бұрын
The tooling available today to make this so easy to do on your desktop computer is just amazing. The tech here is very dense and can be challenging to do without the tools available today.
@tonyg_nerd7 күн бұрын
Serious double take here. I worked with the PDP-11/34, recognized the reference immediately, and booted from the panel every day. 😊
@JaimeWyant17 күн бұрын
Excellent video, thanks Dave!
@rnairn0116 күн бұрын
Great info. Will start playing around with this info for sure.
@tonyg_nerd7 күн бұрын
Your pre-recorded summary following the RAG demo was about your Custom GPT. Good vid anyway. Thanks. 18:20
@TroySkirchak16 күн бұрын
I love your AI stuff. I have learned a lot about AI and how it works from your channel. THANK YOU!!!
@marekjakimowicz17 күн бұрын
That the informations i need. RAG is what i want. Thank you.
@petergray279013 күн бұрын
Thanks Dave, I'm buying another mug...
@doozowings467217 күн бұрын
Love being a sub, and love to like videos with this quality content. This is one of the most exciting projects I have got to play with in decades ... This AI stuff is the door to a whole new world of learning . LOVE IT !!!! I love that for now it is free....
@Igbon516 күн бұрын
It took a while, but it turns out to be easy to add documents using web-ui in a docker container under windows. Just using the add documents + button. I don't know where they go, if anywhere but it does work. ChatGpt suggested it. It wasn't obvious on my screen layout, but it is there.
@Ultimatebubs17 күн бұрын
Thanks Dave, this is good to know!
@Kauffy90110 күн бұрын
Exactly what I needed, exactly when I needed it.
@txkflier17 күн бұрын
Fascinating! Thanks, Dave.. 👍
@ArthurFlimbimlinson-x1r11 күн бұрын
Very helpful. Thank you
@RogerValor9 күн бұрын
You could use docker and mount a physical drive/folder on the host as the doc folder or a subfolder of it, for which you probably need to adjust the Dockerfile setup, so you won't need to copy, if you have access to whatever runs the docker container
@distantcomets9 күн бұрын
Uh, wow! I had no idea this was possible. I'm about to build a gpt out of the manuals folder I keep for my music equipment. Mind blown.
@oleleclos6 күн бұрын
Thanks for this Dave, great inspiration for playing around with LLMs - and maybe even get something useful out of them :-)) One thought: If - as they claim - LLMs had already absorbed all the knowledge know to man, there should be nothing more to add… so this shows that’s just hype, like much of the rest ;-)
@bogdantanasa137414 күн бұрын
thank you for the brief intro into setting up a custom GPT, online and local. but the openwebui+ollama combo as docker container is still fit, you just need to map your host folder to the openwebui container path, there's no need to COPY something INTO the container
@nexxsys16 күн бұрын
Create information Dave! The lastest version of OpenWeb UI has changed this a bit with Knowledge Collections. There is no scan button anymore under Admin Panel --> Settings --> Documents .
@CristovamPeres16 күн бұрын
So where is it?
@RobCurrent11 күн бұрын
Same, container or bare metal, seems to be gone.
@eugrus14 күн бұрын
Exactly when I needed it! 👍
@ematta16 күн бұрын
14:28 You can mount a local drive into docker easily with -v. It's dead simple to get cwd and have docker read it.
@Leonardo_A110 күн бұрын
great Video Thanks a lot!! greetings from Germany
@adrianolopes_sp17 күн бұрын
Extremely interesting. Thank you
@UserErrorGille17 күн бұрын
I'd just use LM Studio. Has a UI, has built in RAG stuff, can easily turn on an API for local dev, etc. etc. etc.
@4X6GP17 күн бұрын
I downloaded Ollama and llama3.2 and asked it if it was running locally. No, it said, it could only run from a server, not locally. So then I pulled out my internet connection and continued to chat with it!
@BBWahoo16 сағат бұрын
Hahah, nice!
@nschul48 күн бұрын
NotebookLM also lets you upload documents into the context and chat with them. The interface is really great.
@dennisfahey237916 күн бұрын
Fantastic Dave!
@DanSnipe-k8o17 күн бұрын
This is very interesting to me. I recently downloaded langchain to start playing with local documents.
@Matt0x0017 күн бұрын
If you really wanted to use your docker image you could mount the docs directory as a docker "volume" with docker run -v :
@AIG-Development16 күн бұрын
Great video, I did get Ollama. Very useful in work. However have had hickup in connecting the front end to the Ollama, which is required for RAG. Could we get a detailed process on that?
@DaveEtchells10 күн бұрын
Super interesting! A couple of practical questions: 1) How can you deal with constantly updating files? I’d like to just have my entire Obsidian directory stack accessible by RAG (including PDF attachments), but all the ongoing new documents and changes to existing ones would need to be ingested somehow. Can it do that incrementally, or would I just need to re-run the ingesting process? (Running overnight via a cron job would be fine, I’m on a Mac if that matters - and yeah, I know it’ll be slow on my M1 Pro/64gb) 2) Is there any way to have it provide a link, index or other reference to specific documents it found the answer on? My use case would be as much about access and retrieval as query and summarization. (3 - For unrelated bonus credit if any readers might have a solution: Any good way of integrating Google Docs with local Obsidian? I don’t want to use Obsidian’s publishing function, but would like to be able to integrate individual Google Docs into my Obsidian system.)
@ah24489517 күн бұрын
Going ro have to rewatch this...
@HeavyD660017 күн бұрын
Thanks, Dave!
@slappy007915 күн бұрын
I use Alpaca, would you be willing to make a video doing similar things on uncensored locally running LLMs? Great video! Thanks.
@doublej4215 күн бұрын
Great starter, thank you. The info on webui is out of date now. I had to create a knowledge . It did not do great but I'll just have to try various models.
@jacobzindel98717 күн бұрын
Amazingly good videos!
@bluedunn37417 күн бұрын
Now you’re talking! Nice work
@DavidKing-wk1ws17 күн бұрын
I had been using what would be considered more structured ai many years ago. Then it was called ultra hal. Today I no longer use that one I use a combination of semantic word net, conceptnet, xgboost, and distilli gpt2 for fluency. Using much less resources for a wearable pc.Still playing around with it, may be a little in the woods for many. Like Dave is pointing to you may need a much more powerful machine to get reasonable results. I am working with 32 and 64 gb machines for a wearable applications. Think gigabyte brix mini pc. While there are lower power consuming hardware, you half to strike a balance for the application you may be using it for and the results you may expect in my use context of a wearable pc. I will definitely be playing around with rag and my setup.
@Rich-ld6mj12 күн бұрын
Dave, great video! I have a question that I haven’t been able to find an answer to so far. I have about 60 eBooks, mainly in PDF and EPUB formats, along with some personal notes in text files. Since I started using ChatGPT, I’ve tried copying and pasting the text from one book into the engine, but this doesn’t work well since it can only handle a limited amount of text. Even when I try to slice it, it forgets the beginning as soon as I add more text. This is very frustrating, and I’m sure many users have experienced the same issue. The whole reason I wanted the AI engine to read all my books is to make it more useful for my research by extracting certain chapters or information that is most relevant to what I’m currently working on. I want to avoid manually reading all the books and taking hundreds of side notes and references every time I’m looking for something specific. Could you provide a video on how to do that? I was also wondering if there is a local GPT or AI engine that can simply add all my documents, scan them, and provide an intelligent way to analyze and output the results in a similar manner to ChatGPT. Please advise or let me know if you will be providing a video on this topic.
@jahu336 күн бұрын
fundamental question, cause I'm confused - you've said that in RAG approach, when document is updated it's automatically pulled into the context, whereas IMO you still need to "scan it" again (create embeddings in vector db) so that updated chunks can be retrieved, am I right? And that's the challenge with RAG's, to keep vector embeddings db up-to-date (remove chunks that are not up-to-date etc.)
@DytliefMoller16 күн бұрын
Thanks @DavesGarage for making video ❤❤
@davocc24053 күн бұрын
That's going to be fascinating when the concepts settle into a viable product for enterprise and SME clients; though there's going to be some potentially messy security implications though particularly if used in a legal field (so if a firm has confidential/sensitive documentation that is fed in or is operating with a Chinese Wall in place internally, even demonstrating the integrity of that would be awfully difficult). I'm keen for AI like this to be used as a starting point or an "ideas expander" rather than a be-all and end-all of expertise, to that end it'd be nice if it could respond with the actual documentation link references or index links - though it doesn't seem to be built that way at this stage which is fair I guess. I do love the notion of being able to chomp on company technical and archival documentation like this though - places that discarded or threw things out (purely for handling reasons) decided that on primitive estimations of retrieval and usage, best to err on the side of science fiction with that I think (unless disposal is required by legislation of course).
@int_pro5 күн бұрын
Do we have a way to feed video clips into a self hosted LLM yet? I have terabytes of recorded lectures and would love to be able to extract knowledge from these without needing to re-watch all the weeks, maybe months, of footage. 😅
@RobCurrent11 күн бұрын
There's a "sync dir" option now instead of "scan" for a folder, so you can keep your documents in place, I believe. However I do not know or understand the underlying mechanism yet.
@georgeknerr8 күн бұрын
OpenWebUI Docker Folder - you can start the container and map/mount an internal folder to a local
@FlintStone-c3s16 күн бұрын
I wonder if it can handle the my local copy of the Gutenberg Project?
@ScottLahteine16 күн бұрын
Scripting with Open Interpreter and Ollama in Python has been blowing my mind this week. The myriad of added value and possibilities this new toolbox provides are seemingly endless.
@mgm-140-atacms14 күн бұрын
in recent open-webui versions, there is no longer "docs" directory. the way you do RAG now is create "Knowledge" (look for a tab in "Workspace"), where you upload files via the open-webui web interface. during model creation, you then select this "knowledge" I've tried llama3.2 3B with ~1000 of podcast transcripts and I'm not impressed. my "system prompt" may be weak, but llama3.2, despite being very fast, is quite dumb and has poor reasoning. such a homelab "AI" solution is no competitor to chatgpt
@adamsnook954216 күн бұрын
The numbers for the "tell me a story" test are quite interesting on the Big Box. FWIW my setup at home is a Ryzen 9 with a 4080 Super. It managed about half the performance of your box on the Llama 3.2 tests with 173.74 tokens/s on the short story and 168.20 on the long story. Your server trounced it on the 70B model though. The short story was a whopping 1.12 tokens/s and I really didn't have the patience to do the longer one.
@ronm658517 күн бұрын
Thanks Dave.
@sandryseibert17 күн бұрын
I was able to install and run Ollama as you’ve explained on your previews video, however each time I reboot my machine I’m losing all my models. Couldn’t find a way to fix it
@АлексейГриднев-и7р10 күн бұрын
It's probably because you're running it in Docker which "resets" every time you restart it be default. Google "docker persistent changes"
@JoePosillico16 күн бұрын
Looks like the new version has a different way of working with documents. I see there is now an API to add the Documents, but they are not showing in the Knowledge section in 0.3.32
@JoePosillico16 күн бұрын
I figured it out, it wasn't very intuitive to me. First you go into workspace, then to the knowledge tab. From there you create a knowledge base. Once that is created you can add files to it. It can now be done exclusively in the UI, so thats nice.