LocalGPT Updates - Tips & Tricks

[@engineerprompt]

Want to connect? 💼Consulting: calendly.com/engineerprompt/consulting-call 🦾 Discord: discord.com/invite/t4eYQRUcXB ☕ Buy me a Coffee: ko-fi.com/promptengineering |🔴 Join Patreon: Patreon.com/PromptEngineering ▶ Subscribe: www.youtube.com/@engineerprompt?sub_confirmation=1

[@Shogun-C]

It'd be great if you could create a step by step series for all of this aimed at complete novices such as myself. Starting from the very beginning and assuming no prior knowledge or other supporting software already installed (e.g. git or conda).

[@YounessMazouz]

I support this request. Thank you

[@nattyzaddy6555]

There's many tutorials on how to get git, conda, and python setup

[@paulbishop7399]

kzbin.info/www/bejne/qHeok61vaKxsi5o

[@GregSimon-c1f]

How can I clear the cache from the last time I ran the model? I swapped all the docs with a new set of documents and my LOCALGPT model keeps giving me answers from the last set of docs which are no longer relevant for this version?

[@sadiqkavungal9256]

Please share what is the devicetype for intel IRIS GPU ?

[@shortthrow434]

Excellent work. Thank you for the walk through, especially on the mac/linux side. The modular approach will make much easier to maintain moving forward.

[@engineerprompt]

Glad you found it helpful. I agree, that’s the plan

[@kevinyuan2735]

谢谢！

[@engineerprompt]

Thank you 🙏

[@lalpremi]

thank you for sharing, have a great day 🙂

[@Nihilvs]

Thank you very much for this project, it's such a pleasure to use it !

[@engineerprompt]

Glad you like it!

[@hectornonayurbusiness2631]

For installing LLama CPP on windows, this worked for me: setx CMAKE_ARGS "-DLLAMA_CUBLAS=on" setx FORCE_CMAKE 1 pip install llama-cpp-python==0.1.83 --no-cache-dir Also if your computer defaults to using the cpu use --device_type cuda for windows Even with all that it kicks me out, BLAS=0

[@kevinfutero7166]

Same here .. I haven’t investigated why yet

[@Yeti9693]

@@kevinfutero7166 Any idea why its not using gpu?

[@venugopalasrinivasa7418]

This is very good project. How do we do fine tuning using Quantization?

[@MikeCreuzer]

Great update. I appreciate the video format for the updates like this. Thank you.

[@engineerprompt]

Thank you 🙏

[@shreyamahamuni1021]

Still im getting llm corpus data if I ask other than source document

[@42svb58]

Can this be used with dual K80 GPUs?

[@vasimraja6811]

what are factors to consider when choosing a cloud server for the above project? say am using Llama-2-7b-Chat-GGUF model, which instance is best? how much gpu memory is required?

[@vservicesvservices7095]

Do we need a tone of video ram for localgpt? Doesnt look like lenovo p51 can cope....

[@prestonmccauley43]

Great new changes. I have customized a lot of the core scripts myself, but maybe you could put an example that just shows how to access the persist database standalone? I was trying to run some data viz on my chroma, but running into issues understanding how to access it just standalone.

[@intellect5124]

can local gpt installed on windows11? very exiting video

[@MichealAngeloArts]

Great! Thanks for sharing! Does LocalGPT support code-tuned LLMs such as Codellama?

[@stephenthumb2912]

No reason it should not as long as you're using the right format, gguf, ggml, gptq, no clue on others like gpt4

[@engineerprompt]

Yes

[@deltawp]

Hello, thank you for this video and the brilliant work how is it possible to force the response of the model in a language other than English typically here French the model that I use mistral, 7B already knows how to respond in French but yet the most answers are in English

[@ps3301]

How is localgpt different from quivr project ?

[@stupidbollywood4066]

hey i have m1 8 gb ram model can i run a 4bit quantized model

[@edwardchan3521]

Hi, I like this project and wanna have a try in my notebook. However, it has 6GB VRAM only. Shall I use the openai API key? Being a layman to machine learning and Python, I would like to know any VRAM requirements for embeddings. If no need for VRAM, I may try on an old notebook without VRAM. Thanks a lot. :)

[@tiagoneto3902]

Congrats on the great content, it is flawless! 🎉 Are we able to connect the project with slack and build a chat bot?

[@ninahaller9913]

Hi, great video! You said if the "BLAS" variable is set to 1, llama.cpp uses my GPU, if it is set to 0 it is not. I have a M1 Mac and I want to run it on CPU, which I specify with --device_type cpu. However, BLAS is still set to 1. Can someone explain? The LLaMA 7B Chat model also takes forever to answer and if I load other models there is no answer at all.

[@engineerprompt]

BLAS=1 means that llamacpp is able to see your GPU. if you explicitly set device_type to cpu, then the code will use cpu. That might explain why its running so slow. How much RAM do you have on your system and what quantization level are you using?

[@khalidal-reemi3361]

How if I use ubuntu with cpu? Is there any change ? I am struggling with llama_cpp

[@shreyamahamuni1021]

Im using cpu getting errro like int object not collable

[@hassentangier3891]

can it produce structured article from prompts

[@Antaromran]

Thanks for sharing this, when is the support for falcon expected?

[@engineerprompt]

You should be able to run gguf and gguf version models including falcon even now

[@wasserbesser]

Amazing Work! It would be really awesome, if you could give it a GUI and a 1click installer, like other tools have it already. Like GPT4ALL or subtitle edit. For Mac, Windows and Linux. This would extend the userbase dramatically and give your more coffees ;)

[@engineerprompt]

Thanks for the idea! will see what I can put together.

[@jafizzle95]

@@engineerprompt Seconding this request. I've spent the last 18 hours installing (and uninstalling and reinstalling and then uninstalling and reinstalling again) literally hundreds of gigabytes of CUDA and Visual Studio nonsense trying to get this thing to work.

[@BabylonBaller]

@@engineerprompt I agree, it needs a GUI. Noone prefers a command prompt over a GUI

[@engineerprompt]

@@BabylonBaller There are two UI options now. One is via the API and the other is dedicated UI via streamlit. I am working on a gradio one that will make thing much more easier.

[@BabylonBaller]

@@engineerpromptmuch appreciated , will look into streamlit

[@Koorawithsedky]

This is exactly what I was looking for quite some time. I am just wondering if I can use it for generating Code in a specific structure, by ingesting the documents as .py or .java files and tries to use one of code generation model so that it can generate code in a specific structure as well as spot a snippet of code which is doing a particular functionality?

[@engineerprompt]

This is more of a search feature. Basically it will be looking for specific information in the document. You could use it to retrieve certain function or code snippet but then you will need to have a subsequent LLM call to use it.

[@blackstonesoftware7074]

How can I use an Arabic LLM with this ?? How would I set that up and what steps would I need to take? This is awesome!!!

[@sbacon92]

excellent updates. now can you write it in Nodejs?

[@engineerprompt]

I don’t have experience with Nodejs but hopefully someone can implement it

[@wennis89]

Thanks, this video was extremely helpful! Could you make a video on how to use another language than english for the local chatgpt? For example if I feed it with documents in Swedish, and I want to ask the question in Swedish and also get an answer in Swedish? Is that possible?

[@AI-Tech-Stack]

Should be the same just use the multilingual embedding models

[@engineerprompt]

Yes, as mentioned above you need an embedding model that supports the language you are working with as well as an llm

[@juandavidpenaranda6136]

@@engineerprompt Do you know which LLm from TheBloke (or in Hugging Face) provides answers in spanish? My docs are in spanish. I tried some models called Falcon, Roberta, Bert, but they are not compatible with localGPT. Thanks in advance. Amazing project

[@caiyu538]

Great, great ,great

[@photize]

Setx on visual studio gives a syntax error !

[@kevinfutero7166]

just use "set VARIABLE_NAME"

[@42svb58]

Would love the code walk through!

[@richierosewall3035]

If the pdf has the tables in it, it's not been extracted the same format as it is. What is the best way to ingest a pdf having tables which has lof of missing values?

[@engineerprompt]

Look at the unstructured loader for working with pdfs

[@richierosewall3035]

@@engineerprompt tried it. Row columns alignments are mismatched after loading. Bcz of that llm is giving incorrect response while asking n×m based questions.

[@malikrumi1206]

With regard to the splitting operation, I want to split on paragraphs, not random chunks. Can localgpt accommodate that out of the box, or would I need to hack your source code? Could I achieve my result with a decorator? Thx

[@engineerprompt]

It’s using the recursive character text splitter which uses paragraphs for splitting. So I think it will work for your use case

[@jimitnaik]

How well does it work on excel or CSV file? Overall great info and thanks for sharing an update.

[@engineerprompt]

This setup will work with cvs and excel files but you will need to experiment with embeddings and models for better performance

[@ImranKhan-wr2il]

Is anyone else struggling to run llamacpp on windows using cublas No matter what, blas is always 0 😐

[@engineerprompt]

If you have NVidia gpu, my recommendation is to use gptq models

[@ImranKhan-wr2il]

@@engineerprompt Gptq in giving nice inference speed on my nvidia 3070Ti but I am struggling to use conversationbuffermemory with it. What options do we have for memory for GPTQ models.

[@broomandmopmop]

can you release the code you are showing every example I am finding using openai I made several changes locally to localgpt but I would like to see this spin on it

[@engineerprompt]

Code is in the localgpt repo

[@broomandmopmop]

Oh ok my sorry busy day writing code when i seen this KZbin video notification I thought was new code sorry@@engineerprompt

[@ayanbandyapadhyay]

I like to have a cup of coffee with you. Great.

[@craig6095]

Dude you gotta start learning to put relevant links to your videos.

[@vivekkawathalkar9071]

Can i use this with pdfs and docs having complex tables and images