Great stuff!! Ngrok is now asking for auth -- solved this by adding await asyncio.gather( run_process(['ngrok', 'config', 'add-authtoken','']) ) before: await asyncio.gather( run_process(['ollama', 'serve']), run_process(['ngrok', 'http', '--log', 'stderr', '11434']), )
@lamechemohh91139 ай бұрын
please i use windows how to use ngrok with him?
@FilipeBento9 ай бұрын
@@lamechemohh9113 you mean Ollama? You will need to run WSL2 (if you have any Win version that is not the Home edition).
@techwithmarco9 ай бұрын
Thanks a lot for adding this! Just pinned the comment
@techwithmarco8 ай бұрын
and by the way, I updated the github repository to reflect your proposals :)
@burncloud-com13 күн бұрын
@@techwithmarcoyou video is cool, I still have many question, can I contact you?
@SethuIyer958 ай бұрын
Thank you so much. I was killing my Intel mac with the LLM questions xD. This gives a good rest for it.
@techwithmarco8 ай бұрын
Perfect!
@techwithmarco10 ай бұрын
If you want to learn more about ollama.ai, head over to my initial video about it :) kzbin.info/www/bejne/rIbbcp55mMaaa9U
@d3mist0clesgee129 ай бұрын
great stuff bro, keep them coming, thanks again.
@techwithmarco9 ай бұрын
Thanks! I will :)
@mobilesales4696Ай бұрын
Tell me how can I add Tele-FLM-1T local llm model but directly install in Google colab and how host on server using Google colab and how can I put those address in any framework I mean how to configure it plz plz kindly tell me instructions plz I
@iamderrickfoo4 ай бұрын
This is awesome stuff! Would like to know after this up can we connect this to Webui or Anythingllm?
@thoufeekbaber85975 ай бұрын
Thank you. I could run this succesfully in the terminal, but how can access the model or the collab through jupyter notebook instance?
@renega9913 ай бұрын
Hi amazing stuff! Is there a way to connect the ngrok to jupyter notebook? Thanks!
@jeffsanaraujo8 ай бұрын
That’s a fantastic video! Do you know if Ollama has OpenAI API compliant endpoints? So we could use Google Colab as a “Backend-as-a-Service” for some time in our chatbots :) One way I saw people doing is to create a long audio (like 12 hours of audio), loading it in the Google colab, and giving it a play, it’s a silence audio. It seems to work to keep the session opened for more time.
@techwithmarco8 ай бұрын
There is currently an issue at the ollama gh project, so feel free to check that out and track the progress :) github.com/jmorganca/ollama/issues/305 And good tip with the audio sound, never thought of that ... 😄
@tuliomop8 ай бұрын
great tip
@CharlesDubois-f7p2 ай бұрын
How can I make this work with the ollama library in a python script? This works well when typing the prompts directly in the terminal, but my script still seems to run on my local instance.
@CharlesDubois-f7p2 ай бұрын
For anyone running into the same issue, I figured it out. I had to set the environement variable in the script with os.environ["OLLAMA_HOST"] = ngrok_url BEFORE importing ollama
@QHawk77 күн бұрын
*Great! Thanks, can you do it with kaggle? , and with a local notebook/VSC?* Any update to this ?
@ralfrath699Ай бұрын
I have win 10. How can I start the model?
@barskaracadag3923Ай бұрын
Hi, I am jsut curious what is gonna happen once collab kicks us from using the gpu. Restart it all?
@Shivam-bi5uo8 ай бұрын
how do i save the progress, because everytime i run it, it downloads the model all from the start?
@WhyWouldHeSayThat8 ай бұрын
use your google drive bro, pay for 100gb. its worth it if you're an ai guy
@omerfarukagtoprak23982 ай бұрын
Thank you Wonderful video!!
@SethuIyer958 ай бұрын
Thank you!
@إضاءةذهبية2 ай бұрын
very thanks, you help me alot!😍
@yanncotineau6 ай бұрын
i got a 403 forbidden error, but replacing run_process(['ngrok', 'http', '--log', 'stderr', '11434']) with run_process(['ngrok', 'http', '--log', 'stderr', '11434', '--host-header="localhost:11434"']) fixed it for me.
@tiagosanti36 ай бұрын
Fixed it for me too, thanks
@MR-kh8ve6 ай бұрын
for me worked too, thank you!
@nicholasdunaway26056 ай бұрын
THANK YOU
@Kursadysr5 ай бұрын
You are a life saver!!!
@techwithmarco5 ай бұрын
great spot! I already updated the script on github :)
@pathsvivi4 ай бұрын
Thanks for the video. One question though, how can I avoid downloading the language models every time I run Colab notebook? Can I save Ollama and its models in Google drive and retrieve them when running the notebook?
@kunalbhooshan966715 сағат бұрын
Can you add code for adding model from colabe rather then ollama
@aryanflory5 ай бұрын
hey, how to the export step on windows? I have the ollama installed
@biological-machine4 ай бұрын
just use "set OLLAMA_PATH=the_url"
@jameschan62772 ай бұрын
Please help if I use windows PC desktop, how can I open terminals like MAC?
@mellio196 ай бұрын
but can't run stable diffusion this way?
@bennguyen13137 ай бұрын
I imagine it's costly to run LLMs.. is there a limit on how much Google Colab will do for free? I'm interested in creating a Python application that uses AI.. from what I've read, I could use ChatGPT4 Assistant API and I as the developer would incur the cost whenever the app is used. Alternatively, I could host a model like Ollama, on my own computer or on the cloud (beam cloud/ Replicate/Streamlit/replit)? As a 3rd option, could Google Colab work in my situation? Is OpenAI's Assistant API totally different from the API to programmatically interact with llama2 , mistral , etc?
@AnonymousAccount51423 күн бұрын
has this stopped working? have they caught on to us?
@vg28126 ай бұрын
Error: something went wrong, please see the ollama server logs for details am getting this error after running export OLLAMA_HOST= ... what should i do????
@techwithmarco5 ай бұрын
See the other latest comments or check out the new version on github. Should resolve the issue :)
@vg28125 ай бұрын
@@techwithmarco okay I will check
@vg28125 ай бұрын
@@techwithmarco thank you for the reply
@asdfg1346on2 ай бұрын
can such a llm model be used in a web app not just in a terminal locally and how?
@attilavass69357 ай бұрын
How can we keep our downloaded LLMs permanently, eg. on a mounted Google Drive? It would speed up the start of inference in a new ollama server start.
@techwithmarco6 ай бұрын
Yes, that's a brilliant idea! You can save those in google drive with this snippet for example: import os # Mount Google Drive from google.colab import drive drive.mount('/content/drive') # Create a folder in the root directory !mkdir -p "/content/drive/My Drive/My Folder" # Start Ollama with a path where models are stored OLLAMA_MODELS=/content/drive/My Drive/My Folder ollama serve
@attilavass69356 ай бұрын
@@techwithmarco that's great, thank you! :)
@Codescord10 ай бұрын
can we just make it as api end point and create good frontend on top of it?
@techwithmarco10 ай бұрын
Yes, kind of. The url which is getting exposed via ngrok, it is also usable as url in front ends especially built for ollama.ai Check out my other ollama linked video, there I show how to start up a front end for that. (Last section)
@MultiverseMayhemtoyou9 ай бұрын
This is Fire Can you help me connect open Interpretur Like this with So I can Give access to my computer But it wont load my PC that much
@py_man8 ай бұрын
You can
@DCS-um9oc4 ай бұрын
i got windows machine, do i need ollama locally tooo?
@abhishekratan24966 ай бұрын
Very usefull video also the code btw i can't get it running on windows what would be the way to set OLLAMA_HOST variable on window set OLLAMA_HOST= "--" doesn't seem to work it still runs on local machine
@techwithmarco5 ай бұрын
I think it depends on the terminal and shell you are using. Are you using the standard windows terminal?
@TirthSheth1085 ай бұрын
Hii @@techwithmarco , thanks for chiming in. I'm actually experiencing the same issue as @abhishekratan2496 , but I'm running it on the Ubuntu terminal. Setting the OLLAMA_HOST variable doesn't seem to work for me either. Any insights on how to resolve this would be greatly appreciated! Thanks.
@techwithmarco5 ай бұрын
@@TirthSheth108 Okay that's weird. I just used it a few days ago and it worked perfectly. I'll investigate and let you know :)
@AllMindControl4 ай бұрын
did anyone figure this out? it just tells me that export is not a recognized command
@AfnanQasim-wk8nq4 ай бұрын
canw e load 70B model with this same technque ?
@harsh95589 ай бұрын
4:33 The model is downloading on colab or locally? Also can u plz tell what command changes will be there if we are using windows terminal?
@techwithmarco9 ай бұрын
The model is being downloaded on the remote machine (colab) The commands stay the same, if you use WSL2 on windows with ollama.
@groshatc5 ай бұрын
awesome man
@khushalsharma20319 ай бұрын
Thanks for the video. You mentioned disconnecting the runtime. So I am assuming google will itself shut the running notebook in few hours. Do you know for how many hours continuously we can run this?
@techwithmarco9 ай бұрын
I just googled because I did not know, but apparently 90 minutes if you do not interact, or absolute 12 hours
@khushalsharma20319 ай бұрын
@@techwithmarco so if we leave the server running colab tab ideal. I am assuming it will auto shut in 90 minutes.
@techwithmarco9 ай бұрын
Honestly I am not sure because I haven't used it for that long in one run. I would assume it will be up for 12 hours because the tunnel is working in the background and the jupyter notebook is still running :)
@clashgamers40729 ай бұрын
It will ask for are you a robot? captcha if you're inactive for a while , you could write a small javascript fn in browser to randomly click some ui elements but yeah 12 hours is the hardlimit after that you can't connect to a GPU instance for another day or so
@stargate-s84 ай бұрын
Found a Gem 💎
@asir9129Ай бұрын
Missed opportunity to say "say less" as supposed to "say no more", I think it sounds funnier
@techwithmarco28 күн бұрын
I really don't get it as I am not a native speaker 😂
@AlexandreCastanet8 ай бұрын
Good do you have idea to benmarch mixtral on colab ?
@techwithmarco8 ай бұрын
No sorry I am not that deep into AI stuff so that I know how to benchmark the performance 🥲
@thepsych36 ай бұрын
i get error like 403 forbidden
@ricardomorim94446 ай бұрын
replace: run_process(['ngrok', 'http', '--log', 'stderr', '11434']) with run_process(['ngrok', 'http', '--log', 'stderr', '11434', '--host-header="localhost:11434"']) That fixed it for me.
@paulopatto82835 ай бұрын
@@ricardomorim9444 tkx very much guys, solved my issue.
@lamechemohh91139 ай бұрын
Please what about windows user?
@techwithmarco9 ай бұрын
You can use ollama with WSL2, it is not available yet in windows