Can Jina handle sites with lazy load? Looking at dealership websites
@alienPear18 күн бұрын
Thanks for sharing, bro! Greetings from Colombia
@devlearnllm18 күн бұрын
My pleasure!
@juroo18 күн бұрын
pure gold, thanks man!
@Breaking_Bold20 күн бұрын
I like this format of video...background has a large monitor...Nice video
@SergeyNumerov21 күн бұрын
I wonder how this would handle dynamic content: as in scraping websites where you have to click stuff to reveal valuable content.
@ratnpriyarai479322 күн бұрын
It was quite useful for me.
@__________________________691024 күн бұрын
No, I know why you want long-term memory-because your AI girlfriend doesn't remember your past conversations. She forgets you, that's why.
@devlearnllm11 күн бұрын
T.T
@ajkdrag26 күн бұрын
Hi. I have a video request. Is there a way to contact you?
@devlearnllm18 күн бұрын
tally.so/r/n9djRQ
@ajkdrag17 күн бұрын
@@devlearnllm done. Thanks
@robelbelay406529 күн бұрын
Great stuff man, thanks a lot!
@devlearnllm29 күн бұрын
Cheers!
@Phoenix-gi3guАй бұрын
For experimenting I would recommend using no database at all. You can simply use the cosine similarity (i.e. from torch functional) or quickly implement it and you are nearly done. Just use some argsort to get the best matches. It's like five lines of code or so. For easy store/load you can use pickle to serialize/unserialize the object that holds the embeddings. It is fast on CPU too, but of course you can run it on GPU without any bigger changes. No services required.
@devlearnllmАй бұрын
good point
@jonathanpark873Ай бұрын
I wonder if you would update it to be able to use gpt-4o-mini as its much cheaper
@devlearnllmАй бұрын
yep
@jakobkristensen2390Ай бұрын
Im curious how you handle pages where the content exceeds token window
@devlearnllmАй бұрын
I'm sure Firecrawl or Jina would have a rolling context window for extraction. It's an easy thing to implement.
@vijishmadhavan6093Ай бұрын
what happens if we use all the 25000 cases, will it work?
@devlearnllmАй бұрын
Most likely. Pinecone, Weaviate and pgvector are very performant.
@switch8291Ай бұрын
you havent updated us on how much does scrapegraph-ai takes in comparison
@devlearnllmАй бұрын
Ah shoot I forgot about that.
@antronx7Ай бұрын
Would be cool to make AI website scraper that strips away all javascript bloat from a webpage and converts it into lightweight basic html page while preserving functionality. Would be great as a proxy service to make loading modern web pages fast on slow phones on poor data connections. Modern web is way too bloated. I sometimes manually archive a page by deleting all javascript in notepad++ and modify image embed links to point to locally saved .png files. That takes a long time but I can reduce 5MB page down to 200kB and save that. Would be nice to have smart automated tool to do that in seconds.
@CkoraybingolАй бұрын
Great intro and work flow. Thanks a lot.
@devlearnllmАй бұрын
Much appreciated!
@MrPkmonsterАй бұрын
Thank you so much for the presentation. Just in time with the latest scraping technology
@devlearnllmАй бұрын
You bet!
@IkerCasilliasrocksАй бұрын
Why not use Chatgpt and ask it to find information. Cant GPT just search the web itself?
@manfredmichael_3ia097Ай бұрын
I think there should be a microphone in the middle of the audience. You had an insightful discussion with them, amazing audience!
@devlearnllmАй бұрын
Great idea!
@thisiswillАй бұрын
The motion-tracking is a bit distracting.
@zaid6527Ай бұрын
I dont know if my question is stupid, but can you tell me can we take snapshots of website and use ocr and llms to scrape the useful info, instead of sending request to that website since it would look more humanly , and also use less requests
@devlearnllmАй бұрын
Yeah you can probably do that!
@zaid6527Ай бұрын
@@devlearnllm thanks 🤝
@dhineshprabakaran1786Ай бұрын
Hi, I'm trying to scrape webdata from my Org Docs which is accessible only within VPN. Failed to goto 'docs url'. Can you help me with this ?
@ThoughtfullySoАй бұрын
You should've tried Qdrant.
@artur502 ай бұрын
is it possible to run it with Ollama?
@devlearnllm2 ай бұрын
Most likely
@ArunKumar-bp5lo2 ай бұрын
great
@ofrylivney3672 ай бұрын
Nice workshop! I'll definitely try out the hybrid search. Do you recon it'll work with nomic text embeddings and ollama?
@devlearnllm2 ай бұрын
Most likely!
@zuowang51852 ай бұрын
Is Openai embedding v3 model better than Bert?
@devlearnllm2 ай бұрын
Hard to tell unless experiments are run. huggingface.co/spaces/mteb/leaderboard
@JamesRBentley2 ай бұрын
Nice video sir. I have already been experimenting with the colab - sincerest thanks
@devlearnllm2 ай бұрын
Great to hear!
@devlearnllm2 ай бұрын
Hey yall, in case you didn't get good full text search results like me, the CEO of Supabase (Paul Copplestone) sent me this to use instead: supabase.com/docs/guides/database/extensions/pgroonga
@florianhonicke54482 ай бұрын
@LLMs for Devs. I'm from Jina AI. Cool that you are using our reader app. I like seeing the exact use-cases people use that one - very interesting.
@devlearnllm2 ай бұрын
Big fan of Jina.
@jeffc173628 күн бұрын
@@devlearnllm hi times are tough. can I borrow 10000k? I need rent money and lost my job as a retail worker at Dicks sporting goods in dallas.
@gregmeldrum2 ай бұрын
Very informative! A great resource. Thanks for sharing your wealth of knowledge!!
@flor.77972 ай бұрын
I just use Google 🙃
@ironbondar2 ай бұрын
very good workshop. straight to the point
@You.Got.Lucky_2 ай бұрын
This video was really helpful for the people like me looking for webscrapping tools. Though I wonder if jinaAi is really free. Is there any challenge in using it for more number of links? Does it have rate limit on hitting urls with prefix? Any clarification on this is appreciated. : )
@devlearnllm2 ай бұрын
No hard limits as far as I know. Free for now (I think this is intentional), but definitely will change in the future.
@danieldesenna76112 ай бұрын
Great video!! Thanks for sharing the code! One question though: Inside A:tier code -> "print_ai_answer" function, you wrote: for like in extracted_personality["likes"]: text_to_embed = f"The user likes {like}" current_embeddings = embedding_client.embed_query(text_to_embed) dislike_with_metadata = { "id": str(uuid.uuid4()), "values": current_embeddings, "metadata": {"type": "likes", "content": like} } embeddings.append(dislike_with_metadata) Was it not supposed to be something like "likes_with_metadata ={...}" and then "embeddings.append(likes_with_metadata)" ? I guess repeating "dislike_with_metadata" does not make a difference for the code functionality, but it was a bit confusing to understand the code for a moment. Thanks!
@devlearnllm2 ай бұрын
Good catch!
@nve-c5d2 ай бұрын
so what did you find out about scrapegraph ai performance , tokens
@chanliah19182 ай бұрын
Great demo, thank you!
@devlearnllm2 ай бұрын
My pleasure!
@frasonfrancis96982 ай бұрын
I don’t know how effective will this be in a long run especially due to the security update of cloudflare to block AI web scraping agents
@shuaiwang40922 ай бұрын
So valueable video content! Many thanks for sharing~~
@SonGoku-pc7jl2 ай бұрын
thansk, but difference or what is better gina reader or Scrapegraph-ai
@theadaloguy2 ай бұрын
Great video, thanks. Is there a way to provide our own scraped data (so we can make sure we use a good stealth scraper and get all the content), and then the LLM analyses it like this?
@devlearnllm2 ай бұрын
Yeah, you can always just build an LLM chain to just extract data. You can find the example in the Google Colab I provided.
@lomash_irl2 ай бұрын
I guess selenium is still the choice for javascript heavy websites... any tips on this?
@SurajSingh-y3n3e3 ай бұрын
bro i watched 4 minutes add before jumping actual video
@devlearnllm2 ай бұрын
That's crazy. Let me see if I can change that somehow
@ronaldokun3 ай бұрын
Thank you!!!!!!
@eyoo3693 ай бұрын
Jina is almost perfect.. too bad it's not smart enough to scrape content from "accordions" where you first click to make the content visible. I feel a smart AI scraper should be able to grab that text and determine based on CSS class that it's probably valuable text.. just hidden at the time
@devlearnllm3 ай бұрын
That's too bad. What's the alternative?
@PaulFidika3 ай бұрын
"The entire internet hates him for this one simple trick"
@devlearnllm3 ай бұрын
9/10 prompt engineers recommend this
@mrRambleGamble3 ай бұрын
The camera moves too much
@devlearnllm3 ай бұрын
its the worst
@mrRambleGamble3 ай бұрын
@@devlearnllm Aside from that, great video.
@SuperLano983 ай бұрын
How is your final implementation? I’m really curious about it, because make sense to have this abstracted, but we have some differences between than that can make this process tricky
@devlearnllm3 ай бұрын
Honestly handy because I went from Pinecone -> pgvector, and with all the abstracted methods declared, I was confidence in flipping and nothing broke.
@SuperLano983 ай бұрын
@@devlearnllm Oh amazing, great news ! I will try to start with this implementation pattern, thank you !
@JohnMcclaned3 ай бұрын
such an inefficient and unreliable way to scrape the web
@rwz3 ай бұрын
Please do not move the camera all the time
@haganlife3 ай бұрын
Definitely loosen up the tracking to center. OSBTail?
@devlearnllm3 ай бұрын
It's actually built-into the DJI Pocket 3 camera. I just had it for a few weeks. Just need to find the settings for it.
@forrest7143 ай бұрын
@@devlearnllm change the follow speed to slow instead of fast.