Web Scraping for LLM in 2024: Jina AI Reader API, Mendable Firecrawl, and Crawl4AI and More

Рет қаралды 24,664

Күн бұрын

Пікірлер: 43

@engineerprompt 8 ай бұрын

If you want to build robust RAG applications based on your own datasets, this is for you: prompt-s-site.thinkific.com/courses/rag

@lorenzo.padoan 6 ай бұрын

Thanks for mentioning ScrapeGraphAI, I'm one of the co-founders, we have implemented new features like code generator for scraping to minimize the number of calls to LLM on sites that have a shared structure on different pages, we are preparing something big related to KG, stay tuned :))))

@unclecode 8 ай бұрын

Thanks for mentioning Crawl4Ai! I'm adding some new features, such as extracting all media tags (video, image, audio), Breadth-First Search (BFS) Crawling, and more. I do it with the aim to generate quality data without relying on large language models (LLM). I think firing up GPUs for just crawling data from a page with billions of parameters is a bit over the top. Developers can use LLMs themselves once they have the right raw data from web sources.

@engineerprompt 7 ай бұрын

Crawl4AI is shaping up pretty nicely. I will do a deep dive on it.

@mjacfardk 8 ай бұрын

Yes PLEASE, Do a videos on {Crawl4Ai and ScrapeGraphAI}, and thank you for everything you do and your time 🙏

@engineerprompt 8 ай бұрын

Yes, its on my list.

@TimTruth 8 ай бұрын

I just use selenium web driver and JavaScript or Jquery to interact with and get the parts of pages I want. If they use cloud flare or other bot blocking you can run js in console and utilize the copy command then paste in a txt file

@d.d.z. 8 ай бұрын

Is there any path for learning you can recommend me? i´m generating reports from a web using python, looking for an alternative. Thanks in advance.

@ahassan7270 8 ай бұрын

Thank you so much for sharing this valuable information. It is absolutely helpful.

@engineerprompt 7 ай бұрын

Glad it was helpful!

@GetzAI 8 ай бұрын

Great review. Please do a review on ScrapeGraphAI. Maybe a comparison to Uncle Code's Crawl4AI? I like Crawl4AI and hope UC incorporates PDF options.

@engineerprompt 8 ай бұрын

thanks, yes, both of them are on my TODO list.

@beemerrox 7 ай бұрын

Nice comparison! Please continue work on scraping for AI applications. Hot topic!

@engineerprompt 7 ай бұрын

thanks, will do

@sethhavens1574 6 ай бұрын

Super handy, thanks 🙏

@jarad4621 8 ай бұрын

For jina reader Api key free for 1 million tokens which was 570 sites then pay 10 for 500 mil worth is 250k sites which is totally insane just pay the tiny amount for much better rate limits

@j4cks0n94 7 ай бұрын

Scrapegraph is pretty amazing, highly recommended

@MeinDeutschkurs 8 ай бұрын

Crawl4ai sounds perfect!

@SeeFoodDie 8 ай бұрын

Thank you. If you could dive deeper into scrapegraph, specifically the knowledge graph feature.

@engineerprompt 8 ай бұрын

thanks, will look into it.

@AJ-lg4zr 5 ай бұрын

Can you make a detailed video on scrapegraphai? It’s kinda buggy right now for me

@ahassan7270 8 ай бұрын

Thank you so much for sharing this valubale information. It is absouletly helpful. But, is it possible,as far as jina ai is concerned, to specify in the code the number of pages that I want to scrape, as spmetimes the pdf file has more than 500 pages .

@engineerprompt 8 ай бұрын

I am not sure, their api seems to be very simple and I haven't noticed any customizations yet.

@ai-whisperer 8 ай бұрын

brilliant 🙌🙌

@engineerprompt 8 ай бұрын

thanks :)

@bardaiart 7 ай бұрын

Thanks a lot! :)

@JPy90 8 ай бұрын

great thx!

@stefleur 8 ай бұрын

Probably a silly question, but in what is all this complicated proccess better than doing a simple copy paste from the url?

@engineerprompt 8 ай бұрын

There are a couple of reasons. 1. Even if you were to just copy and paste, the you not preserve the structure in most cases, there will be table, images etc which will mess up the formatting. 2. Even if copy paste were to give you perfect results, you can scale that to 100s or 10,000s of webpages. Using these automated tools, you need to provide list of urls and they will be able to parse at scale.

@hypnoz7871 2 ай бұрын

Good luck copy pasting+ cleaning millions of pages for llm feeding. Also good luck for manual updating :)

@chuckcarlson7940 6 ай бұрын

Do any of these solutions work on sites you have to log in to? You can give them a url, but if the site requires you to log in, you will not be able to scrape further.

@engineerprompt 6 ай бұрын

Good question, I am not sure. you might have to add authentication yourself to these.

@chuckcarlson7940 6 ай бұрын

@@engineerprompt If any of these solutions are Chromium based, then one could load the page, go through the authentication process, and select the page to be scraped. Then invoke the scraping tool.