ElasticCC: Crawl the Web on a Large Scale with Stormcrawler and Elasticsearch

By Julien Nioche
StormCrawler is a popular and mature open-source web crawler. It is written in Java and is both lightweight and scalable, thanks to the distribution layer based on Apache Storm. One of the attractions of the crawler is that it is extensible and modular, as well as versatile. In this presentation, we will have a closer look at the Elasticsearch module of StormCrawler and see how it is being used in production by various organizations, sometimes on a very large scale.

[@NoName-ep2xp]

I'm continously surprised by the inability of the elasticsearch community to encourage people from outside the community to get involved. I'm a developer with 15years experience and most of the elasticsearch tutorials are close to useless as they demand so much effort. There seems to be no attempt to make the tutorials useful to anyone but people who already know about the complete tech stack (in which case it's a reference not a tutorial). I've looked at all your tutorials on stormcrawler and elasticsearch and you assume quite a lot: that I don't know about (or can be bothered to research) Storm, flux!, bolts, topology, elasticsearch, kiban, am a Java developer!. Do you really expect this as a prerequisite to the tutorial being useful? (or maybe I'm just too intellectually challenged). Whilst I could wade through all the stuff needed to work it out, for the simple problem I have (a simple search of a relatively few sites, I probably won't be bothered. I can see why AWS/opensearch believe they can kill elasticsearch given the poor value delivered by the elasticsearch community videos & documents. So that my contribution isn't totally negative, If you want some suggestions on how to improve/create your Stormcrawler videos for beginners I'm open to helping. Just reply and we'll work out a way to talk. Sorry if this is uncomfortable to read but this whole technical area is missing a usable on-ramp for beginners. Hours of research is not a usable path if you want this thing to grow beyond existing insiders. Cheers 🙂