What are Elasticsearch shards? Why do they matter? Elasticsearch cluster architecture explained.

Elasticsearch is a fantastic tool but it's easy to muddle through without knowing the fundamentals. It's only a matter of time before your cluster performance drops, errors start happening, and you wonder just what a shard actually is.
I talk and wave my hands about, explaining how we could have - maybe - built - some of - Elasticsearch ourselves. At the end of the video you'll know what a shard is, why they're so important, and want to learn more to really improve the performance of your cluster.
Watch on (and watch the rest of the free course linked below) to figure out how to answer that mysterious question: How many primary shards do I need for my index?
My course - Fundamentals of Elasticsearch architecture and shards - is available for free here: school.georgebridgeman.com/p/...
You can read more about Elasticsearch and The Elastic Stack in general on my blog: georgebridgeman.com/posts/

[@HaisumUsman]

Man! You are not from this planet! You deserve a thousand thumbs up.

[@nch77884]

Hands down the best explanation and introduction to Elasticsearch. Can't thank enough for making this video.

[@GeorgeBridgemanData]

Thank you for your lovely comment. I'm so pleased you enjoyed the video!

[@zaeemahmedabbasi]

​@@GeorgeBridgemanData⁷77777777777777777777777777777777777777777777777777777777777777777777😅😮

[@netlob]

Holy sh*t! This must've been the best tutorial I've ever seen on KZbin. High production, clear presentation and well thought through. +1 subscriber for sure!

[@GeorgeBridgemanData]

That's very kind of you to say. Thanks so much!

[@sridharnuthi1]

This is a video that should go to a reference library about Elasticsearch. Thank you for putting such a good, clear and methodical overview of ES. Just brilliant!

[@GeorgeBridgemanData]

Such a kind comment. Thank you so much!

[@harrisonleong4283]

I really wish this was the 1st elasticsearch video that I had watched, so as to save me so much time watching other video which could not teach me the same level of information that I need. Thank you very much, and I shall check out your courses.

[@GeorgeBridgemanData]

That's the reaction I was hoping for - a useful first video on Elasticsearch. Thank you so much for posting!

[@lawlade]

Watched this 5 times, rewound several times and i understand it FULLY. Thanks so much for such clear explaining

[@GeorgeBridgemanData]

You're welcome! I'm so pleased you understood everything. I hope it helps!

[@yazzy9975]

This video changed my life. No exaggeration.

[@blossomwithcurls]

I just started learning Elastic search and this is the best an clear information on Elastic search architecture. Thanks for sharing!

[@TheOtmane007]

What a clear , and progressively explained architecture. Thank you so much

[@Transactional]

Thank you. It feels like my brain is getting clearer.

[@user-tm6cd8sg5l]

Many thanks for your work! It's awesome video!

[@pseudolimao]

where was thsi video 1 month ago. you should be paid by these software companies... bless your heart

[@thomasanderson8478]

This is the best explanation of elasticsearch I've ever seen. So many videos skip over the details, and it's been making it difficult to understand what elasticsearch is doing under the hood. I normally don't comment on videos, but this is too high quality not to. Please continue to put out content!

[@GeorgeBridgemanData]

What a lovely comment! Thank you so much. It means a lot!

[@crujzojam7004]

Please post more videos…ur videos are easy to understand and quite informative….please carry on the good work

[@GeorgeBridgemanData]

Done! New video just posted! There will be more. I have lots of ideas but I'm really trying to get this my Elasticsearch course finished, and work the day job.

[@hieungo770]

Its two year late but the lesson is extremely value

[@user-nt7of9us8p]

Very clear! Thanks!

[@jhoyl]

Thanks - the perfect introduction to Elasticsearch architecture.

[@GeorgeBridgemanData]

Thank you!

[@jupudivinod]

This is fantastic! Bricks till walls in a nutshell! Thanks much for this great presentation.

[@GeorgeBridgemanData]

You're welcome! I'm really pleased you enjoyed it.

[@riazbacchus3962]

this is great content. thank you.

[@hnyc1986]

Awesome explanation about Elasticsearch!!!

[@GeorgeBridgemanData]

Thank you!

[@sobhan285]

Wonderful. Looking forward to more courses from you.

[@GeorgeBridgemanData]

Thanks so much! Elasticsearch Engineer Essentials is in the works, and I'll be posting shorter content on here as well.

[@thsu1]

thanks for the clear and awesome explanation to Elasticsearch and Lucene. really appreciate this useful content

[@GeorgeBridgemanData]

I'm so pleased you appreciated it. It was great to make it!

[@danielsantiago11]

Premium content, thank you!

[@tkousek1]

Thank you very much sir for this information. Awesome people like you are what's good about this world!!! Much appreciated!!!

[@GeorgeBridgemanData]

You're very welcome! Thanks for your kind words.

[@johnsonakanbi367]

Thanks so much for this great presentation.

[@GeorgeBridgemanData]

You're welcome. I'm glad you enjoyed it!

[@arpit9163]

Thank You for making this fantastic video !

[@GeorgeBridgemanData]

You're more than welcome. I'm glad you enjoyed it!

[@rakeshkush1234]

wonderful technical story.

[@GeorgeBridgemanData]

Thank you!

[@PhanTanThangTH]

Thank you so much :)

[@moritzlgrs401]

Absolutely fantastic!

[@GeorgeBridgemanData]

Thank you!

[@cicd]

Great content, thanks for sharing!

[@GeorgeBridgemanData]

My pleasure!

[@ucthuannguyen6432]

Wonderful. Thank you so much.

[@GeorgeBridgemanData]

You're very welcome!

[@andy_ltluan]

I think that shard in ES has the same concept with partition in Kafka when they have all partition replicas in different nodes

[@slapcanister]

This is so good.

[@GeorgeBridgemanData]

Thanks very much!

[@DrewIsFail]

Is it fair to say you could build ES from dynamoDB? I'm trying to compare the two. I would love a video on the query language, does it have a mathematical basis like sql does to sets? It goes without saying, but I'm say it, thanks for making this clear, concise, focused high level content.

[@GeorgeBridgemanData]

Hi there. I'm really pleased you enjoyed the video. I'm not sure you could build an equivalent of Elasticsearch using DynamoDB. There's a *lot* more to Elasticsearch than I talked about in this video! There's more content coming, including an introduction to the query language. The in-depth content will be in a training course instead of KZbin, though. I've never considered if there's a mathematical basis to the query language. I doubt there is in terms of what Elasticsearch offers, but all Elasticsearch queries are converted to Lucene query language, which may be more thoroughly researched. Interesting question!

[@yazzy9975]

If elasticsearch distributes the data between the shards of an index such that each lucene store roughly holds the same number of documents, when you run a search query, elasticsearch, despite the inter-node communication, only knows which shards hold that index and not which particular shard will have that document? So it has to run the query against all the shards and merge results, it cannot just search the one shard that contains that document? It does not know beforehand based on how documents are distributed among shards.

[@akshaychawla7413]

I am not able to enroll for your course, tried with 2 different emails, please have a look into this.

[@GeorgeBridgemanData]

Sorry for the very late response. I've had feedback from a couple of people using Firefox, who worked around it by using a different browser. I'm not sure if that's your issue but thought I'd mention it. Let me know if you're still having issues and I'll try responding quicker this time!

[@carlosroberto366]

Isn't the cluster the server (i.e. AWS EC2 instance) itself? To my mind, a node is not a server because you can create several nodes in the same machine. I was expecting to see MyCluster1 and MyCluster2 each having a single node, hence, high availability via cross-cluster communication. 11:11 node = server in his example 15:05 node = process

[@GeorgeBridgemanData]

A node is an Elasticsearch process running on a host. You're right that you can run multiple nodes on the same host (even not containerised), but it's not recommended and it's widely accepted that you only run a single node on a host. If you did run two nodes on a single host, you could have either one or two clusters on that host. The node is configured with the cluster name it's expected to join, so you could configure each node with a different cluster name and have two clusters on that host! Cluster formation can get quite involved. There are configuration settings that need to be applied specifically at the formation stage. I can do a video on how that works at some stage.

[@Milostrosic]

Very clear explanation!

[@sv_n]

1000th like 😅