databricks cluster configurations Part 1

this video gives basic information related to databricks cluster configurations.

[@sankarkumarazad3843]

Great Explaination. How do we decide which worker and driver type is to be selected. And how many instances of workers are to be used. Are there any set of rules or calculations to decide??

[@KnowledgeSharingjkb]

It should be based on the work load. Normally we will not do any work on the driver unless the user using data science codes using pandas. If you add multiple nodes, then your parallelism increase. Again please note that if the high volume data processing required from the beginning, then you can add more capacity to the nodes. It requires separate session to explain. Let me add video