Dynamic Partition Pruning in Apache Spark

Spark Programming and Azure Databricks ILT Master Class by Prashant Kumar Pandey - Fill out the google form for Course inquiry.
forms.gle/Nxk8dQUPq4o4XsA47
-------------------------------------------------------------------
Data Engineering using is one of the highest-paid jobs of today.
It is going to remain in the top IT skills forever.
Are you in database development, data warehousing, ETL tools, data analysis, SQL, PL/QL development?
I have a well-crafted success path for you.
I will help you get prepared for the data engineer and solution architect role depending on your profile and experience.
We created a course that takes you deep into core data engineering technology and masters it.
If you are a working professional:
1. Aspiring to become a data engineer.
2. Change your career to data engineering.
3. Grow your data engineering career.
4. Get Databricks Spark Certification.
5. Crack the Spark Data Engineering interviews.
ScholarNest is offering a one-stop integrated Learning Path.
The course is open for registration.
The course delivers an example-driven approach and project-based learning.
You will be practicing the skills using MCQ, Coding Exercises, and Capstone Projects.
The course comes with the following integrated services.
1. Technical support and Doubt Clarification
2. Live Project Discussion
3. Resume Building
4. Interview Preparation
5. Mock Interviews
Course Duration: 6 Months
Course Prerequisite: Programming and SQL Knowledge
Target Audience: Working Professionals
Batch start: Registration Started
Fill out the below form for more details and course inquiries.
forms.gle/Nxk8dQUPq4o4XsA47
--------------------------------------------------------------------------
Learn more at www.scholarnest.com/
Best place to learn Data engineering, Bigdata, Apache Spark, Databricks, Apache Kafka, Confluent Cloud, AWS Cloud Computing, Azure Cloud, Google Cloud - Self-paced, Instructor-led, Certification courses, and practice tests.
========================================================
SPARK COURSES
-----------------------------
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/s...
www.scholarnest.com/courses/d...
KAFKA COURSES
--------------------------------
www.scholarnest.com/courses/a...
www.scholarnest.com/courses/k...
www.scholarnest.com/courses/s...
AWS CLOUD
------------------------
www.scholarnest.com/courses/a...
www.scholarnest.com/courses/a...
PYTHON
------------------
www.scholarnest.com/courses/p...
========================================
We are also available on the Udemy Platform
Check out the below link for our Courses on Udemy
www.learningjournal.guru/cour...
=======================================
You can also find us on Oreilly Learning
www.oreilly.com/library/view/...
www.oreilly.com/videos/apache...
www.oreilly.com/videos/kafka-...
www.oreilly.com/videos/spark-...
www.oreilly.com/videos/spark-...
www.oreilly.com/videos/apache...
www.oreilly.com/videos/real-t...
www.oreilly.com/videos/real-t...
=========================================
Follow us on Social Media
/ scholarnest
/ scholarnesttechnologies
/ scholarnest
/ scholarnest
github.com/ScholarNest
github.com/learningJournal/
========================================

[@anikethdeshpande8336]

super explanation! simple to understand, thanks for showing the execution plans!

[@EugenePetrash]

Genious explanation. Not only on that certain topic, but all of the author's videos and articles are also totally clear. Thanks a lot. Subscribed!

[@ylchen5975]

Very useful and expiation is pretty clear, thank you!

[@hierfnhg]

Very informative thanks for deep diving.

[@skywalker66ful]

Best Explanation I have found till date about Dynamic Partition Pruning and infact about Predicate Pushdown and Partition Pruning as well

[@andre__luiz__]

Amazing explanation!!!

[@octo3010]

Neat feature

[@akshaygupta013]

Nice explanation. I do have a doubts what's the need for broadcast if the filter condition is already being applied to dimensions table and if it is required than tables which are greater than broadcast threshold in those case will this technique not work or just join type will be different.

[@feelings__flicks]

Same doubt brother. If u get the answer can u please share it.

[@mohammedsafiahmed1639]

from what I understand, filter condition, or predicate pushdown as Databricks calls it, works only when querying single table. When you join two tables, you need to 'broadcast' the filter to the other table being joined.

[@artemvolkov5682]

What if I just add year and month to the 'ON' statement ? I believe partition pruning will work, but AQE should be enabled.

[@babyscookbook2751]

Hi sir., can we use apache kafka for sending emails? Please sir, I need it help

[@bhomiktakhar8226]

Nicely explained !....but how does the filter is transferred to order table..since where condition is on year and month , query on fact table would still have to figure out what is full_date (of dimension table)column values on for 2021 Feb...could be multiple full_dates for month,year .?

[@kolketzz]

probably it would do date like '2021-02%'

[@lancequin5209]

Him: Make Sense? Me: Nope Him: Great

[@artemvolkov5682]

hahah, feel exactly the same

[@trainsam22]

Hi Prashant, you know your concepts. but stop saying : makes sense.. or simple etc.. that is too uncle like.. b cool