thank you

Great content!!!!

wow great content thanks a lot

What about aggregateByKey function in RDD

To whom may be concerned when to use GroupByKey over ReduceByKey: groupByKey() can be used for non-associative operations, where the order of application of the operation matters. For example, if we want to calculate the median of a set of values for each key, we cannot use reduceByKey(), since median is not an associative operation.

Hindu again 🤢

Hi, I like your Spark videos. Please create a dedicated video for top 100 most frequently used Spark Commands. - Pankaj C

Video recommendatin at the end are blocking the content...

i see, for RDD its memory and for Dataframe it is mem + disk

why are you talking like sleppy mode ??

Please provide aws questions and answers. Thank you 🙏

what is MSCK ?

What if I have multiple spark jobs in parallel in on spark session

For ORC format, schema evolution is not just limited to adding new columns. Backward Compatibility: Adding Columns: New columns can be added to the schema without affecting existing data files. When reading old ORC files with a new schema that includes additional columns, the new columns will be treated as optional and filled with default values. Removing Columns: Similar to Parquet, existing columns can be removed without breaking compatibility. When reading old ORC files with a new schema that excludes certain columns, those columns will be ignored. Changing Data Types: Data types of existing columns can be changed, and ORC will attempt to convert the data to the new type. However, similar to Parquet, this conversion might result in data loss if the types are not compatible. Forward Compatibility: Adding Columns: New columns can be added, and existing files can still be read without errors. The new columns will be filled with default values when data from the old files is read. Removing Columns: Files written with a schema that has fewer columns can still be read with a newer schema containing additional columns. The additional columns will be treated as optional. Changing Data Types: Forward compatibility is generally maintained for changing data types, but careful consideration is needed to avoid potential data loss or conversion issues. above points are what I found supplementing with your content. thanks for your videos and dedication in making them, it is really helpful for my preparation.

Hi! Why you say Avro is row oriented, isn't also columnar storage?

Thank you

Super content thank you

Good sir

Could you please tell what is the difference between partition pruning and predicate pushdown

Very Nice and clear explanation before this video i was very confused regarding executor tuning part now after this video it is now crystal clear.

Hi, 10 nodes means including the master node? i have a configuration like this: "Instances": { "InstanceGroups": [ { "Name": "Master nodes", "Market": "SPOT", "InstanceRole": "MASTER", "InstanceType": "m5.4xlarge", "InstanceCount": 1 }, { "Name": "Worker nodes", "Market": "SPOT", "InstanceRole": "CORE", "InstanceType": "m5.4xlarge", "InstanceCount": 9 } ], "KeepJobFlowAliveWhenNoSteps": false, "TerminationProtected": false },

@TechWithViresh: no recent videos. Can you please add . your videos are very useful brother. thanks

Thanks! A great and concise explanation!

The 2nd map will not executed as no action performed on result data set after collect.

hello, i find the content very interesting especially on when the hash join is better than the sort merge join. could you please tell me where you found the documentation on that?

Many thanks to you sir. 😊 i learnt spark from you

very good. please make videos as interview questions on spark as a group of videos

nice

Audio quality is not good content is good

Limit comes after order by in query execution order, how using limit will reduce the number of records to be sorted? Am I missing anything here?

Why are you converting dataframe to rdd ?? It is very bad practice in terms of performance

video from 11:30, we are adding random key to exiting towerid key for Example. tower id: 101 and salt key : 67 then 101+67= 168 hash value of the 168 would be a final value right. what in case of partition column is string datatype. ??

bhai ye hindi m bta dega toh tera kuch chla jaa rha h kya??

Perfect 👌 explanation

Very good and descriptive comparison. Thank you!

You gave the all information about Hive.. is this enough for interview?

How the last map operation will run on driver see till collect a job will be completed and whenever we call another action it will create new job with new Dag which will again distributed and run on executors??

Good explanation.. Thank you 👍

can we get ppt that you show in the videos?

What If each node has only 8cores?? How does spark allocate 5cores per jvm ?

Awesome✨

bro if you have 6 blocks in Hadoop 3 then it consumes 15 blocks. Suppose we have a file which consists of 2 Blocks (B1 and B2). 1) With current HDFS setup, we will have total (2×3 = 6 blocks in total). For Block B1 -> B1.1, B1.2, B1.3 For Block B2 -> B2.1, B2.2, B2.3 2) With EC setup, we will have total (2×2 + 2/2 = 5 blocks in total). For Block B1 -> B1.1, B1.2 For Block B2 -> B2.1, B2.2 The 3rd Copy of each Block will be Xor’ed together and stored as a single Parity Block as (B1.1 xor B2.1) -> Bp In this setup: If B1.1 is corrupted, we can recompute B1.1 = Bp xor B2.1 If B2.1 is corrupted, we can recompute B2.1 = Bp xor B1.1 If both B1.1 and B2.1 are corrupted, then we have another copy of both the blocks (B1.2 and B2.2) If parity Block Bp is corrupted, then it is again recomputed as B1.1 xor B2.1

@Ankit Bansal can you please solve this using SQL please

Is there any differences with performance issues?

Crisp , concise and to the point explanation in great detail. Anyone can understand through this video. Extremely well done. Kudos...

Thank you

Thank you

Good content

Voice and explanation not clear!