Spark Interview Question | Scenario Based Questions | { Regexp_replace } | Using PySpark

[@gayathrilakshmi6087]

Can you please attach data set and solution set ..so that we can practice...thanks for all the excellent videos

[@deepakkini3835]

This was thoroughly explained, nice scenario . Could you please do some videos on weekly cohort analysis using window functions in spark?

[@AzarudeenShahul]

Sure, we will plan.. thanks for your support

[@sensibleandhrite]

@@AzarudeenShahul can you explain expressions used in regex replace? i didnt understand whats $0.

[@arnabbangal766]

Sir, can you explain the regex expression more clearly or provide any youtube link where regex is explained nicely ? Thanks. Your videos are very helpful.

[@sumantaghosh9299]

Nice explanation..nd gd questions too

[@murrthuzaalibaig1205]

can u expalin about regex in detail and how did u get the expression

[@SurendraKapkoti]

Great explanation.. Keep it 👆

[@sravankumar1767]

superb explanation, do more videos bro........

[@riyazalimohammad633]

Hello Azar! Amazing Video. Is there a way we could replace the 5th pipe occurrence rather than adding "-". I want to replace the pipe with "-".

[@anuragvinit]

Thanks for this . This has been asked to be during Mindtree interview

[@AzarudeenShahul]

Cool.. hope you were able to answer the question and crack interview...

[@bunnyvlogs7647]

Great brother...God bless u

[@pavanp7242]

Good work bro

[@dipakchavan4659]

Hey Azharuddin, Superb 👍🏻. Can u plz provide dataset.

[@jk_gameplay9195]

Awesome bro

[@umakanthtagore6003]

Thanks for this information, can you please help me that Is this aproch works large data as well ? Thanks in advance !!

[@AzarudeenShahul]

Yes, this approach can be scale out to large dataset. Let me know if you face any problem

[@PraveenKumar-xk5zv]

Very helpful.. thank you!

[@sangramrajpujari3829]

good video to improve our logic.

[@shiyamprasath3105]

hii bro....i have seen this 5 th occasion change in Scala but code is too difficult compared with pyspark... please share easy step of code for scala bro

[@aN0nyMas]

In this the last record in my test data had just four columns. hence I got a schema error. is there a way to specify to handle this by ignoring the malformed data?

[@sudarshanthota4444]

Thank you very much for your video's

[@AzarudeenShahul]

Thanks for you support:)

[@bikersview9926]

@@AzarudeenShahul please txt file and code snippets

[@sumitrastogi1]

Can you please share how to deploy in production environment for pyspark job.your videos very helpful

[@arifulahsan8803]

Hi do you teach spark course?

[@jittendrakumar3908]

Please upload the video for ingesting the data from sap server. This is very important as we need to ingest the data from different source via pyspark.

[@maheshk1678]

Thanks for the nice video

[@ankitapriya6671]

Can you share the post where you have provided answer for this

[@maheshk1678]

Could you explain the same with kafka message streaming

[@AzarudeenShahul]

Sure

[@jittendrakumar3908]

Also please upload for reading the occurance of an string in an word.

[@ihba02_official]

Thank you so much bro

[@purnimabharti2306]

I didn't understand at some places you convert rdd to df and then df to rdd...why is it so?

[@khushbusalunkhe677]

Some transformations are not allowed in dataframe but they are availble in RDD so to perform those operations it was converted into RDD and then back to DF

[@VinodR-vx8uh]

Please someone explain that regexp pattern(.*?\\){5} and why $0 in $0-

[@duskbbd]

Why you kept $0 before delimiter -

[@riyazalimohammad633]

The $0 in awk syntax means to return the output, so when azar uses "$0-" in the function, it will preserve the output of the regex and add "-" to it.

[@Azardeen-sb1wr]

Mohammed,Azar,BE-4year Prakesh,Kummar,Btech-3year Ram,Kumar,Mtech,3year jhon,smith,BE,2year # any one can share the pyspark code to delimit the "-"

[@shyammtv.v.s.p4262]

123#Australia,india,Pakistan 456#England,France 789#canada,USA output is 123#Australia 789#canada 456#England 456#France 123#india 123#Pakistan how to solve this using pyspark or scala