Apache Spark / PySpark Tutorial: Basics In 15 Mins

[@mongooon6931]

We use spark for our data pipeline at work -- we have tables with 10+ billion records, and our applications end up moving trillions upon trillions of records of data per month. Unfathomable numbers that spark is capable of. Great video!

[@GregHogg]

Yeah, it's insane! Thanks so much.

[@EclipsyChannel]

that's the power of distributed systems and parallel computing... computer science is beautiful

[@Nedwin]

I'm a freelance data scientist and I really thankful to find this video, Gregg. Can't expect more! Thank you so much. Good luck with everything. 🙏

[@GregHogg]

That's awesome best of luck in that! And you're very welcome it's my pleasure 😊

[@AnVinhNguyen]

Your explanation is clear and the examples are practical and useful for beginners. Thanks a lot and keep it up!

[@GregHogg]

I really appreciate this. You're very welcome 😃

[@joshuabradshaw1647]

Thank you for sharing to the world. I'm currently a supply chain analyst and aspiring supply chain data scientist 🙏

[@GregHogg]

That's excellent to hear and very exciting Joshua! I wish you the best of luck 🥰

[@ashleyb5849]

Awesome video. I love using spark at work

[@andersborum9267]

I'm just getting into DataBricks and PySpark and this introductory tutorial was a great starter.

[@GregHogg]

Awesome! Hope that goes well :)

[@mtamjidhossain]

You are awesome. Just delivering the right videos. Subscribed a few days back already but hit notifications on for you rn. Cause I wanna watch all your videos

[@GregHogg]

Well that's really great to hear! Thanks so much Tamzid!

[@yashmodi6762]

Which big data tools one must learn for beginners and from where to learn( please provide some resources)

[@GregHogg]

Of course I'd recommend my channel - SQL and Spark are the most important ones in my opinion :)

[@260056]

@greg, plz share the link of 1 hr video.. I am unable to find it

[@dominicaleung7329]

Greg, thank you so much. I am new to PySpark, and your video is very good in explanation and you did those simple example and I am able to follow you and write in my own Python Notebook to try it out. Will watch your DataFrame basics video next.

[@GregHogg]

Amazing! Sorry for the late reply

[@ericcarmichael3322]

Thanks for sharing, appreciate the quick run down on this stuff

[@GregHogg]

Glad to hear it!

[@victorroy525]

Just the type of samples we need to begin with. Meaningful content. thnx.

[@GregHogg]

Glad you enjoyed it!

[@SpecialGreg66]

Hi Greg! Great video, do you have one that explains how you convert spark to dfs and vice versa? We pull millions of rows from csvs and looking to do transformations before dropping into a db. Also, how does the distributed computing work on a singular computer? Just distributes it across the cpu cores?

[@EclipsyChannel]

you are a great teacher... keep doing what you do my man

[@BHANUCHAUDHARY-eb4ul]

Thanks Greg for the wonderful explanation !!

[@parvathirajan.n]

No words man! Simply loved it. Appreciate your efforts.

[@GregHogg]

Really glad to hear that! Thank you 😊

[@noorhake9087]

Hi , I'd like to ask you a question I'm working on a project that is how linear regression selected feature by apache spark when I want to execute the code for pyspark it gives an error that pyspark dont define and I tried to figure it out in many ways it didn't solve that problem💔

[@hypebeastuchiha9229]

What was your degree in Computer Science or a Data Science course? I'm in my third year for a Computer Science BSc and I feel like I'm at a disadvantage for Data Science. We didn't learn statistics or have a lot of math modules. Most Data Science jobs require a Masters or PhD but I don't want to get a Masters straight after uni so I'm looking at Data Engineering since they accept BSc's. Is that a realistic path into Data Science or am I wasting my time?

[@GregHogg]

I'm a statistics major. I don't think you're at a disadvantage, people very widely respect computer science majors. If anything I'd feel I'm at a disadvantage lol. But agreed, you get less stats courses. I would think some certificates and projects would be enough without needing a masters, unless you're aiming for FAANG or the other top jobs

[@GregHogg]

This video may help; kzbin.info/www/bejne/ZmmqXqhvfbNrgcU

[@antarcticadventure]

Never used Spark before. Thank you.

[@GregHogg]

Me too for the longest time; PySpark is a life changer though!

[@lakshaydulani]

now thats what i was looking for

[@tarunodaysarma9741]

Greg ,had a question on pyspark...how do I find latest parquet files stored in hdfc path using pyspark code

[@GregHogg]

Sorry I don't know! 🤔

[@ganeshkaushik2290]

Hi Bro, could you please make a video on learning process on bigdata?and what job roles which big data skills i'm really confsed where to start and what to learn! I know python, sql I learned some basics of hdfs, hive, sqoop now i'm trying to learn pyspark

[@GregHogg]

Thanks for the feedback, I'll keep this in mind!

[@hsoley]

You are awesome, thanks for sharing your knowledge with the world

[@GregHogg]

I really appreciate that Hamid!!!

[@andrewhancock2451]

This is an awesome video. I wonder, however, whether you could explain why the end results shows numbers with 12 characters. Didn't you set of numbers only go up to a million, which has 6 digits? You also referred to your hour-long PySpark course. Would you be able to link to it in the show notes, please? Thanks!

[@yizharalmagor8539]

At a certain point he squared all the numbers in the RDD and then kept using the squares from then on.

[@Vlapstone]

sc command is not working on my Colab as it's working on this vide... can anyone help?

[@aeigreen]

Explained so well. 5 stars. Love to see more videos..

[@GregHogg]

Really glad to hear it thanks so much!

[@samusaran1692]

Very good examples. Thanks man :)

[@GregHogg]

Glad it helped!

[@keerthanamurugesan-xe6mr]

It look like using numpy, pandas what is the difference between this and pyspark.

[@GregHogg]

It looks very similar to us coders, which is great. But pandas and numpy are mainly for dealing with data on the computer you're using. Spark allows us to distribute our workloads across a cluster of machines

[@keerthanamurugesan-xe6mr]

@@GregHogg Thankyou

[@mehmetkaya4330]

Concise and very well explained! Thank you so much!!

[@GregHogg]

Thank you and you're very welcome!

[@aparfeno]

Thank you for great video and for useful education links!

[@GregHogg]

You're super welcome 😃

[@MrChilo89]

Hello and thanks for this video, I ve been trying to follow and to your average way, but i receive an error : avg = nyt.map(lambda x: (x.title, int(x.rank[0]))) grouped = avg.groupByKey() grouped = grouped.map(lambda x:(x[0], list(x[1]))) averaged = grouped.map(lambda x: (x[0], sum(x[1]) / len(x[1]) )) averaged.collect() 'TypeError: Invalid argument, not a string or column: [1, 3, 7, 8, 12, 14, 20] of type . For column literals, use 'lit', 'array', 'struct' or 'create_map' function.'

[@manpritsingh3972]

This video is really helpful. Thanks a lot Gregg.

[@GregHogg]

You're super welcome!

[@geethsn1866]

Thanks for the tutorial. It was simple and easy to follow. However, when I tried the code in Colab, just by typing "sc" is not invoking spark. Is there any prerequisites - to be installed in Colab before "sc" ?

[@GregHogg]

Please check out my notebook. You'll need to pip install PySpark, and write a line or two of code to set it up

[@geethsn1866]

@@GregHogg Thank you Greg.

[@boudhayanism]

Cool video, thanks for making it

[@GuilhermeMendesG]

What an amazing content you're putting here man... thanks for everything!

[@GregHogg]

Thanks so much for the kind words. You're very welcome 🤠

[@r3d_robot594]

Good PySpark Primer! Others are either too lengthy or short and vague.

[@GregHogg]

Thanks so much I'm really glad to hear that! :)

[@krishj8011]

very fine details covered. really useful and easy to understand the spark concepts.

[@GregHogg]

Really glad to hear that.

[@demohub]

Great overview. Thanks

[@hyeonjukwon3638]

Very useful and interesting! Subscribed :)

[@GregHogg]

Glad to hear it, thanks a ton!

[@maxitube30]

and what are the machine on what we parallelize the work? They have to be configurated? i mean,if pyspark or spark parrallelize on a cluster,we have to configue the cluster too?

[@GregHogg]

Someone has to configure it. Probably won't be your job though. You'll just select it, kinda like a Python virtual environment, and act as if it's the same as in this video because nothing changes from the programming point of view :)

[@maxitube30]

@@GregHogg understood. Thx :)

[@rohanjoseph1531]

Hi @Greg Hogg, I can't seem to access the "sc" object on Google Colab. Which library let's you use that object?

[@GregHogg]

github.com/gahogg/KZbin/blob/master/PySpark%20In%2015%20Minutes.ipynb

[@rohanjoseph1531]

@@GregHogg cheers!

[@capper3360]

Can you share the link to the hour long tutorial you mentioned at the end, couldn't find it in your spark playlist.

[@GregHogg]

Here you go: kzbin.info/www/bejne/bqrTeoWma6mDm9k

[@JaylaScousa]

Concise and well presented 👍

[@GregHogg]

Very glad you found it useful, James!!

[@nataliaresende1121]

Hi Greg, how can I convert .csv files into .txt files (with comma as delimiter) using pyspark? Do you have a code snippet?

[@GregHogg]

I think you can just change the extension from CSV to txt

[@高智衡]

great great content! BTW, please give us the link of the an-hour-long spark tutorial mentioned in the end,thanks a lot.

[@GregHogg]

Thanks! Here you go: kzbin.info/www/bejne/bqrTeoWma6mDm9k

[@PranitKothari]

Nicely explained.

[@charlescoult]

Took a minute to get going but well done

[@clintp3504]

Great stuff! Thanks

[@GregHogg]

You're very welcome ☺️

[@ajanieniola9172]

Can you also use apply instead of map

[@GregHogg]

Probably

[@caiocalo1]

such a good tutorial

[@abdullahsiddique7787]

How is future of spark is flink replacing it ? Is it worth learning for career in big data ?

[@GregHogg]

I don't know what flink is.

[@abdullahsiddique7787]

@@GregHogg thanks for reply gregg can u pls also tell me the career scope of Apache spark for future

[@GregHogg]

@@abdullahsiddique7787 Spark is and will stay essential for Data science, ML, analysts and big data for a long time.

[@abdullahsiddique7787]

@@GregHogg thanks gregg appreciate your quick response

[@GregHogg]

@@abdullahsiddique7787 Of course!

[@AlexFosterAI]

appreciate this vid. thanks man

[@chanta2809]

What is the URL to practice? How to setup data for practicing?

[@GregHogg]

Thank you! You made me notice I accidentally removed the notebook from the video description. You can grab the notebook code in the video description now. You can actually get PySpark in google colab very easily, with simply !pip install pyspark and then import pyspark, then continue following the steps in this video.

[@sohamsonone5440]

Hi Greg, which one good among data science, data analytics or machine learning, AI.. could you pls give a suggestion

[@GregHogg]

Data science / ML

[@nataliaresende1121]

very good, thanks!

[@GregHogg]

You're very welcome Natalia!

[@grantholomeu3725]

I don't understand why in tutorials like this I often get errors saying, "module x has no attribute 'y.'" In this case, I can't get Python to recognize parallelize.

[@GregHogg]

Not sure sorry!

[@pardonmasuka2]

Awesome starter!

[@ahmadsaad1888]

You mentioned an hour long spark video, I can't find it.

[@GregHogg]

kzbin.info/www/bejne/bqrTeoWma6mDm9k

[@agnelamodia]

@@GregHogg Could you please paste this link in the description?

[@GregHogg]

@@agnelamodia please see above

[@cetilly]

Sensational!

[@GregHogg]

Thank you 😊😊😊

[@RossittoS]

Great!

[@GregHogg]

Thank you!

[@javidhesenov7611]

nice explanation

[@GregHogg]

Thanks a bunch Javid! :)

[@sndselecta]

I thought performance issue between scala and py isnt an issue anymore.

[@GregHogg]

I personally doubt it. I'm not an expert on this one, but I'd be pretty surprised if python wasn't significantly slower than scala. Of course, if we're talking practically- they're both very fast, but in computational time, I would suspect python is much slower. Thanks!

[@GregHogg]

You are correct, and I am incorrect! Thank you for updating me!

[@sndselecta]

I think we are both correct. I've been reading up on it, with regards to refreshing my scala or keep chugging away with pyspark. Bottom line: it's good to know both. It depends on the use cases. But in general Scala will perform better monotonically. However what Ive read is: it isn't always about one way gains based solely upon performance or more importantly "one" sole factor, there are pros and cons and sometimes the cumulative gains can weigh either way. For example pythons rich ecosystem can weigh in for achieving a faster result trying to do the same thing with Scala. Another interesting discussion you should start is Koalas. I wrote a blog, trying to get people to weigh in. forums.databricks.com/questions/65646/thoughts-on-if-its-worth-it-to-work-in-koalas.html

[@GregHogg]

@@sndselecta Sorry I missed this! Absolutely and thank you for the great reply.

[@jimbocho660]

The Spark people themselves are advising against learning Scala for only marginal gains over pySpark.

[@emirhanbilgic2475]

thanks mate

[@GregHogg]

Very welcome!

[@e.s298]

Good for learn RDD

[@paraklesis2253]

Thank you

[@GregHogg]

You're very welcome!

[@Somethingaweful]

back up from the camera my dude. I feel like your staring directly at my soul

[@GregHogg]

Maybe I am

[@Chris-qg6kc]

​@@GregHoggget em bro.

[@dw61w]

how is this more useful than numpy?

[@GregHogg]

NumPy works on one computer. Spark works on as many as you want

[@dw61w]

@@GregHogg thanks!

[@pinkomoore]

Pyspark seems to be pandas on steroids + distributed resources usage

[@pauweldalmeidaayivi5310]

Great!

[@smash4929]

Hey Greg, The knowledge in the video is great but the background music is distracting.

[@MrPeacefulsoul2610]

A detailed video probably would be more helpful.

[@marjorielarkins5178]

Johnson Jason Miller Jason Martinez George

[@vladx3539]

great video… but please step away from the camera sir

[@GregHogg]

Ouch

[@vladx3539]

@@GregHogg just kidding with you! great content

[@TeresaGonzalez-r2q]

Thomas David Young Shirley Rodriguez Scott

[@teddymisderrick7034]

Smith Jennifer Brown Frank Lewis William

[@yelgabs]

So you’re just gonna teach us the wrong way of doing things then leave us on a cliff hanger? 😅

[@WoolleyStanford]

Hall Angela Taylor Scott Gonzalez Daniel

[@YoungMerle-f6v]

Johnson Jennifer Miller Richard Anderson Betty

[@BostonSummit-v1l]

Young Richard Thomas Melissa Robinson Ronald

[@MikeKing-c5k]

Young Karen Miller Christopher Johnson Thomas

[@RoyanaHaque]

Garcia Angela Hall Jeffrey Moore Larry

[@ranvijaymehta]

Thanks Sir

[@MUSKAN0896]

this was an amazing and clear video! thanks so much!

[@GregHogg]

Very glad to hear that!!

[@AntoinetteFanny-l8s]

Williams Matthew Brown Jason Young Michelle