Orlando Karam - Introduction to Spark with python - PyCon 2015

"Speaker: Orlando Karam
In this tutorial we will cover the basics of writing spark programs in python (initially from the pyspark shell, later with independent applications). We will also discuss some of the theory behind spark, and some performance considerations when using spark in a cluster.
Slides can be found at: speakerdeck.co... and github.com/PyC..."

[@mappilakty]

Excellent tutorial. Very well explained and demonstrated.

[@mpgrewal00]

Great instructor.. great content

[@bobkay9278]

I was following well until 1:24:50, --py -files!! Where does this code even suppose to go? It just didn't do any good. I'm sure there is more into it; couldn't follow after that!

[@sidhusam]

you need to do following (Canopy 64bit) C:\SparkCourse\spark-pycon15-master\code\simple> C:\spark\bin\pyspark --py-files person.py one spark shell is invoked then import person

[@rohitdhankar360]

Excellent many thanks :)

[@shawnz9833]

any idea on, why there is a "," not a "." in front of func??????????????????? sales.groupBy('day').agg(funcs.min('store').alias('minStore'),funcs.max('quantity').alias('MaxQty')).show()

[@jjwei4578]

Aggregating those two functions. Two functions performed separately

[@shawnz9833]

Any idea? I am using spark2.0.1 pyspark-shell File "", line 1, in AttributeError: 'SQLContext' object has no attribute 'jsonFile'

[@shawnz9833]

work around by SQLContext.read.json() ## pls let me know if you have any ideas but you have a warning: WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0 16/11/11 12:47:59 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException

[@hernandezurbina]

from pyspark.sql import SQLContext people = sqlCtx.read.json("people.json") worked for me

[@shawnz9833]

Victor Hernandez-Urbina cool thank you man which version of spark do you use?

[@hernandezurbina]

spark 2.1.0 on python 3

[@saurabhmehta5643]

from where we can peolpe.txt data? In data Folder i don't have peolple.txt file, I am using different pyspark version1.6

[@orlandokus]

Git repo is at github.com/okaram/spark-pycon15 People.txt is at github.com/okaram/spark-pycon15/blob/master/data/people.txt If you already downloaded it, you may be mistyping it ; you have an extra l

[@orlandokus]

Git repo is at github.com/okaram/spark-pycon15 People.txt is at github.com/okaram/spark-pycon15/blob/master/data/people.txt If you already downloaded it, you may be mistyping it ; you have an extra l

[@orlandokus]

Code and data for *this tutorial* are at github.com/okaram/spark-pycon15 ... the data is not standard spark; just my examples :)

[@ahmadmaroof2809]

Hello, How can I run my python codes from .py file. I don't want to use shell. Thanks

[@joelcastellon9129]

+Ahmad Maroof It depends on your editor. I use vim. If you install pymode (super simple, just google it) it is just r

[@mgamboacavazos]

Hi Orlando, would it be possible to get a copy of your slide? Thanks!

[@arunnadda8072]

Hi Mario, you can get these @ onedrive.live.com/redir?resid=84334a138ac1cce0%2114907

[@anirudhparmar9124]

Is there any tutorial for simple setting up in linux machine.

[@shawnz9833]

1) set java jvm 2) download spark 3) tar xvf it 4) export your path Then, enjoy

[@orlandokus]

Not a tutorial, but I have the script I used at github.com/okaram/spark-pycon15/blob/master/scripts/install_ubuntu.sh Will probably want to change version number :)

[@shawnz9833]

nice talk and like the Persian accent. Is Orlando a real Persian name

[@AbeOnline66]

it is not persian actually.

[@shawnz9833]

The accent or the Name

[@AbeOnline66]

Neither of them are Persian. I am Iranian so I know :D

[@shawnz9833]

Cool, thank you. I had a professor who was from Iran and like his course a lot.

[@orlandokus]

Thanks for the compliment; the accent is Mexican, although the last name is middle eastern (that part of my ancestry is from Syria). Orlando is not uncommon in Spanish speaking countries