06. Databricks | Pyspark| Spark Reader: Read CSV File

#ReadCSV, #DatabricksCSVFile, #DataframeCSV
#Databricks, #DatabricksTutorial, #AzureDatabricks
#Databricks
#Pyspark
#Spark
#AzureDatabricks
#AzureADF
#Databricks #LearnPyspark #LearnDataBRicks #DataBricksTutorial
databricks spark tutorial
databricks tutorial
databricks azure
databricks notebook tutorial
databricks delta lake
databricks azure tutorial,
Databricks Tutorial for beginners,
azure Databricks tutorial
databricks tutorial,
databricks community edition,
databricks community edition cluster creation,
databricks community edition tutorial
databricks community edition pyspark
databricks community edition cluster
databricks pyspark tutorial
databricks community edition tutorial
databricks spark certification
databricks cli
databricks tutorial for beginners
databricks interview questions
databricks azure,

[@sowjanyagvs7780]

am trying to grab an opportunity on data bricks, glad i found your channel. Your explanations are far better than these trainings

[@rajasdataengineering7585]

Welcome aboard! Thank you

[@gulsahtanay2341]

Explanations couldn't be better! I'm very happy that I found your work. Thank you Raja!

[@rajasdataengineering7585]

Hope it helps you learn the concepts! Thanks

[@abhinavsingh1173]

Your course it best. But problem with you course is that you are not attching the github link for your sample data and code. Irequest you as your audience please do this. Thanks

[@shivayogihiremath4785]

Superb! Concise content, properly explained! Thank you very much for sharing your knowledge! Please keep up the good work!

[@omprakashreddy4230]

what an explanation sir ji !!! Please continue making videos on adb. Thanks a lot !!

[@rajasdataengineering7585]

Thanks Omprakash. Sure, will post more videos

[@patiltushar_]

Sir, you way of teaching is fabulous. Earlier i learnt spark, but your teaching is better than that.

[@rajasdataengineering7585]

Thanks and welcome! Glad to hear that

[@kketanbhaalerao]

Very Good Explanation!! really great >> Can anyone please share those csv file/ link. Thanks in advance.

[@prasenjit476]

Your videos are lifesavers .. !!

[@rajasdataengineering7585]

Thank you

[@nurullahsirca8819]

thank you for your great explanation. I love it. How can I reach the data and code snippets? where do you share them?

[@deepanshuaggarwal7042]

Can you please explain in the video why these many job and stages are created. To understand internal working of spark is very necessary for optimisation purpose

[@AtilNL]

To the point explanation. Thank you sir! Have you tried to import using sql from a sharepoint location?

[@rajasdataengineering7585]

No, I haven't tried from SharePoint

[@VinodKumar-lg3bu]

Neat explanation to the point ,thanks for sharing

[@rajasdataengineering7585]

Glad it was helpful! You are welcome

[@Jaipreksha]

Excellent explanation. ❤❤

[@rajasdataengineering7585]

Glad it was helpful!

[@lalitsalunkhe9422]

Where can I find the datasets used in this demo? is there any github repo you can share?

[@shahabshirazi6441]

Thank you very much! very helpful!

[@rajasdataengineering7585]

Thanks for your comment

[@battulasuresh9306]

Raja Sir, hope these videos all are in series

[@rajasdataengineering7585]

Yes all videos are in series

[@sravankumar1767]

Nice explanation bro.. simply superb

[@upendrakuraku605]

Hi bro , it was nice explanation..👍 Can you please help on below points points to cover in Demo : how to read CSV, TSV, Parquet, Json, Avro file formats, how to write back, how you can add unit tests to check transformation steps outputs, how to read DAG, how to work with Delta tables, how to create clusters

[@rajasdataengineering7585]

Sure Upendra, I shall cover all these topics

[@upendrakuraku605]

@@rajasdataengineering7585 day after tomorrow I have to give demo on this can you please solve this as soon as possible 🙏

[@MrTejasreddy]

Hi raja really enjoyed u r content information is very clear and clean explanation...one of my frd refered u r channel..really nice...but i noticed that pyspark playlist some off the videos are missed...if possible pls check on it..thanks in advance.

[@rajasdataengineering7585]

Hi Tejas, thank you. Few videos are related to azure synapse analytics. So they might not be part of Pyspark playlist

[@user-kp5sl6se7l]

can you able send those Csv files. i will try in my system.

[@ramsrihari1710]

Hi Raja, Nice video.. quick questions.. What if I want to override the existing schema? Also if we add schema in the notebook, will it not be created over and over whenever the notebook is executed? Is there a way to have it executed one time?

[@suman3316]

please upload the github link of these files also..

[@ravisamal3533]

nice explanation!!!!!!!!!

[@rajasdataengineering7585]

Glad you liked it!

[@battulasuresh9306]

Please acknowledge This will help to lot of people All videos are in series or not?

[@rajasdataengineering7585]

Yes all videos are in series

[@sumitchandwani9970]

Awesome

[@rajasdataengineering7585]

Thanks!

[@pcchadra]

when I am runing schema_alternate in ADB notebook its throwing error [PARSE_SYNTAX_ERROR] Syntax error at or near 'string'.(line 1, pos 24) am i missing something

[@user-dk7ut6zu4c]

Sir, Can we get practice notebook?share with us?

[@patiltushar_]

Sir, could you share all those datasets used with us for practice purpose, it will be helpful for us.

[@himanshubhat3252]

Hi Raja, I have a query, that while writing data to csv format, the csv file contains the last line as blank/empty, ( Note : data is ok, but seems last line blank/empty is the default nature of spark ) Is there any way to remove that last blank line while writing the csv file.

[@rajasdataengineering7585]

Usually it doesn't create empty line. There should be specific reason in your use case. Need to analyse more to understand the problem. Using python code, we can remove last line of a file.

[@himanshubhat3252]

@@rajasdataengineering7585 I tried writing csv file using PySpark on Databricks, when i downloaded the file on my system and tried to open it using Notepad++, it shows the last line as blank / empty

[@ANKITRAJ-ut1zo]

could you provide the sample data

[@SurajKumar-hb7oc]

What is the solution If I am reading two files with different column names and different number of columns with a single command? Because I am finding inappropriate output. Please...

[@subhashkamale386]

Hi Raja...I hav some doubt..I wanted to read and display a particular column in data frame...could you please tell me which command should I use... 1. To read single column in data frame 2. To read multiple columns in data frame

[@rajasdataengineering7585]

Hi Subhash, you can use select command in dataframe to read specific columns

[@subhashkamale386]

@@rajasdataengineering7585 could you pls send me the command...I am giving different sytaxes but getting error...I am giving below command df.select(column name)

[@rajasdataengineering7585]

You can give df.select(df.column_name) There are different approaches to refer a column in dataframe. We can prefix dataframe name in front of each column in this method. You can try this method and let me know if still any error

[@subhashkamale386]

@@rajasdataengineering7585 ok Raj..I am trying this in spark data bricks...will let you know if it is working fine..thanks for ur response

[@rajasdataengineering7585]

Welcome

[@SPatel-wn7vk]

please provide ideas to make project using Apache Spark

[@Aspvillagetata]

Hi bro I have some facing issues reading all CSV files and the same all files how to write delta format finally. Finally how delta tables access user view in table format?

[@rajasdataengineering7585]

Hi Pinjari, you can keep all CSV files under a folder and create a dataframe by Spark reader. Then write that dataframe into some other folder in delta format. Delta format is actually parquet file internally. After creating delta table as above, you can use SQL language to do any analytics

[@surajpoojari5182]

I am not able to create a folder in pyspark community edition in DBFS File system please tell me how to do it and not able to delete existing files

[@rajasdataengineering7585]

You can use dbutil command

[@sachinchandanshiv7578]

Hi Sir, Can you please help in understanding the .egg and .zip files we use in --py-files while spark-submit job. Thanks in advance 🙏

[@vinoda3480]

can i get files which your are worked for demo

[@vamsi.reddy1100]

aa intro sound theesivesi manchi pani chesav anna

[@hkpeaks]

What is time required if loading billions of rows?

[@rajasdataengineering7585]

It is depending on many parameters. One of the important parameter is your cluster configuration

[@hkpeaks]

@@rajasdataengineering7585 My desktop PC can process a 250GB Seven billion-row csv kzbin.info/www/bejne/Z3-5YaqhfM-qpbM (for this use case, 1 billion-row/minute)

[@keshavofficial4542]

hi bro, how can i find those csv files?

[@user-xy3vv6zw6z]

can you provide sample data as well

[@sk34890]

Hi Raja where can we access files for practice

[@4abdoulaye]

What happen if you read multiple files that do not have same schema?

[@rajasdataengineering7585]

The rows which don't have same schema will go to corrupted record

[@4abdoulaye]

@@rajasdataengineering7585 Thanks sir. 😎❤

[@rajasdataengineering7585]

Welcome

[@user-bl8hi7je1z]

Can you mention full projects done by pyspark

[@tripathidipak]

Could you please share the sample input files.

[@dinsan4044]

Hi, Could you please create a video to combine below 3 csv data files into one data frame dynamically File name: Class_01.csv StudentID Student Name Gender Subject B Subject C Subject D 1 Balbinder Male 91 56 65 2 Sushma Female 90 60 70 3 Simon Male 75 67 89 4 Banita Female 52 65 73 5 Anita Female 78 92 57 File name: Class_02.csv StudentID Student Name Gender Subject A Subject B Subject C Subject E 1 Richard Male 50 55 64 66 2 Sam Male 44 67 84 72 3 Rohan Male 67 54 75 96 4 Reshma Female 64 83 46 78 5 Kamal Male 78 89 91 90 File name: Class_03.csv StudentID Student Name Gender Subject A Subject D Subject E 1 Mohan Male 70 39 45 2 Sohan Male 56 73 80 3 shyam Male 60 50 55 4 Radha Female 75 80 72 5 Kirthi Female 60 50 55

[@SurajKumar-hb7oc]

I am writing code for the same data but find inappropriate output. What is the solution ?

[@ANJALISINGH-nr6nk]

Can you please share these files with us?

[@shaulahmed4986]

same request for me as well

[@bashask2121]

Can you please provide sample data in the video description

[@rajasdataengineering7585]

Sure Basha, will provide the sample data

[@varun8952]

@@rajasdataengineering7585 , Thanks for sharing the video, is there any GIT link with the data sets and the files you used in the tutorial? If so, could you please share?

[@dataengineerazure2983]

@@rajasdataengineering7585 Please provide sample data. Thank you

[@naveendayyala1484]

Hi Raja Plz share you github code

[@ps-up2mx]

.