Building our first Spark Streaming Application! | PySpark Tutorial

  Рет қаралды 10,231

AmpCode

AmpCode

Күн бұрын

Пікірлер
@abdullahyahya2471
@abdullahyahya2471 11 ай бұрын
Perfectly structured Video. Thanks.
@ampcode
@ampcode 11 ай бұрын
Glad it was helpful!
@vivianhuang7670
@vivianhuang7670 Жыл бұрын
I followed this run the whole script, but same with vscode, it's not showing anything...it's like my execution just stuck after one warning msg: 23/12/29 15:44:07 WARN ResolveWriteToStream: spark.sql.adaptive.enabled is not supported in streaming DataFrames/Datasets and will be disabled... May I know why?
@QuangHuy-is7jo
@QuangHuy-is7jo 9 ай бұрын
you can change to pycharm, i tried and succeeded
@HuyPham-id7us
@HuyPham-id7us 2 ай бұрын
@@QuangHuy-is7jo bạn ơi, mình xem series này mà lúc connect với database postgres thì nó báo bị lỗi không xóa được bộ nhớ đệm, bạn có gặp lỗi như vậy không ạ, giúp mình với ERROR ShutdownHookManager: Exception while deleting Spark temp dir
@pratibhakoli4047
@pratibhakoli4047 Жыл бұрын
Sir I have project base on spark i.e. real time wether forcast using spark streaming. So how can I start this project.
@pratibhakoli4047
@pratibhakoli4047 Жыл бұрын
Any material or udemy courses to complete that one
@ampcode
@ampcode Жыл бұрын
Sorry for late reply. Thanks for your suggestion and I would really like to do this project once we'll complete all this basic stuff. Thanks again!
@pratibhakoli4047
@pratibhakoli4047 Жыл бұрын
Ok sir
@sanooosai
@sanooosai 9 ай бұрын
great thank you
@daleo156
@daleo156 Жыл бұрын
🌟 'promosm'
@hamnakhalidbutt7166
@hamnakhalidbutt7166 Жыл бұрын
# Start streaming defining the necessary configurations streamingQuery = (counts .writeStream .format("console") .outputMode("complete") .trigger(processingTime="1 second") .option("checkpointLocation", checkpointDir) .start()) streamingQuery.awaitTermination()......This step is not executing as there is an issue with starting the streaming query .there is checkpoint directory in C drive and The specific error code "exitCode=-1073741515" indicates a problem with the Hadoop file system permissions.Could you please make any suggestion?
@ampcode
@ampcode Жыл бұрын
Sorry for late response. Could you please check if you have created the checkpoint directory and spark is able to access it?
@vigneshpillai5360
@vigneshpillai5360 Жыл бұрын
Facing same issue. checkpoint directory is created but getting "py4j.protocol.Py4JJavaError: An error occurred while calling o76.start." error and exitCode as ": ExitCodeException exitCode=-1073741515:".
@atv8992
@atv8992 Жыл бұрын
@@vigneshpillai5360 yours might be because of directory path
Counter-Strike 2 - Новый кс. Cтарый я
13:10
Marmok
Рет қаралды 2,8 МЛН
Air Sigma Girl #sigma
0:32
Jin and Hattie
Рет қаралды 45 МЛН
🎈🎈🎈😲 #tiktok #shorts
0:28
Byungari 병아리언니
Рет қаралды 4,5 МЛН
Streaming from Kafka Source
16:56
Learning Journal
Рет қаралды 7 М.
The most important Python script I ever wrote
19:58
John Watson Rooney
Рет қаралды 212 М.
Big Data Engineering Full Course Part 1 | 17 Hours
9:31:53
The Data Tech
Рет қаралды 288 М.
Building Production-Ready RAG Applications: Jerry Liu
18:35
AI Engineer
Рет қаралды 340 М.
The ONLY PySpark Tutorial You Will Ever Need.
17:21
Moran Reznik
Рет қаралды 147 М.
Counter-Strike 2 - Новый кс. Cтарый я
13:10
Marmok
Рет қаралды 2,8 МЛН