Пікірлер
@CloudMaster_Abhiram 12 сағат бұрын
Syntax for Find the top N most frequent words in a large text file from pyspark import SparkContext # create your spark context sc = SparkContext("local", "WordCount") # import a text file from a local path lines = sc.textFile("path/to/your/text/file.txt") # split and map the words # then reduce by using the words as keys and add to the count word_counts = lines.flatMap(lambda line: line.split(" ")) \ .map(lambda word: (word, 1)) \ .reduceByKey(lambda a, b: a + b) # order the words and take only the top N frequent words top_n_words = word_counts.takeOrdered(N, key=lambda x: -x[1]) print(top_n_words)
@CloudMaster_Abhiram 2 күн бұрын
import pyspark from pyspark.sql import SparkSession from pyspark.sql.functions import expr #Create spark session data = [("Devara",1000,"India"), ("Kalki",1500,"India"), ("Pushpa",1600,"India"), \ ("Devara",4000,"USA"), \ ("Pushpa",1200,"USA"),("Kalki",1500,"USA"), \ ("Pushpa",2000,"Canada"),("Kalki",2000,"Canada"),("Devara",2000,"Mexico")] columns= ["Product","Amount","Country"] df = spark.createDataFrame(data = data, schema = columns) df.printSchema() df.show(truncate=False) Output: root | -- Product: string (nullable = true) | -- Amount: long (nullable = true) | -- Country: string (nullable = true) +-------------+-------------+-------------+ | Product | Amount | Country | +-------------+-------------+--------------+ | Devara | 1000 | India | | Pushpa | 1600 | India | | Kalki | 1500 | India | | Devara | 4000 | USA | | Pushpa | 1200 | USA | | Kalki | 1500 | USA | | Devara | 2000 | Mexico | | Pushpa | 2000 | Canada | | Kalki | 2000 | Canada | +-------------+--------------+--------------+ To determine the entire amount of each product's exports to each nation, we'll group by Product, pivot by Country, and sum by Amount. pivotDF = df.groupBy("Product").pivot("Country").sum("Amount") pivotDF.printSchema() pivotDF.show(truncate=False) This will convert the nations from DataFrame rows to columns, resulting in the output seen below. Output: root | -- Product: string (nullable = true) | -- Canada: long (nullable = true) | -- Mexico: long (nullable = true) | -- USA: long (nullable = true) | -- India: long (nullable = true) +-------------+-------------+------------+-------+---------+ | Product | Canada | Mexico | USA | India | +-------------+-------------+------------+--------+---------+ | Devara | null | 2000 | 4000 |1000 | | Pushpa | 2000 | null | 1200 | 1600 | | Kalki | 2000 | null | 1500 | 1500 | +-------------+-------------+------------+---------+---------+
@Sudarshan-d7s 2 күн бұрын
@@CloudMaster_Abhiram thanks I do follow your channel For spark related stuffs❤️ I want to do deep preparation for spark how should I go ahead 😅
@CloudMaster_Abhiram 2 күн бұрын
@@Sudarshan-d7s How about getting enrolled in us
@Sudarshan-d7s 2 күн бұрын
Syntax kidhr he 😂
@CloudMaster_Abhiram 2 күн бұрын
Haha i forgot just pinned the comment do check it.
@datningole1038 12 күн бұрын
Good.. keep it up
@sspinjari 15 күн бұрын
Please do not add background music it is too much distracting. Or you can add silent music
@anilanche5753 16 күн бұрын
Hi Abhiram Could you please let me know how to migrate job from datastage to informatica power center or iics
@CloudMaster_Abhiram 17 күн бұрын
from pyspark.sql import SparkSession, Row from pyspark.sql import functions as F # Initialize Spark session spark = SparkSession.builder.appName("concatenate_columns").getOrCreate() # Sample DataFrame data = [ Row(struct_col=Row(a=1, b="foo"), array_col=[Row(c=3, d="bar"), Row(c=4, d="baz")]) ] df = spark.createDataFrame(data) # Flatten the struct column flattened_struct_col = F.concat_ws( ",", *[F.col("struct_col." + field.name) for field in df.schema["struct_col"].dataType.fields] ) # Flatten the array of structs column flattened_array_col = F.expr(""" concat_ws(",", transform(array_col, x -> concat_ws(",", x.*))) """) # Concatenate the two columns df = df.withColumn( "concatenated_col", F.concat_ws(",", flattened_struct_col, flattened_array_col) ) # Show result df.show(truncate=False)
@abhishekshah581 Ай бұрын
Please don't put any background music and give little more time to read the slides
@CloudMaster_Abhiram Ай бұрын
@@abhishekshah581 Noted
@achievement7545 Ай бұрын
Kindly stop that background music
@CloudMaster_Abhiram Ай бұрын
😅ok from next time new bgm 👍
@thelayer5211 Ай бұрын
Pls send data file
@Relax_dude_with_food 2 ай бұрын
Voice rale sir
@CloudMaster_Abhiram 2 ай бұрын
Do watch entire video. In the end you will find the overview explanation clearly
@irugugopi203 2 ай бұрын
No voice
@CloudMaster_Abhiram 2 ай бұрын
Watch entire video
@irugugopi203 2 ай бұрын
@@CloudMaster_Abhiram chusina sir
@CloudMaster_Abhiram 2 ай бұрын
@@irugugopi203 Do watch entire video. In the end you will find the overview explanation clearly
@venkateshbatchu5925 2 ай бұрын
Thank you so much. Starting lo nay input data tho dataframe create chesay time lo arguments error vachindhi kadha. Adhi okasari cheppandi anna fix
@Relax_dude_with_food 2 ай бұрын
Nice xplaining sir
@Relax_dude_with_food 2 ай бұрын
Nice explanation sir
@CloudMaster_Abhiram 2 ай бұрын
Enroll Now: "Azure Data Engineer Training & Placement Program" Start Date: Every Month 1st week || 7:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 👉Features of Online Training: 👉Real-Time Oriented Training 👉Live Training Sessions 👉Interview Preparation Tips 👉FAQ’s 👉100% Job Guarantee Program 👉Mock Interviews
@CloudMaster_Abhiram 2 ай бұрын
Enroll Now: "Azure Data Engineer Training & Placement Program" Start Date: Every Month 1st week || 7:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 👉Features of Online Training: 👉 Real-Time Oriented Training 👉Live Training Sessions 👉Interview Preparation Tips 👉FAQ’s 👉100% Job Guarantee Program 👉Mock Interviews
@CloudMaster_Abhiram 2 ай бұрын
Enroll Now: "Azure Data Engineer Training & Placement Program" Start Date: Every Month 1st week || 7:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 👉Features of Online Training: 👉 Real-Time Oriented Training 👉Live Training Sessions 👉Interview Preparation Tips 👉FAQ’s 👉100% Job Guarantee Program 👉Mock Interviews
@CloudMaster_Abhiram 2 ай бұрын
Enroll Now: "Azure Data Engineer Training & Placement Program" Start Date: Every Month 1st week || 7:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 👉Features of Online Training: 👉 Real-Time Oriented Training 👉Live Training Sessions 👉Interview Preparation Tips 👉FAQ’s 👉100% Job Guarantee Program 👉Mock Interviews
@CloudMaster_Abhiram 2 ай бұрын
Enroll Now: "Azure Data Engineer Training & Placement Program" Start Date: Every Month 1st week || 7:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 👉Features of Online Training: 👉 Real-Time Oriented Training 👉Live Training Sessions 👉Interview Preparation Tips 👉FAQ’s 👉100% Job Guarantee Program 👉Mock Interviews
@CloudMaster_Abhiram 2 ай бұрын
Enroll Now: "Azure Data Engineer Training & Placement Program" Start Date: Every Month 1st week || 7:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 👉Features of Online Training: 👉 Real-Time Oriented Training 👉Live Training Sessions 👉Interview Preparation Tips 👉FAQ’s 👉100% Job Guarantee Program 👉Mock Interviews
@CloudMaster_Abhiram 2 ай бұрын
Enroll Now: "Azure Data Engineer Training & Placement Program" Start Date: Every Month 1st week || 7:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 Features of Online Training: 👉Real-Time Oriented Training 👉Live Training Sessions 👉 Interview Preparation Tips 👉FAQ’s 👉100% Job Guarantee Program 👉Mock Interviews
@CloudMaster_Abhiram 2 ай бұрын
Enroll Now: "Azure Data Engineer Training & Placement Program" Start Date: Every Month 1st week || 7:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 Features of Online Training Real-Time Oriented Training Live Training Sessions Interview Preparation Tips FAQ’s 100% Job Guarantee Program Mock Interviews
@aakashtijare1801 2 ай бұрын
How to join
@CloudMaster_Abhiram 2 ай бұрын
Contact me +91 9281106429
@irugugopi203 2 ай бұрын
Spark=sparksession.builder.master THIS LINE NEED!?
@srilakshmivanka9500 2 ай бұрын
What about standard cluster....?
@CloudMaster_Abhiram 2 ай бұрын
It is a set of computation resources and configurations on which you run data engineering, data science, and data analytics workloads, such as production ETL pipelines, streaming analytics, ad-hoc analytics, and machine learning.
@kicknaveen786 2 ай бұрын
Hi sir can you please share document for what you are explained in short videos about azure data bricks interview questions
@irugugopi203 2 ай бұрын
👍
@CloudMaster_Abhiram 2 ай бұрын
😄😄
@praveenmek 3 ай бұрын
Thanks Abhiram for this informative content.
@CloudMaster_Abhiram 3 ай бұрын
Thank you for your feedback! I'm glad you found the content informative.
@RANJITHKUMAR-sn2gp 3 ай бұрын
👍
@CloudMaster_Abhiram 4 ай бұрын
Enroll Now: "Azure Data Engineer Training & Placement Program" Start Date: Every Month 1st week || 7:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 Features of Online Training Real-Time Oriented Training Live Training Sessions Interview Preparation Tips FAQ’s 100% Job Guarantee Program Mock Interviews
@CloudMaster_Abhiram 4 ай бұрын
✍Enroll Now: "Azure Data Engineer Training & Placement Program" 📅 Start Date: Every Month 1st week || 7:00 pm IST For More Details: 📱 Call: +91 9281106429 👉 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 💥 Features of Online Training ✅ Real-Time Oriented Training ✅ Live Training Sessions ✅ Interview Preparation Tips ✅ FAQ’s ✅ 100% Job Guarantee Program ✅ Mock Interviews
@CloudMaster_Abhiram 4 ай бұрын
✍Enroll Now: "Azure Data Engineer Training & Placement Program" 📅 Start Date: Every Month 1st week || 7:00 pm IST For More Details: 📱 Call: +91 9281106429 👉 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 💥 Features of Online Training ✅ Real-Time Oriented Training ✅ Live Training Sessions ✅ Interview Preparation Tips ✅ FAQ’s ✅ 100% Job Guarantee Program ✅ Mock Interviews
@CloudMaster_Abhiram 4 ай бұрын
✍Enroll Now: "Azure Data Engineer Training & Placement Program" 📅 Start Date: Every Month 1st week || 7:00 pm IST For More Details: 📱 Call: +91 9281106429 👉 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 💥 Features of Online Training ✅ Real-Time Oriented Training ✅ Live Training Sessions ✅ Interview Preparation Tips ✅ FAQ’s ✅ 100% Job Guarantee Program ✅ Mock Interviews
@CloudMaster_Abhiram 4 ай бұрын
✍Enroll Now: "Azure Data Engineer Training & Placement Program" 📅 Start Date: Every Month 1st week || 7:00 pm IST For More Details: 📱 Call: +91 9281106429 👉 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 💥 Features of Online Training ✅ Real-Time Oriented Training ✅ Live Training Sessions ✅ Interview Preparation Tips ✅ FAQ’s ✅ 100% Job Guarantee Program ✅ Mock Interviews
@Potti845 4 ай бұрын
Good explanation
@CloudMaster_Abhiram 4 ай бұрын
Thanks and welcome
@CloudMaster_Abhiram 4 ай бұрын
✍Enroll Now: "Azure Data Engineer Training & Placement Program" 📅 Start Date: Every Month 1st week || 7:00 pm IST For More Details: 📱 Call: +91 9281106429 👉 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 💥 Features of Online Training ✅ Real-Time Oriented Training ✅ Live Training Sessions ✅ Interview Preparation Tips ✅ FAQ’s ✅ 100% Job Guarantee Program ✅ Mock Interviews
@CloudMaster_Abhiram 4 ай бұрын
✍Enroll Now: "Azure Data Engineer Training & Placement Program" 📅 Start Date: Every Month 1st week || 7:00 pm IST For More Details: 📱 Call: +91 9281106429 👉 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 💥 Features of Online Training ✅ Real-Time Oriented Training ✅ Live Training Sessions ✅ Interview Preparation Tips ✅ FAQ’s ✅ 100% Job Guarantee Program ✅ Mock Interviews
@CloudMaster_Abhiram 4 ай бұрын
Enroll Now: "Azure Data Engineer Training & Placement Program" Start Date: Every Month 1st week || 7:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 Features of Online Training Real-Time Oriented Training Live Training Sessions Interview Preparation Tips FAQ’s 100% Job Guarantee Program Mock Interviews
@CloudMaster_Abhiram 4 ай бұрын
Enroll Now: "Informatica Power Center Training & Placement Program" Start Date: Every Month 1st week || 8:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 Features of Online Training Real-Time Oriented Training Live Training Sessions Interview Preparation Tips FAQ’s 100% Job Guarantee Program Mock Interviews
@CloudMaster_Abhiram 4 ай бұрын
Enroll Now: "Informatica Power Center Training & Placement Program" Start Date: Every Month 1st week || 8:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 Features of Online Training Real-Time Oriented Training Live Training Sessions Interview Preparation Tips FAQ’s 100% Job Guarantee Program Mock Interviews
@CloudMaster_Abhiram 4 ай бұрын
Enroll Now: "Informatica Power Center Training & Placement Program" Start Date: Every Month 1st week || 8:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 Features of Online Training Real-Time Oriented Training Live Training Sessions Interview Preparation Tips FAQ’s 100% Job Guarantee Program Mock Interviews
@CloudMaster_Abhiram 4 ай бұрын
Enroll Now: "Informatica Power Center Training & Placement Program" Start Date: Every Month 1st week || 8:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 Features of Online Training Real-Time Oriented Training Live Training Sessions Interview Preparation Tips FAQ’s 100% Job Guarantee Program Mock Interviews
@CloudMaster_Abhiram 4 ай бұрын
Enroll Now: "Informatica Power Center Training & Placement Program" Start Date: Every Month 1st week || 8:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 Features of Online Training Real-Time Oriented Training Live Training Sessions Interview Preparation Tips FAQ’s 100% Job Guarantee Program Mock Interviews
@CloudMaster_Abhiram 4 ай бұрын
Enroll Now: "Informatica Power Center Training & Placement Program" Start Date: Every Month 1st week || 8:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 Features of Online Training Real-Time Oriented Training Live Training Sessions Interview Preparation Tips FAQ’s 100% Job Guarantee Program Mock Interviews
@CloudMaster_Abhiram 4 ай бұрын
Enroll Now: "Informatica Power Center Training & Placement Program" Start Date: Every Month 1st week || 8:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 Features of Online Training Real-Time Oriented Training Live Training Sessions Interview Preparation Tips FAQ’s 100% Job Guarantee Program Mock Interviews
@kamasanichittibabu5013 4 ай бұрын
I saw the first video and comining to second looks skipped the part
@kamasanichittibabu5013 4 ай бұрын
where did you explained setting up secure agent?
@CloudMaster_Abhiram 4 ай бұрын
Check it out here Part 1: kzbin.info/www/bejne/bneZZXiFadpmd5o&ab_channel=CloudArchitectAbhiram Part 2: kzbin.info/www/bejne/maHIh3-Ho8pjq9E&ab_channel=CloudArchitectAbhiram
@CloudMaster_Abhiram 4 ай бұрын
Enroll Now: "Informatica Power Center Training & Placement Program" Start Date: Every Month 1st week || 8:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 Features of Online Training Real-Time Oriented Training Live Training Sessions Interview Preparation Tips FAQ’s 100% Job Guarantee Program Mock Interviews
@CloudMaster_Abhiram 4 ай бұрын
Enroll Now: "Informatica Power Center Training & Placement Program" Start Date: Every Month 1st week || 8:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 Features of Online Training Real-Time Oriented Training Live Training Sessions Interview Preparation Tips FAQ’s 100% Job Guarantee Program Mock Interviews
@CloudMaster_Abhiram 4 ай бұрын
Enroll Now: "Informatica Power Center Training & Placement Program" Start Date: Every Month 1st week || 8:00 pm IST For More Details: Call: +91 9281106429 Chat with us on WhatsApp: wa.me/qr/PSW2ILTYJHTZI1 Features of Online Training Real-Time Oriented Training Live Training Sessions Interview Preparation Tips FAQ’s 100% Job Guarantee Program Mock Interviews