End To End Data Engineering Project Using Snowflake | Real Cricket Analytics Use Case

🏏 Dive into the world of one day international (ODI) cricket data with my latest data engineering project, where we've built a comprehensive end to end data engineering solution using Snowflake. This video covers everything from design considerations to the intricacies of data flow and database layer/table architecture.
🔍 Design Considerations: Learn about the key decisions made in designing the data modelling and the overall project architecture.
🌐 Data Flow: From data loading and curation to transformation into Snowflake, understand the seamless flow of information. We'll break down the steps involved in ensuring data quality, integrity, and how Snowflake enhances the efficiency of this process.
Once you complete this end to end real time snowflake based data engineering project (ETL/ELT), you would be able answer of following questions
1. How to load data from location machine to Snowflake stage location.(Small JSON files as well as many JSON files at on go).
2. How to load delta data set from local machine to internal stages?
3. How to use $ notation to query stage file?
4. How to run copy command to load JSON files into tables?
5. How to design different layers & fact/dimension tables?
6. How to automate data flow using task and task tree?
7. How to build quick dashboard using Snowsight?
🚀🚀 How To Connect
✏ Instagram: / learn_dataengineering
✏ Medium Blog: / data-engineering-simpl...
✏ Exclusive Snowflake Facebook Group: / 627874916138090
🚀🚀 Sample JSON Files & SQL Scripts
✏ / 8f8e4f0fd1d0
🚀🚀 Chapters
➥ 00:00:00 Introduction
➥ 00:01:56 Welcome Note
➥ 00:03:07 Udemy Course Offers
➥ 00:03:33 Review Match Level JSON File
➥ 00:06:19 Review E2E Data Flow Diagram
➥ 00:11:07 Part-01 - Pre Requisite
➥ 00:11:58 Part-02 - Database/Schema Creation
➥ 00:15:13 Part-02.1 - Load JSON Files Using SnowSight
➥ 00:17:35 Part-02.2 - Bulk Load JSON Files Using SnowSQL CLI
➥ 00:19:01 Part-03 - Stage to Raw Data Loading
➥ 00:28:46 Part-04 - Cleansing + Curation Introduction
➥ 00:36:09 Part-04.1 - Extract Clean Match Data
➥ 00:48:28 Part-04.2 - Extract Clean Player Data
➥ 00:54:01 Part-04.3 - Extract Clean Bowling Data
➥ 01:12:57 Step-05 - Quick Data Validation
➥ 01:15:22 Step-06 - Populate Fact & Dimension
➥ 01:41:36 Step-07 - Quick Visualisation
➥ 01:43:49 Step-08 - Automate Continuous Data Flow
➥ 01:50:12 Step-09 - Summary
➥ 01:51:00 Thank you note
🚀🚀 Other Popular Playlist
✏ Complete Snowflake Master Class - • Snowflake Tutorial
✏ Snowflake Productivity Hacks • Snowflake Productivity...
✏ Snowflake Cost Optimisation & Cost Management Techniques | Tips To Reduce Snowflake Cost • Snowflake Cost Optimiz...
✏ Snowflake Table - Complete Guide With Example • Snowflake Table - Com...
✏ Snowflake SnowPro Core Certification - Complete Guide • Snowflake SnowPro Core...
✏ SnowPro Core Exam Mock Test - • Snowflake Cerfication ...
✏ Learn Snowpark Using Python API Examples - • Snowpark Tutorial | Le...
✏ Everything About Snowpark - • What Is Snowpark | Lea...
✏ Working With JSON Data In Snowflake - • Working With JSON Data...
✏ How To Load Data Into Snowflake - Masterclass - • How To Load Data Into ...
#dataengineering
#snowflake
#snowflaketutorial
#snowflakedatawrehouse
#snowflakecomputing
#clouddatawarehouse
#snowparktutorial
Disclaimer: All snowflake-related learning materials and tutorial videos published in this channel are the personal opinions of the data engineering simplified team and they're neither authorised by nor associated with Snowflake, Inc.

[@pravachan6577]

This is incredible and passionate work Sir God bless you abundantly

[@abhijitmohanty734]

Most awaited video..Thank you so much Sir !!

[@DataEngineering]

Most welcome.. and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35

[@AnkitGupta-tp3ln]

The amount of effort and dedication that you have put in this video is tremendous. Great work!!

[@DataEngineering]

Thank you so much 😀 if you would like to go more deep in snowflake.. you can also watch my udemy contents.. My current 3 courses are available in discounted price www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=NEW-YEAR-2024 www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=NEW-YEAR-2024 www.udemy.com/course/snowflake-dynamic-table-masterclass-e2e-data-pipeline/?couponCode=B1E84B2CB4AA82CB95E3

[@pavitersingh4698]

this is super helpful

[@saisumanthnagasuri7741]

Excellent 👌👌

[@DataEngineering]

Thanks a lot and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35

[@rpafacts5574]

Great video

[@DataEngineering]

Glad you enjoyed it and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35

[@lu3ma304]

Thank you so much for the video

[@DataEngineering]

You are so welcome! Please try to access following link if want to get access to content. ✏ medium.com/@data-engineering-simplified/8f8e4f0fd1d0 alternatively, you can join my Facebook group and 200 ODI data set is already published there. facebook.com/groups/627874916138090/?mibextid=c7yyfP and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35

[@jaspermurphy8865]

Hi I didn't the get all the file from the gitlab or from medium source link. Could you help me

[@rpafacts5574]

Thanks for the video

[@DataEngineering]

You are so welcome! Please try to access following link if want to get access to sql scripts ✏ medium.com/@data-engineering-simplified/8f8e4f0fd1d0 alternatively, you can join my Facebook group and 200 ODI data set is already published there. facebook.com/groups/627874916138090/?mibextid=c7yyfP and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35

[@Ashu23200]

more projects like this please using more features.

[@DataEngineering]

Thanks for your note... I assume, you have already seen the other end2end project using snowpark kzbin.info/www/bejne/Z5umamuOhtx1kNk

[@vishal_uk]

Amazing stuff!!! May I know what JSON visualizer you're using in this video, please?

[@volukulachaitanya5406]

hi wiil u do the video on generative AI and LLMS with snowflake detailed manner or suggest any resources for that , it will be more helpful for lot of members

[@shubhamalsunde3230]

sir please give data file also so that we can also make the project according with you

[@cargouvu]

Why did you need to connect to Dbeaver to see the PK/FK relationship? Could we not do that in Snowflake?

[@alpitgoyal7532]

The resources that you have used are free or it required subscription to use them??

[@RGDataJourney]

This video helps lot but i don't see any 8 json files in gitlab pls can u provide...

[@faizalqureshi7840]

Couldn't find the first 6 json files to be loaded within landing schema Can anyone please help me with the files so i can proceed further with the projy

[@naveensai2175]

Hi , At 41 minute how do u get result attribute in case statement --> when info:outcome.result = 'tie' then 'Tie' because we have -> "outcome": { "winner": "Sri Lanka", "by": { "wickets": 8 } we have only 2 attributes 'winner' and 'by' in outcome object

[@DataEngineering]

if you try yourself, you would understand how it works.. but your question is not super clear to me... all code is available in my medium page.. so you can download and try it out..

[@ouzaslnn]

Hey, thanks for this nice hands-on project, just wanted to ask which tool do you use to visualize the json ?

[@DataEngineering]

drop me a note to my instagram

[@prateeksmithpatra4454]

FROM WHERE CAN I GET THE DATA THAT YOU USED IN CRICKET DATA, PLS PROVIDE THE DATA ATLEAST NO SIR

[@akshaynaik5926]

I am stuck with multilevel and different attribute json to snowflake conversion using snowpark

[@DataEngineering]

Unless, you share more detail, it is hard to extend help.. You can also watch the complete JSON playlist and try to see if that can help. kzbin.info/aero/PLba2xJ7yxHB6ybgtaIsTKmmF2Nl2wAe2S

[@ashokannam1346]

can we use this project in our resume ?

[@cargouvu]

What tool are you using to visualize the data as nodes?

[@sajaldadhich2743]

json crack.. it's extension is available in VS Code

[@ravikreddy7470]

Do you provide any snowflake online trainings?

[@DataEngineering]

Planning to do it soon, but for now.. No.... pls join my facebook group so if I do it.. you will come to know about it.

[@amanpirjade9]

Do we have to code in snowflake

[@DataEngineering]

To interact with snowflake, you can either do it using standard ANSI SQL or you can also write program using python API or Java API or Scala API. ---- I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35

[@mukundareddy2681]

Data set we do we get?

[@DataEngineering]

check video description

[@nagendradantam4117]

can we get sample json files please

[@DataEngineering]

Please try to access following link ✏ medium.com/@data-engineering-simplified/8f8e4f0fd1d0 alternatively, you can join my facebook group and 200 ODI data set is already published there. facebook.com/groups/627874916138090/?mibextid=c7yyfP and yes, I know many of us are not fully aware of snowpark Python API, if you want to manage snowflake more programatically.. you can watch my paid contents (data + code available) .. many folks don't know the power of snowpark... these 2 videos... will help you to broaden your knowledge.. These contents are available in discounted price for limited time.. (one for JSON and one for CSV).. it can automatically create DDL and DML and also run copy command... 1. www.udemy.com/course/snowpark-python-ingest-json-data-automatically-in-snowflake/?couponCode=DIWALI50 2. www.udemy.com/course/automatic-data-ingestion-using-snowflake-snowpark-python-api/?couponCode=DIPAWALI35

[@nagendradantam4117]

@@DataEngineering Thank you so much😊

[@naveengvkvs1915]

Hi Thanks for the detail explanation, I have a query After loading of data from stage to raw table don't we need to move or clean the files from stage location? Copy command only stores 64 days of metadata then after 64 days if old files are still there it will be reprocessed again? Can we think of a mechanism to archive the processed files from stage location?

[@DataEngineering]

I have never tried this 64 days concept and there is a parameter to control it. But thanks for the note, I will try it to check if 64 old data is re-loaded or not.

[@borisvainrub6341]

What name for visual graph program? kzbin.info/www/bejne/p3XQonhubqmGh7M

[@kskr101]

Hello sir, could you share your LinkedIn id

[@DataEngineering]

for any queries.. you can reach out to me via insta or from my facebook page.