Excellent video bro!

I bet the h1-bs are still there.

Really helpful, practical advice! Thank you

Your content is great man, keep it up! you've helped a lot

What many fail to realise is that a bad data warehouse is not just bad table structure, but also, low documentation, redundant calculations, unnecessarily complicated ETL (mostly tech debt). All of which make the warehouse unusable and difficult to maintain. I also see wrong approach when creating a warehouse e.g. just looking at existing reports to create a data model, no data profiling, no business study, no articulation of data loading rules, heavy on undocumented assumptions. Eventually the new shiny warehouse by modellers is also discarded by analysts as it is not fit for purpose, because the same mistake is repated again and again.

4:55 while a good video introduction to Apache iceberg, there are a few point that needs to be clarified here. Delta Lake is not owned by Databricks but by the Linux Foundation since October 2019, unlike the 2 other table formats Hudi and Iceberg. Apache Kudu, Cassandra and Druid are not table formats so cannot and shouldn’t be compared to Apache Iceberg. Databricks don’t have a Delta engine. Delta, Hudi and Iceberg are all 3 available on AWS in AWS Amazon Glue, AWS EMR and Amazon Redshift Spectrum. It’s not just Iceberg which is supported there. All 3 table formats are fully open source, not owned by any company and community driven. They are all 3 supported by the same set of Query engine such as Spark, Flink, Presto, Trino and more. Delta lake is not just supported by Databricks (which you will hear a lot in the Iceberg community) but also by Google BigQuery and Snowflake. Microsoft Fabric platform is built on top of Delta Lake table format.

"that's it"

Everything is “AI” today because greedy people want to monetize it. Same thing happened with “Blockchain” around 2016. Seemingly overnight there were hundreds of startups selling “blockchain” solutions. Don’t buy they hype, literally

I dunno man, our management identified that our business may be at risk if nuclear warfare breaks out, so we are building a truly resilient platform which replicates data on Mars. We call it "Storm Data Warehouse", because there's not enough water on Mars to use a Cloud. We're also expecting that every person, dead or yet unborn will be incarnated as an AI consciousness, and thus will be our user, so we have to build high-quality robust pipelines that ingest exabytes of data in real-time, so that the CEO can bask in the glow of twenty-four screens that erupt to life as he wakes up at 4:32am to look at the KPI dashboards to change the world.

Aio as an data engineer idk but several times i just laughed when you say moving data from one place to another 😂 but it was fun

Wow..had a lot of fun and it removed anxiety

“In a world deluged by irrelevant information, clarity is power.” Yuval Noah Harari

Very clean

Steps to make pipeline better 1. Good auditing and logging: error handling 2. Repeatable and identical 3. Self healing: finding a way to find the delta , log files and compare, add a data lake before data warehouse , add hash or water marks before compare 4. Decouple EL and T: Landon Rae formate, transform to Dwh, make reporting table clean, 5. Always available: trancate and load refresh faster than update. Or build semantic layer 6. CICD: coded, git connected, versioned , rollbacks

Finally, a short and good guide.

Great explanation!

Spot on! That's what every data job seeker should understand. By the way, I'm a big fan of your videos, your channel deserves a lot more attention. Keep up the good work 🙌

All of analytics and data engineering is just applied accounting. The more you know about finance, the stronger you are as an analytics professional. - Lauren Balik

Glad to see you're back on YT.

Amazingly simplified with your explanation. Thanks!

speaking speed too fast

Nice presentation !! This solution has one down side , you pay 2 services.

I like your videos, great explanation with clear and nice examples. I just need more time and effort to focus because of the loud music in the background. I hope to make it low in rest of your videos. thanks again for your explanation and your time.

how did you learn especially your cloud skills?

You have a great friend 🔥🔥🔥🔥✨✨

It is all about problem solving. Unfortunately, companies are too consumed with hiring the best technical talent putting them through rigorous coding challenges. But, they forget to ask how you solve a problem, take us through your thought process when you are trying to solve something.

You will grow in KZbin... I vouch for you 👍

“That snake game” 😂😂😂😂

Really, the best explanation.

This all comes from universities creating too many types of majors

You shouldn’t call yourself a data engineer if you are not a software developer yourself

I love your videos, I see that you aren't trying to clickbait, hype them up or dumb them down to get more likes and views, mostly because KZbin isn't your income stream. As a result, we just get a great analysis of whats out there. Legend

That was a great walkthrough

Well explained in pictorial format. But there should be some use case or an example so the newbies can understand more easily.

Hello. I hope you well. Is it possible to become a Data Warehouse developer without leaning any programming language?

You are contradicting yourself. Inmon's model has a datamarts layer with denormalized and ready to consume data, so how should that slow down reporting queries?

very nice explanation

2:00 Using a dimensional model to build OBT May be okay, but seems that analysts should build this via pivot table query instead of data team.

funny enough descriptive analysis falls under data science as well and many data analyst jobs keep you in meeting and creating quick dashboards with little to no analysis. To do well, have all skills under your belt just in case

I have been saying this for 5 y already

Good video (no apostrophe in 'lots' btw). Love - Grammar Police

how to find that, when table is last refresh in snowflake for tables without date field

That is so true. Thanks for the honesty

I am actually really confused whether to go for data engineer or devops engineering.. less code , less responsibility

One of the best videos

Why so little views? Good stuff. Well made.

This is pure gold. Too many DEs don't realize we exist to solve business problems. FYI, Your videos are excellent and separated from the tons and tons of videos that are just trying to sell a course.

Excellent video. Thank you for sharing!

sad for me as a data scientist

Awesome video. Minor nitpick: the D in DAG is “directed” as in directed arrows. (End nitpick.) As popular as data engineering is getting, it’s really hard to find conceptual videos like this on the basics.