Yes. It's on my list. :-) Thanks

Yes. Databricks has a workflow service that is powerful. See docs.databricks.com/en/jobs/index.html

You create a separate environment for each project so you must deactivate one to switch to another.

Hmm... not sure about the idea of throwing away configuration that is written down with a bunch of non-documented, non-recreatable jobs.

Good question. When I did on prem ETL, we used SQL Server Integration Services (SSIS) which is proprietary but an excellent ETL tool. For open-source, I really like Dagster (dagster.io/) which is a Python based data orchestration framework. While it has some of the same issues as Airflow, the data centric nature of Dagster including integrated data validation, data lineage tracking, and composability make it far superior to Airflow in my opinion. I have not used it in production so I would recommend a POC and pilot before committing to Dagster. The Dagster university free online training is good to get started. Dagster is still an orchestrator, not an ETL service. I've seen and looked at some open source ETL tools but have not dug in enough to recommend any. Does that help?

Interesting. I think that must have been added to a new release of Spark since I recorded this video. Thanks for the comment.

Sorry for any confusion. 2,000 is a fact b/c is a quantity and likely to be summarized by the business. 8% is an attribute of the payment method so in this case, I am saying I would add that to the payment dimension. If the business wants to focus on the credit card interest, like aggregate over dimensions, then it could be created as a fact. This is where there is ambiguity. How the data is used will drive whether it is a fact or dimension but facts must be quantifiable. Thanks for the question.

You're welcome!@ Glad my content is helpful.

Yeah. It does add complexity but you have the gist of it. Surrogate keys are particuarly important when you want SCD history since natural keys would result in duplicate keys on the dim tables. Also, they isolate changes from the backend systems to the dw. But they do add some extra work.

Thank you for confirming my understanding and great presentation!

You're welcome!

Good question. 1) Fabric is a end user self service platform whereas Databricks is more of a Data Engineering and AI solution building platform. 2) Fabric is extremely well suited to supporting Power BI reporting and can even eliminate the need to load a Tabular Model., 3) the t-shirt Fabric capacities means you select a fixed compute level that all the Fabric objects and services share so the cost is contained. However, you could find that you need to increase the Fabric capacity at some point. 4) with Fabric, you don't need to manage or even explicitly create the underlying Azure resources, Fabric does it for you and even hides the details. This is extended with the OneLake and shortcuts concepts. Make sense?

@@BryanCafferky Would you then say, that Fabric is more akin to a Modern Data Stack and more focused on the Analytics Engineer role rather than to a dimensional modeled warehouse?

@@mubar2 I think Fabric is aimed at the end user, not the engineer. It gives business users tools to do analysis, build dashboards and build solutions but they will not have the skills to build a a Kimball styled data warehouse in my opinion. They will get something up and running fast.

Thank you! Glad it is helpful.

Thanks for your thoughts.

Thank You!

Thank You! Yes. Spark is on fire. :-)

Hmmm... Not sure what you mean by building manually. I think that's the only way you can create DLT pipelines. Bear in mind, you can NOT run a notebook directly in the notebook UI for DLT.

Hey sorry I didn't mean building manually, I meant running manually after managing the cluster configurations (node type Id for both driver and the worker) and then managing whether to store the target schema at the catalogs schema. I've given the location of the notebook for the pipeline. Not sure where it's going wrong..

@@shreyasd99 Take a look at this blog about DLT cluster configuration docs.databricks.com/en/delta-live-tables/settings.html

I think it is really the compute you use that cost the money. If its 40 nodes running in parallel over 200 TBs of data for 3 hours, that will cost whereas a single node running for a few minutes is cheap. I do it all the time on my personal account.

@@BryanCafferky thank you good sir

Looks interesting. How does it fit in and work with Databricks Asset Bundles (DABs)? Does it work with Azure DevOps? Thanks

@@BryanCafferky It can actually be used as a replacement to DABs. I started working on this project before DABs was announced :) It offers more or less the same capabilities as DABs, except for a few key differences: - State management is more aligned with Terraform than DABs, meaning that the state is not automatically saved to your Databricks workspace. As a consequence, the deployments are global by default and not "user-specific". - Laktory supports multiple IaC backends. You can use terraform, but you can also use Pulumi. - Laktory supports almost any Databricks resources. You can not only deploy notebooks and jobs, but also Catalogs, Schemas, Tables, clusters, warehouses, vector search endpoints, queries, secrets, etc. - As per my initial comment, Laktory is an ETL framework, so you can use it to define all your data transformations through SQL statements or Spark Chain, a serialized expression of spark commands. In other words, Laktory is like DAB + dbt in a single framework, but with a strong focus on DataFrame and Spark transformations. I haven't used it with Azure DevOps yet, but they would definitely work nicely together using Laktory CLI. You can find an example of a git action here: github.com/okube-ai/lakehouse-as-code/blob/main/.github/workflows/_job_laktory_deploy.yml The syntax is a bit different than with Azure DevOps, but it would be very similar.

Glad they are helping.

Thank You!

Looks like you have Unity Catalog enabled and it is conflicting with this code. UC came out after I created this video.

Looks very similar but split apply works on dataframes whereas map reduce works on RDDs.

@@BryanCafferky I thought it’s just a pattern?

@@ichtot71 It is supported in the PySpark library. See www.databricks.com/blog/2020/05/20/new-pandas-udfs-and-python-type-hints-in-the-upcoming-release-of-apache-spark-3-0.html

Yes. You can. It won't scale out to multiple machines but the code should still work.

I have not dared to try it again. lol I don't think it is the best idea for a scaled out service to be honest. Ok if updates are limited but I would still try to find an alternative.