Manage complex SQL workflows and transformations in BigQuery

[@Luke7389]

This is a great intro to showcase the concepts, now a video about how to move from the BQ UI to this would be good. In particular how to test locally before deployment.

[@Atee1995]

This video was awesome until you opened VS code. I think an approach from the UI side to VS would have been much better, also explaining local dependencies and what needs to be installed to make it work

[@patriciomendez9240]

Excellent point. I agree.

[@Satenc0]

Any examples on how to use variables on the queries? like parametrizable variables, for example if you have a query that fetches records between 2 dates, how could you make those 2 dates variables that are inputs to the sqlx file

[@goodmanshawnhuang]

thanks for good video, what's the difference between dataform and DBT please? thanks.

[@resagratia]

They achieve the same purpose. DBT is able to connect with various data warehouses including BigQuery, Redshift and Azure. While Dataform is an internal framework for BigQuery.

[@zeleravideos]

The first build seems to have failed, so I think the video should be edited to remove content from 15:40 to 16:15. 😅

[@ScottEdwards2000]

thanks - what about data lineage? does dataform support that?

[@ZachRenwickData]

lineage is supported within dataform - click on "compiled graph" to show a visual of your dataform pipeline dependencies/lineage

[@ZachRenwickData]

this assumes you are using the dataform UI within BigQuery

[@chad_the_stud]

dataform should have notebook. sql notebooks are much more efficient at creating modular workflows rather than managing a monolithic massive queries with tens of ctes

[@emmanuelvillarbarrera2588]

I want the repo in git for dataform please

[@kirilllosik7054]

yeah, there is supposed to be a git repo, but it's missing

[@saad-c1]

are you left handed?

[@parthamishra09]

LOL, you call this 'complex'