Manage complex SQL workflows and transformations in BigQuery

Data pipelines continue to grow in complexity and size, yet 43% of all companies continue to identify data talent as their largest gap. Join Google Cloud as we go on a transformative journey to unlock the power of data engineers and data analysts to write complex data pipelines, all in a managed framework that can be easily deployed using Google Cloud's latest data innovation. Data professionals can now build complex SQL logic that engineers can deploy using various deployment patterns including CLI to support enterprise needs.
Stat Link → goo.gle/3an6l6M
Speaker: Jason Davenport
Watch more:
Watch all Data Engineering sessions → goo.gle/3yMj6Bk
Subscribe to Google Cloud Events → goo.gle/Google...
#GoogleCloudSpotlight

[@Atee1995]

This video was awesome until you opened VS code. I think an approach from the UI side to VS would have been much better, also explaining local dependencies and what needs to be installed to make it work

[@patriciomendez9240]

Excellent point. I agree.

[@goodmanshawnhuang]

thanks for good video, what's the difference between dataform and DBT please? thanks.

[@resagratia]

They achieve the same purpose. DBT is able to connect with various data warehouses including BigQuery, Redshift and Azure. While Dataform is an internal framework for BigQuery.

[@zeleravideos]

The first build seems to have failed, so I think the video should be edited to remove content from 15:40 to 16:15. 😅

[@Satenc0]

Any examples on how to use variables on the queries? like parametrizable variables, for example if you have a query that fetches records between 2 dates, how could you make those 2 dates variables that are inputs to the sqlx file

[@ScottEdwards2000]

thanks - what about data lineage? does dataform support that?

[@ZachRenwickData]

lineage is supported within dataform - click on "compiled graph" to show a visual of your dataform pipeline dependencies/lineage

[@ZachRenwickData]

this assumes you are using the dataform UI within BigQuery

[@emmanuelvillarbarrera2588]

I want the repo in git for dataform please

[@kirilllosik7054]

yeah, there is supposed to be a git repo, but it's missing

[@saad-c1]

are you left handed?

[@parthamishra09]

LOL, you call this 'complex'