Microsoft Fabric: Processing Bronze to Silver using Fabric Notebooks

[@endjin]

Thank you for watching, if you enjoyed this episode, please hit like 👍subscribe, and turn notifications on 🔔it helps us more than you know. 🙏

[@carlnascnyc]

this is indeed a great series, the best one of all youtube imo, it's direct to the point and straightforward, just the right amount of complexity, can't wait for the next chapters (especially the gold zone design/loading), please keep up the good work and many thanks!!! cheers!!

[@endjin]

Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU

[@anders1717]

Hi finally a no frills explanation of an end to end solution in Fabric thank you! Can you also share the architectural diagram (Visio) you showed us in an earlier video? I'm also looking forward to the next video :)

[@endjin]

Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU

[@clyderodrigo9183]

Great series of vids! Keep up the good work. Thank you! Awaiting the next video :)

[@endjin]

Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU

[@richprice5434]

Brilliant video perfect for what I need to do subscribed 🎉

[@endjin]

That's great to hear! There are a few more videos dropping soon.

[@endjin]

Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU

[@Tony-cc8ci]

Hi, have enjoyed the Microsoft Fabric playlist so far, very informative. This one was 2 months ago, so just wanted to find out if you plan on continuing the series like you mentioned with the helper notebooks?

[@endjin]

Yes, very much so! Ed (and Barry) are just heads down busy preparing for their workshops and talks at the SQLBits conference next week. As soon as that's done they'll have more capacity to finish off this series of videos.

[@endjin]

Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU

[@gabrielepagani763]

Hi Ed, thank you for the great and very useful content ! If it's possible... Could you also share the code of the notebooks ?

[@endjin]

I believe the plan is to release some assets once he's finished the series.

[@endjin]

Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU

[@john_britto_10x]

In your architecture diagram, you have mentioned HPA orchestrator pipeline. Is that a separate pipeline that needs to be created than the ones at the process layers below?

[@endjin]

Thanks for the comment! Yes, the orchestrator pipeline is a separate pipeline that includes the configuration that controls the flow of the pipeline. I'll be showing that in an upcoming video, so stay tuned!

[@john_britto_10x]

@@endjinawesome. Awaiting for it.

[@rh.m6660]

Not bad, not bad at all. Could you point to a best practice framework for this type of work? Or is this just the way you like to work? I'm finding i can do to the data what i want but am not sure if im being efficient and scalable. I like the idea of using helpernotebooks, i believe cusom spark environments also support custom library's now.

[@endjin]

This is the way we like to work based on a background of software engineering within the company. We believe data solutions should get the same treatment as software solutions, and for that reason we're huge proponents of applying DataOps practices when building data products. In fact, we've presented at Big Data LDN and SQLBits on this very topic: endjin.com/news/sqlbits-2024-workshop-dataops-how-to-deliver-data-faster-and-better-with-microsoft-cloud Sadly we don't have a specific recording or blog we can share with you (yet!), but part of the point of this series is to expose you to some of our guidance in this area. But stay tuned! Check out "Part 8 - Good Notebook Development Practices" which is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU --- And you're right that Environments in Fabric support custom libraries. This is another way you can package up your common code and more easily use it across workloads. But this increases the CICD complexity a little bit, so take that into account! Ed

[@MdRahman-wl6qi]

how to move data from one lakehouse's to another lakehouse table using pyspark?

[@Ben-p4p]

Connect to both lakehouses on the left pane of the notebook. Use spark sql to select and manipulate the data from your bronzelake, then essentially do the merge similar to what is done in the video while selecting the silverlake.

[@endjin]

Further to Ben's response, I would recommend using two-part naming (.) when transferring this data. However, if you're just wanting to expose the data from one lakehouse in another lakehouse, remember you can avoid copying data altogether by using Shortcuts. And if you need to do this programmatically, there's an API for this: learn.microsoft.com/en-us/rest/api/fabric/core/onelake-shortcuts/create-shortcut?tabs=HTTP#create-shortcut-one-lake-target-example Ed

[@john_britto_10x]

endjin have you published the 8th part video, if so the link please?

[@endjin]

It's coming in the next week or so. We've just started a new series you might be interested in kzbin.info/www/bejne/q5K1loN-mtaph7c

[@john_britto_10x]

Thank you so much for replying quickly.

[@endjin]

Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU