Microsoft Fabric: Processing Bronze to Silver using Fabric Notebooks

  Рет қаралды 6,940

endjin

endjin

Күн бұрын

Пікірлер: 29
@endjin
@endjin 7 ай бұрын
Thank you for watching, if you enjoyed this episode, please hit like 👍subscribe, and turn notifications on 🔔it helps us more than you know. 🙏
@carlnascnyc
@carlnascnyc 9 ай бұрын
this is indeed a great series, the best one of all youtube imo, it's direct to the point and straightforward, just the right amount of complexity, can't wait for the next chapters (especially the gold zone design/loading), please keep up the good work and many thanks!!! cheers!!
@endjin
@endjin 7 ай бұрын
Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU
@sk8_boarding
@sk8_boarding Ай бұрын
Man this is crazy..will have a huge project starting beginning next year and to see how straight forward this is with fabric just gets me excited to start. First time we can provide also all operating systems with data from the bronze layer. The organised storage of data begins already on raw file level and if we doing it right we can control and expand the input quality for other departments without delivering it from a datawarehouse model or report. thx for that content!
@anders1717
@anders1717 10 ай бұрын
Hi finally a no frills explanation of an end to end solution in Fabric thank you! Can you also share the architectural diagram (Visio) you showed us in an earlier video? I'm also looking forward to the next video :)
@endjin
@endjin 7 ай бұрын
Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU
@Tony-cc8ci
@Tony-cc8ci 9 ай бұрын
Hi, have enjoyed the Microsoft Fabric playlist so far, very informative. This one was 2 months ago, so just wanted to find out if you plan on continuing the series like you mentioned with the helper notebooks?
@endjin
@endjin 9 ай бұрын
Yes, very much so! Ed (and Barry) are just heads down busy preparing for their workshops and talks at the SQLBits conference next week. As soon as that's done they'll have more capacity to finish off this series of videos.
@endjin
@endjin 7 ай бұрын
Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU
@clyderodrigo9183
@clyderodrigo9183 11 ай бұрын
Great series of vids! Keep up the good work. Thank you! Awaiting the next video :)
@endjin
@endjin 7 ай бұрын
Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU
@richprice5434
@richprice5434 10 ай бұрын
Brilliant video perfect for what I need to do subscribed 🎉
@endjin
@endjin 10 ай бұрын
That's great to hear! There are a few more videos dropping soon.
@endjin
@endjin 7 ай бұрын
Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU
@gabrielepagani763
@gabrielepagani763 Жыл бұрын
Hi Ed, thank you for the great and very useful content ! If it's possible... Could you also share the code of the notebooks ?
@endjin
@endjin 8 ай бұрын
I believe the plan is to release some assets once he's finished the series.
@endjin
@endjin 7 ай бұрын
Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU
@john_britto_10x
@john_britto_10x 8 ай бұрын
In your architecture diagram, you have mentioned HPA orchestrator pipeline. Is that a separate pipeline that needs to be created than the ones at the process layers below?
@endjin
@endjin 7 ай бұрын
Thanks for the comment! Yes, the orchestrator pipeline is a separate pipeline that includes the configuration that controls the flow of the pipeline. I'll be showing that in an upcoming video, so stay tuned!
@john_britto_10x
@john_britto_10x 7 ай бұрын
@@endjinawesome. Awaiting for it.
@rh.m6660
@rh.m6660 8 ай бұрын
Not bad, not bad at all. Could you point to a best practice framework for this type of work? Or is this just the way you like to work? I'm finding i can do to the data what i want but am not sure if im being efficient and scalable. I like the idea of using helpernotebooks, i believe cusom spark environments also support custom library's now.
@endjin
@endjin 7 ай бұрын
This is the way we like to work based on a background of software engineering within the company. We believe data solutions should get the same treatment as software solutions, and for that reason we're huge proponents of applying DataOps practices when building data products. In fact, we've presented at Big Data LDN and SQLBits on this very topic: endjin.com/news/sqlbits-2024-workshop-dataops-how-to-deliver-data-faster-and-better-with-microsoft-cloud Sadly we don't have a specific recording or blog we can share with you (yet!), but part of the point of this series is to expose you to some of our guidance in this area. But stay tuned! Check out "Part 8 - Good Notebook Development Practices" which is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU --- And you're right that Environments in Fabric support custom libraries. This is another way you can package up your common code and more easily use it across workloads. But this increases the CICD complexity a little bit, so take that into account! Ed
@MdRahman-wl6qi
@MdRahman-wl6qi 5 ай бұрын
how to move data from one lakehouse's to another lakehouse table using pyspark?
@Ben-p4p
@Ben-p4p 4 ай бұрын
Connect to both lakehouses on the left pane of the notebook. Use spark sql to select and manipulate the data from your bronzelake, then essentially do the merge similar to what is done in the video while selecting the silverlake.
@endjin
@endjin 4 ай бұрын
Further to Ben's response, I would recommend using two-part naming (.) when transferring this data. However, if you're just wanting to expose the data from one lakehouse in another lakehouse, remember you can avoid copying data altogether by using Shortcuts. And if you need to do this programmatically, there's an API for this: learn.microsoft.com/en-us/rest/api/fabric/core/onelake-shortcuts/create-shortcut?tabs=HTTP#create-shortcut-one-lake-target-example Ed
@john_britto_10x
@john_britto_10x 8 ай бұрын
endjin have you published the 8th part video, if so the link please?
@endjin
@endjin 8 ай бұрын
It's coming in the next week or so. We've just started a new series you might be interested in kzbin.info/www/bejne/q5K1loN-mtaph7c
@john_britto_10x
@john_britto_10x 8 ай бұрын
Thank you so much for replying quickly.
@endjin
@endjin 7 ай бұрын
Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU
Try this prank with your friends 😂 @karina-kola
00:18
Andrey Grechka
Рет қаралды 9 МЛН
It works #beatbox #tiktok
00:34
BeatboxJCOP
Рет қаралды 41 МЛН
100% Free unlimited Voice-over with ChatGPT
8:45
Learn with LEESI
Рет қаралды 2,4 М.
Learn the Fundamentals of Microsoft Fabric in 38 minutes
38:00
Learn Microsoft Fabric with Will
Рет қаралды 210 М.
Мастер и Мандарины - Уральские Пельмени
1:34:39
Уральские Пельмени
Рет қаралды 449 М.
Organize a Fabric Lakehouse using Medallion Architecture Design
36:06
Kamil Data Geek - Azure explained
Рет қаралды 6 М.
Try this prank with your friends 😂 @karina-kola
00:18
Andrey Grechka
Рет қаралды 9 МЛН