Thank you for watching, if you enjoyed this episode, please hit like 👍subscribe, and turn notifications on 🔔it helps us more than you know. 🙏
@carlnascnyc9 ай бұрын
this is indeed a great series, the best one of all youtube imo, it's direct to the point and straightforward, just the right amount of complexity, can't wait for the next chapters (especially the gold zone design/loading), please keep up the good work and many thanks!!! cheers!!
@endjin7 ай бұрын
Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU
@sk8_boardingАй бұрын
Man this is crazy..will have a huge project starting beginning next year and to see how straight forward this is with fabric just gets me excited to start. First time we can provide also all operating systems with data from the bronze layer. The organised storage of data begins already on raw file level and if we doing it right we can control and expand the input quality for other departments without delivering it from a datawarehouse model or report. thx for that content!
@anders171710 ай бұрын
Hi finally a no frills explanation of an end to end solution in Fabric thank you! Can you also share the architectural diagram (Visio) you showed us in an earlier video? I'm also looking forward to the next video :)
@endjin7 ай бұрын
Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU
@Tony-cc8ci9 ай бұрын
Hi, have enjoyed the Microsoft Fabric playlist so far, very informative. This one was 2 months ago, so just wanted to find out if you plan on continuing the series like you mentioned with the helper notebooks?
@endjin9 ай бұрын
Yes, very much so! Ed (and Barry) are just heads down busy preparing for their workshops and talks at the SQLBits conference next week. As soon as that's done they'll have more capacity to finish off this series of videos.
@endjin7 ай бұрын
Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU
@clyderodrigo918311 ай бұрын
Great series of vids! Keep up the good work. Thank you! Awaiting the next video :)
@endjin7 ай бұрын
Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU
@richprice543410 ай бұрын
Brilliant video perfect for what I need to do subscribed 🎉
@endjin10 ай бұрын
That's great to hear! There are a few more videos dropping soon.
@endjin7 ай бұрын
Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU
@gabrielepagani763 Жыл бұрын
Hi Ed, thank you for the great and very useful content ! If it's possible... Could you also share the code of the notebooks ?
@endjin8 ай бұрын
I believe the plan is to release some assets once he's finished the series.
@endjin7 ай бұрын
Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU
@john_britto_10x8 ай бұрын
In your architecture diagram, you have mentioned HPA orchestrator pipeline. Is that a separate pipeline that needs to be created than the ones at the process layers below?
@endjin7 ай бұрын
Thanks for the comment! Yes, the orchestrator pipeline is a separate pipeline that includes the configuration that controls the flow of the pipeline. I'll be showing that in an upcoming video, so stay tuned!
@john_britto_10x7 ай бұрын
@@endjinawesome. Awaiting for it.
@rh.m66608 ай бұрын
Not bad, not bad at all. Could you point to a best practice framework for this type of work? Or is this just the way you like to work? I'm finding i can do to the data what i want but am not sure if im being efficient and scalable. I like the idea of using helpernotebooks, i believe cusom spark environments also support custom library's now.
@endjin7 ай бұрын
This is the way we like to work based on a background of software engineering within the company. We believe data solutions should get the same treatment as software solutions, and for that reason we're huge proponents of applying DataOps practices when building data products. In fact, we've presented at Big Data LDN and SQLBits on this very topic: endjin.com/news/sqlbits-2024-workshop-dataops-how-to-deliver-data-faster-and-better-with-microsoft-cloud Sadly we don't have a specific recording or blog we can share with you (yet!), but part of the point of this series is to expose you to some of our guidance in this area. But stay tuned! Check out "Part 8 - Good Notebook Development Practices" which is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU --- And you're right that Environments in Fabric support custom libraries. This is another way you can package up your common code and more easily use it across workloads. But this increases the CICD complexity a little bit, so take that into account! Ed
@MdRahman-wl6qi5 ай бұрын
how to move data from one lakehouse's to another lakehouse table using pyspark?
@Ben-p4p4 ай бұрын
Connect to both lakehouses on the left pane of the notebook. Use spark sql to select and manipulate the data from your bronzelake, then essentially do the merge similar to what is done in the video while selecting the silverlake.
@endjin4 ай бұрын
Further to Ben's response, I would recommend using two-part naming (.) when transferring this data. However, if you're just wanting to expose the data from one lakehouse in another lakehouse, remember you can avoid copying data altogether by using Shortcuts. And if you need to do this programmatically, there's an API for this: learn.microsoft.com/en-us/rest/api/fabric/core/onelake-shortcuts/create-shortcut?tabs=HTTP#create-shortcut-one-lake-target-example Ed
@john_britto_10x8 ай бұрын
endjin have you published the 8th part video, if so the link please?
@endjin8 ай бұрын
It's coming in the next week or so. We've just started a new series you might be interested in kzbin.info/www/bejne/q5K1loN-mtaph7c
@john_britto_10x8 ай бұрын
Thank you so much for replying quickly.
@endjin7 ай бұрын
Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU