Data Engineering Observability in Microsoft Fabric!

In part 5 of this course Barry Smart, Director of Data and AI, walks through a demo showing how to improve observability of your data engineering processes using readily available technology and platforms in with Microsoft Fabric and Azure.
Barry begins the video by explaining that one of the pitfalls of using Fabric Notebooks in an operational setting is that it can impede observability of what is going on within the notebook. To address this he shows how standard Python Logging, combined with Azure Application Insights can be used to emit telemetry from key points in your notebook and captured centrally. We demonstrate this approach with a create_logger function, showcasing its application in a Fabric notebook and the benefits of centralizing logs.
This episode helps ensure continuous monitoring and rapid issue detection for optimal data product performance. Application Insights then provides to the tools ot enable you to proactively monitor the behaviour of your entire data estate to sustain value over the long term. It also provides a valuable source of troubleshooting information when things do go wrong to help you quickly diagnose and resolve issues.
Stay tuned for our next episode on deploying resources through infrastructure as code.
- Part 1 - Overview • Microsoft Fabric: Mach...
- Part 2 - Data Validation with Great Expectations - • Microsoft Fabric: Mach...
- Part 3 - Testing Notebooks • Testing Notebooks with...
- Part 4 - Visualise your Medallion Architecture with Task Flows • Visualise your Medalli...
- Part 5 - This video!
#microsoftfabric #observability #dataengineering #medallionarchitecture #lakehouse #notebooks #powerbi #applicationinsights

[@endjin]

Thank you for watching, if you enjoyed this episode, please hit like 👍subscribe, and turn notifications 🔔 on it helps let the KZbin algorithm know that our content is worth watching!

[@applicitaaccount1258]

A really valuable video to help with an often overlooked part of using notebooks!

[@endjin]

We're really glad you found it useful!