Microsoft Fabric Notebooks - Showcase with advanced features

Welcome back to another episode of Fabric Espresso DE&DS series. In this episode, we introduce and explore the nuances of Fabric Notebooks, a web-based interactive surface that's rapidly becoming the go-to tool for data scientists and engineers for writing big data analytic jobs using Apache Spark.
Key points presented by Jene:
What are Fabric Notebooks? Microsoft Fabric notebook is a primary code item for developing Apache Spark jobs and machine learning experiments. It's a web-based interactive surface used by data scientists and data engineers to write code benefiting from rich visualizations and Markdown text.
Distinctive Features and Focus Area: Unlike its counterparts, Fabric Notebooks are tailored for developers working in data scenarios, providing a seamless experience for tasks like data ingestion, data preparation, data transformation, and data analytics.
Comparison to Synapse and Jupyter Notebooks: Compared to Synapse Notebooks, Fabric Notebooks are built on a SaaS platform, offering various SaaS-like features including low-code experience and enhanced collaborative features. They facilitate easier interaction with PowerBI report/dashboard.
Unlike Jupyter Notebooks, Fabric Notebooks are integrated tightly with the lakehouse and other Fabric key features and functionalities and are rich in spark specific features like high concurrency mode to share a spark session, spark job inline monitoring, spark job diagnostics, and built-in Microsoft Spark Utilities.
Advanced Features: The notebooks also support advanced features like IPywidgets and reference run which significantly enhance functionality, although at the time of this interview, there are some compatibility issues with IPywidgets which are being addressed.
🎙 Meet the Speakers:
👤 Guest from Microsoft Fabric Product Group: Jene Zhang, Senior Program Manager of Fabric Notebooks
Jene is a Senior Program Manager at Microsoft with a passion for developing advanced developer tools and AI technology. Currently, she is spearheading the creation of a specialized Spark notebook aimed at enabling data engineers and scientists to excel in their domains.
Linkedin: / edelweissno1
Twitter: / edelweissno1
👤 Host: Estera Kot, Senior Product Manager at Microsoft and a member of the Fabric Product Group. She holds the role of Product Owner for Apache Spark-based runtimes in Microsoft Fabric and Synapse Analytics. Estera is a Data & AI Architect and is passionate about computer science.
LinkedIn: / esterakot
Twitter: / estera_kot
👍 Like this video? Don't forget to hit the 'Like' button and share it with your network!
🔔 Stay Updated: For more insights into Microsoft Fabric Data Engineering and Data Science, and all things tech, make sure to subscribe to our channel and hit the notification bell so you never miss an episode!
#Microsoft #MicrosoftFabric #FabricNotebooks #DataScience #BigData #DataAnalytics #ApacheSpark #JupyterNotebook #SynapseNotebook #IPywidgets #TechInnovation #MicrosoftSparkUtilities

[@KurtBuhler]

FYI: Please timestamp videos > 5 min, it would make it much easier to consume long-form content esp when there's so many cool things like this :) Example: 01:05 Overview of Fabric Notebooks 02:08 Fabric Notebooks vs. Synapse Notebooks 04:04 Fabric Notebooks vs. Jupyter Notebooks 05:56 Feature: Lakehouse Explorer pane drag & drop 08:01 Feature: Notebook resources pane 09:00 Feature: Python module support 09:51 Feature: Notebook sharing with permissions control 10:36 Feature: Real-time co-editing 11:13 Feature: Commenting pane (coming: tagging in comments) 12:04 Feature: Immersive authoring (auto-save, code-assist, etc.) 14:28 Feature: Built-in code snippets 15:00 Feature: iPython kernel and widgets (interactive elements) 16:00 Feature: Magics (i.e. like macros; run child notebooks, %pip install, custom magic) 18:47 Feature: Table of contents 19:30 Feature: Visualization - display function 19:55 Feature: display fnxn - interactive chart view (coming soon: exploration feature) 21:03 Feature: display fnxn - summary attribute 22:23 Feature: Open source library visualization (i.e. matplotlib, seaborn, bokeh etc.) 23:28 Feature: Markdown support (drag & drop images from local PC to import) 25:11 Topics TBD in the next episode

[@benhooker6079]

Awesome

[@sudhirsharma9103]

Interesting fact

[@krish_telugu]

Do we need to manage compute resources for the workloads? Like creating clusters in Databricks or synapse

[@keen8five]

Hi Jene, concerning "build-in display function" (19:39): Currently the *table view does not allow selecting/copying values* from the displayed result. Will this be fixed with the mentioned improvements for GA (20:40)?

[@JeneZhang]

Thanks for the feedback! That a super valid scenario but unfortunately we won't have it by GA, because the "Select" state in the next version will have specific reactions (preview value, calculate summary, etc), but I'll record this request in our backlog, hope we can find some way to get it in next wave!

[@siyandasphesihlengcobo4623]

Please expand more on python modules when I try to use my modules after the drag and drop import I get an error if trying to get functions defined in the module.

[@7effrey]

The data wrangler was not demonstrated, very helpful for gaining insight into your data frame

[@EsteraKot]

Data Wrangler is here kzbin.info/www/bejne/Y5iZfKmbgLaLq5Y

[@VagnerKogikoskiJunior]

Can I use notebook for any python code or should it only be used with Spark?

[@ZubairMuhammad09]

how can we read a variable in Notebook that's set using SetActity in MS Fabric.

[@up_0078]

Can you share notebook please?

[@JeneZhang]

we'll publish a polished version in the "Use a sample" on the data engineering/data scientist workload homepage😊

[@pp56825]

Would be nice if dataframe will be uniqly named rather simple df, name of table or file as default will be nice, Also i could imagine configuring a simple naming pattern on lakehouse level, which could enfoce starnard.

[@JeneZhang]

Are you referring to the dataframe name that automatically generated from drag&drop?

[@pp56825]

@@JeneZhang Yes, that will speed up development, now users have to rename it. Specialy if they are going to use data wrangler, where they need meaningful names to identify df

[@JeneZhang]

Got it, that's a good suggestion! I'll see what I can do to make it smoother, thank you for the valuable feedback!😎@@pp56825