Welcome to your ultimate guide on using the Microsoft Fabric Monitoring Hub! This blog will help new users navigate and utilise the Monitoring Hub to keep track of various Fabric activities efficiently. Whether you're overseeing data pipelines or managing Spark jobs, this guide will provide step-by-step instructions to make your monitoring tasks seamless.
What is the Monitoring Hub?
Before diving into the details, let's briefly cover what the Monitoring Hub is, especially for any new BI/Analytics users reading this blog. The Monitoring Hub in Microsoft Fabric serves as the central location for tracking activities across various Fabric items you have access to, including:
Data Pipelines
Dataflows
Datamarts
Lakehouses
Notebooks
Semantic Models
Spark Job Definitions
How to Access the Monitoring Hub
To access the Monitoring Hub, simply select Monitoring Hub from the navigation pane (On the left of your screen) in Fabric. You will be presented with a table displaying recent activities, sorted by start time with the latest activities at the top.
Exploring the Key Features of the Monitoring Hub in Microsoft Fabric
The Monitoring Hub boasts a suite of impressive features designed to streamline the process of monitoring your solution. These features include:
Modifying the Display Order: Simply click on any column header to sort the table. The sorting order will be indicated by an arrow next to the column title.
Configuring Table Columns: Personalise your view by managing columns:
Add: Select desired columns from the ‘Column options’ list.
Pro Tip
I highly recommend adding the ‘Duration’ column to your default display. It’s incredibly useful to log in each morning and instantly gauge the performance of your solution. This addition can significantly enhance your ability to quickly identify any anomalies.
Adding the ‘Capacity’ column to your display can provide valuable insights, particularly if your solution utilises more than one capacity. This allows you to monitor the usage and availability of each capacity, enabling you to manage resources more effectively. It’s a simple addition that can make a big difference in your monitoring strategy!
Remove: Deselect unwanted columns from the ‘Column options’ list.
Rearrange: Drag and drop columns within the ‘Column options’ list to reorder them.
Applying Filters: Use the ‘Filter’ button to refine your view based on:
Status: Filter activities by their status.
Start Time: Choose a predefined or custom time period.
Item Type: Filter by specific Fabric items.
Submitted By: Filter activities by the item owner.
Location: Select which workspaces’ activities to view.
Taking Actions: Depending on your permissions, you can execute various actions on Fabric items directly from the Monitoring Hub. Use the ‘More options’ button (three dots) next to an activity to see the available actions (Open, Retry, View Detail, and Historical runs).
Pro Tip
If you encounter an issue with a data load that previously ran without problems, often the simplest solution is a straightforward re-run. There are numerous potential issues that could arise, ranging from timeouts and network connectivity issues to cloud provider downtime. As a general guideline, I always suggest attempting a reload at least once or twice before delving deeper into troubleshooting. This approach can save you a significant amount of time in the long run. Trust me, it’s a strategy worth adopting!
Viewing Historical Runs: Access up to 30 days of historical data for a specific item:
Select ‘More options’ next to the item.
Click on ‘Historical runs’.
To return, select ‘Back to main view’.
Viewing Details: To see detailed information about an activity, hover over its name and select the ‘View detail’ symbol.
Conclusion
The Microsoft Fabric Monitoring Hub is a powerful tool for overseeing your Fabric activities. By following this guide, you can effectively monitor and manage your data operations, ensuring a smooth workflow. However, there is room for improvement in several areas:
Detailed Error Tracking: It would be beneficial to see exactly where a failure occurred, not just what failed. Providing error details would be a great enhancement.
Tagging and Annotation: One feature I loved in native Azure Data Factory was the ability to tag and annotate pipelines for better filtering, drill-down, and monitoring. It would also be fantastic to set these values at runtime for dynamically changing pipelines.
Child Pipeline Visibility: Currently, the Monitoring Hub only displays parent pipelines, requiring multiple screens to locate a failed child pipeline. Adding visibility for all child pipelines within the Monitoring Hub would significantly improve usability.
What features do you think should be added to the Microsoft Fabric Monitoring Hub? Are there any pain points you’ve encountered or suggestions you have for enhancing its functionality? Please share your feedback in the comments below!