Recently on a project we created 2 data factories for a customer and needed to provide an insight into the current and previous runs of the data factories in Acceptance and Production,
We could have given reader access but this would mean that they need to login to the azure portal and open up Data Factory and they would not get a nice report style overview. For the end user a PowerBi report is a lot more user friendly and then a Service Principal only requires reader access to the data factories to retrieve the information.
I pull the information directly from the Data Factory and only 30 days of history is currently stored.
You can however adapt what i have built for pointing towards the log analytics workspace for a longer history.
In my example I have left both Data Factory pages in, You can easily copy and paste a new page and the logic from Data Factory Page 1 or 2 will also need to be adapted.
Getting Started:
The report looks like this, DTF was not my choice but does make me giggle :)
I will first explain what the components are on the report and what is being visually shown before advising how to set it up. I think a lot of the report is quite self explanatory.
Please also see the refresh dataset button which I integrated power automate so users can refresh the data as they wish without permissions to the dataset.
Lets start at the top:
- Total Runs: The total runs of all pipelines in the period selected, currently show the last 30 days.
- Latest runEnd: The last pipeline run finished time.
- Latest runStart: The most recent start time of a pipeline
- Last Duration in Hours, Minutes and Seconds. Most recent duration time.
All that you is is filtered when you click a pipeline run or a pipeline by status etc to get further information.
Completion Status and Duration
You can view the pipelines in the data factory and this summarizes each one, the status adapts also if there is a cancelled run. It gives a very good overview of how many times the pipelines fail or runtime.
The data time is not shown in this table but when you select the run it filters the rest of the report to show all the information you require.
Message
This only shows information when a pipeline fails so you can see the exact error message from within the pipeline. a very nice addition for quick troubleshooting.
Overall Completion Status
Within this section you can easily get a good overview of the runs and each pipeline which has ran in the last 30 days.
Run Count by Status
a very easy to digest bar chart which shows all the different status's that have occurred in the last 30 days. this also makes it really easy to filter and find the failed or cancelled pipelines and the messages that occurred during that run.
I will do another post shortly including this dashboard for anyone to use and how to set it up. it takes minutes!
Comments
Post a Comment