Azure DevOps Dashboard

Challenge

While Azure DevOps does offer a dashboard, it is limited in the options users are allowed to query – e.g. each tile on the native dashboard must be set for a specific pipeline. Our DevOps Team needed a solution that would easily display data on our builds, release pipelines and agent usage.

Because we use many release pipeline stages across several different pipeline types, we needed a way to aggregate metrics across them. We also wanted to make the automation tests in our pipelines as efficient as possible, which meant we needed to gather metrics on problem tests and the length of time tests take to execute. Finally, we wanted to spend as little as possible on our pool of MS hosted agents – and to do this we needed to see how often all the agents in the pool are being utilized.

Dashboard walk-through

1. Build – a simple summary of the number of failed builds in the last 14 days.
2. Build Duration – the average time taken for our builds to complete. You may want multiple instances of this tile if you have builds that take different lengths of time to complete. Ours are fairly similar so it makes more sense for us to have a single tile to identify spikes in duration.
3. Build Failures – the number of failures in any release pipeline per day.
4. Build failure by stage name – a count of fails per release stage for the Azure DevOps Release pipeline.
5. Job queue – a count of the number of requests per day that end up in a queue.
6. MS Agent Usage – the total number of MS hosted agents that are in use on a given day. Our max number is 12, so if we see that number hit on multiple days, then we may want to increase the total number of agents in our pool.
7. E2ETesting failures – three tiles showing the count of fails for specific automation tests in the corresponding release pipeline stage.
8. Average E2E Testing Time – three tiles displaying the average run time for our automation stages.