A Dutch Service Provider develops and manages a large number of applications for several customers, mainly in the Netherlands. Together with one of its largest customers, located in the floriculture sector, there was a need to measure the quality of the applications. Additionally they wanted to objectively determine the performance of the DevOps teams delivered by the Service Provider and to compare this performance with the market.
IDC Metri executed its Agile Value Management service on two of the applications maintained by the Service Provider. Based on the supplied code, a system-level assessment was carried out, in which not only was the code analyzed, but also issues in the architecture and the “third-party” components were looked at. IDC Metri uses the worldwide leading technology CAST Application Intelligence Platform (AIP), which measures the software at the system level against more than 1,800 rules and best practices recommended by organizations such as ISO, CISQ, SEI, NIST, OWASP, OMG, and Mitre.
In addition to the technical quality, the functional scope delivered has also been determined in automated function points (AFP) and enhancement function points (EFP). IDC Metri also collected the hours spent, costs, defects, and other project characteristics, making it possible to determine the agile team performance metrics: Productivity, Cost Efficiency, Delivery Speed, and Project Quality.
The Service Provider and the customer have gained complete and fact-based insight into the quality of the applications and the delivered performance of the teams, while also having access to the market averages for comparable teams. This includes the following metrics:
- Total Quality Index (TQI) – the overall quality of the application
- Robustness – measures the risk of incorrect operation and/or incidents
- Security – measures the risk of security breaches
- Efficiency – measures the risk of inefficient use of resources such as databases
- Changeability – measures how quickly a change can be made or an incident can be resolved
- Transferability – measures how quickly a new team member gets up to speed
- Critical Violations – the “never-to-do” in code, measured against all standards and best practices
- Technical Debt – the cost of correcting all major issues in the application
- Function points – the functional scope of the applications (AFP) and the functionality provided in the sprints that have been measured (function points added, changed and removed – EFP)
- Productivity: Hours spent per EFP
- Cost Efficiency: € spent per EFP
- Delivery Speed: EFP per calendar month
- Project Quality: Defects found per EFP
IDC Metri delivered a Management dashboard, an Engineering dashboard and a Team Performance dashboard.
In the Engineering Dashboard, the developers can see at code line level which “critical violations” have been found, why these are violations and how they can be resolved. In this dashboard, IDC Metri has also proposed a number of actions in the Action Plan. Carrying out these actions ensures that the quality is improved with little effort.
After more measurements at the portfolio and application level, the management dashboard shows trends in metrics such as the aforementioned health factors, TQI, the number of critical violations and technical debt. Progress with regard to the Action Plan is also tracked.
The Team Performance dashboard shows the trends in team performance, where the teams are compared between each other and with the market. The outcome of the research is input for the Service Provider management and its customer to decide about the steering mechanism to be set up for application quality and team performance.