C. What data is collected automatically (including tool integration)
The only data collected automatically was the defects. A Microsoft Access based tool was used to record and track defects encountered on the project.
Everything else was recorded on paper and sometimes entered into an Excel spreadsheet or similar. There was little consistency in the format for recording data and the project administrator did not want to impose any standards on the project staff. The project staff themselves just wanted to get the project admin off their back.
D. What percentage of metrics are used by the team? Projects? Management?
The metrics collected by the project admin were used by the project manager to track progress against the schedule, and to ensure that the budget was OK. The total defects count was also provided to the project sponsor.
E. What amount of time is spent on preparing reports
The project manager spent 1 full day each week on the progress report, and the sponsor spent half a day on their report to the executive team.
F. How easy is aggregation of metrics/projects
Because each team used different formats, structure and definitions, aggregation of the data across the project was very time consuming for the project admin and project manager who constantly referred back to the team members for clarification.
G. How often do errors creep into KPI results/reports
As a results of the manual basis of recording and collecting data, errors did indeed creep into the schedule and budget. An example was the removal of a component of the budget for another project – due to an incorrect calculation of the costs incurred which was not picked up till near the end of the project.
H. Does everyone understand the difference between KPIs and metrics
Because so few metrics were collected, staff did not see any difference between KPIs and metrics. Indeed for them they were synonymous. The raw data collected were basically the same as the information presented in progress reports.
I. Are the KPIs/metrics reused in other reports/purposes
The report prepared by the project admin was reworked by the project manager into another report for the project sponsor. The sponsor then reworked the report they received into another report for the exec team.
J. Are the KPIs/metrics reused in future planning/estimating
Eventually the project was completed (over time, over budget and some functionality dropped due to lack of funds).
The final project closedown report prepared by the project manager was then sent to the project sponsor and select stakeholders. The entire project file system was then archived to DVD.
What was learnt?
- The metrics/KPIs collected were poorly defined (ambiguous to collect and report)
- The metrics did not represent a clear picture of the project progress
- Consistency of data collected should have been determined at project commencement
- Excessive time was spent collecting manual data
- Excessive time was spent preparing reports
- Reports should have been standardized
- Data validation should have been made through tool integration and use
- Project financial system and scheduling tool should have been made available to project staff
- Defects should have been categorized into critical, serious, minor and trends monitored
- Having a long term metrics database would enable lessons to be learnt and data reused in estimating future project work