We are now at the end of the second Round of the BTB benchmarking study and there are a number of metrics in the study that are either completely unused or with 1-2 users only. The list of metrics continues to look large, but a percentage is clearly not of value to participants:

• When should we remove the 'dead' metrics - after 1, 2, or 3 rounds of lack of use?
• How should we remove them and what consultation process should take place?
• Where do we put them - is there an archive area that we can use, so these metrics can be brought back if circumstances change (without re-inventing them)?

We need some policies and procedures to manage this activity, and I would advocate removal after 2 rounds of 'unuse', following a consultation round, and archiving for possible re-use. However I'd be interested in other views from across the team.

 

I am of the same opinion, 2 rounds of unused metric (i.e. no data entry by any participant) would result in a consultation session before archiving it. I would suggest a consultation session occure 3 to 4 weeks after a round closes. Several service offerings may be discussed per session. Once the process is well established, we can vote on whether or not to make the consultation process live or via email.

I think that metric's should stay as as long as we have nothing elso to replace them with, but, we need to understand how old the metric's are. In many cases metrics will be captured and then not updated for some time either due to unavalability of new metrics or lack of willingness to share. So old metrics are better than no metrics.

This would also make the policy for updating simpler, if no change leave it as it is and let the user decide if based on the date the metric is still valid, is a new metric then review based on defined acceptability criteria. Par of the value of metrics is to be able to see the history of the metric as this gives an indication of the trend pattern for the metric. This is also of value. So we should maintain an archive of metrics that can be used for this purpose.

Sometimes the situation is that the community may not be ready for a metric, or the situation that makes the metric really critical may not have arisen recently. Therefore keeping good metrics untilyou need them is a very good idea. This is the concept behind the TM Forum's evolution to the Metric Inventory. We found we were truning down many excellent metrics in a push to make the studies as simple as possible.

Now withe the Metric Inventory approach, good metrics reside in the inventory and are pulled out and used for studies as needed. The studies themselves have become smaller and more focused with no loss of good metrics.

ToniaG:

Sometimes the situation is that the community may not be ready for a metric, or the situation that makes the metric really critical may not have arisen recently. Therefore keeping good metrics untilyou need them is a very good idea. This is the concept behind the TM Forum's evolution to the Metric Inventory. We found we were truning down many excellent metrics in a push to make the studies as simple as possible.

Now withe the Metric Inventory approach, good metrics reside in the inventory and are pulled out and used for studies as needed. The studies themselves have become smaller and more focused with no loss of good metrics.

In this context I assume we mean the actual way in which the metric is calculated rather than its current level or the benchmark level if it's from an industry study.

One thing that is worth considering here is whether or not, if a metric is perceived to be of value, it needs to be tracked regardless of present use in order that it already has a wealth of data behind it such that it can be more effectively used in future? Furthermore, this also lends itself to identify influencing factors.