Metric Best Practices

: Public Package

Created:	5/10/2022 12:15:43 PM
Modified:	6/7/2022 8:51:13 PM

Project:
Author:	Natha Paquette
Version:	1.0
Phase:	1.0
Status:	Proposed
Complexity:	Easy
Difficulty:
Priority:
Multiplicity:
Advanced:
UUID:	{CE1F277D-49DE-4791-917B-8526641E99E1}
Appears In:

Metric Best Practices Case 1 – TM Forum GB935 Business Metrics Basic Metric Concepts The business metrics that have been developed and housed in the Business Metrics Scorecard represent areas of business operation that are important in assessing business performance, customer satisfaction/loyalty and operational efficiency. Any metric managed by the BMS team will be called a business metric. A business metric must: <ul>
<li>Be intelligible to anyone familiar with high-level telecom business concepts; i.e. a business metric does not require specialist or technical knowledge</li><li>Be appropriate to include in a business scorecard, i.e. a set of enterprise-level performance indicators of the business</li><li>Be part of a select minority of all metrics; i.e. it is not likely that all measurable things are business metrics.</li></ul>
According to the US National Institute of Standards and Technology, “a measure is for more concrete or objective attributes and metric for more abstract, higher-level, or somewhat subjective attributes”. A measure is a more general concept than a metric. Any quantity can be measured, but a good business metric also has a clearly understood preferred progression, a numerical directionality which indicates how we’d like to see the metric change as the business improves. Preferred progression can come in many different flavors, like: <ul>
<li> The higher, the better (like revenue)</li><li> The lower, the better (like customer response time)</li><li> The closer to a specific value, the better (like body temperature)</li><li> Can be equal to or lower than a value, but must not exceed (like a speed limit)</li><li> Can be equal to or higher than a specific value, but must not be lower (like bank account balance).</li></ul>
A metric is more likely to be some sort of comparison between two values, often expressed as a ratio, less frequently as a difference. But still, not all metrics need be comparisons. So, for example, one can have a metric which is the number of days it takes to go from bill cycle close to bills issued. However, even this simple count does have directionality – since the shorter this delay, the better. A metric begins with the thing being measured. The measurand is the “entity quantified by the measure”. Important kinds of measurands in the BMS work include: <ul>
<li>number of subjects, or more specifically:</li><li>number of things (like # bills)</li><li>number of events (like # customer contacts)</li><li>number of people (like # customers)</li><li>monetary value of subjects</li><li>monetary value of things (like $ value of bills)</li><li>monetary value of processes or capabilities (like $ cost of assurance)</li><li>time durations</li><li>time durations for work effort (like # hours for all order processing)</li><li>time durations between events (like # hours between customer requested date and customer committed date).</li></ul>
A benchmark generally suggests that a number or set of numbers is available for the metric that is indicative of either an average or a target for the industry. While the BMS team is responsible for defining business metrics, the Benchmarking team is responsible for running benchmarking studies using these metrics, which generates reports containing benchmark numbers, both average and 80-percentile values, among others. A dimension is a breakdown scheme for a metric. For example, a metric like “Revenue” can be dimensioned by region, by service, by market segment, etc. or indeed by any combination of these. Ideally, a dimension should not be part of a metric name, since one can instantiate many metrics from one metric by running through all combinations of dimensions – and that just leads to clutter. Note that Frameworx business metrics are service offering (SO) agnostic, and as such can be dimensioned by any appropriate service. That means that the same metric (F-CE-2a Mean Duration to Fulfill) would be used to study service fulfillment time for residential broadband, mobile services, or IP VPNs. Obviously, the data collected would be quite different. Thus, it is generally not appropriate, for example, to compare a fulfillment metric for mobile service against a fulfillment benchmark for DSL service. Business Metric Naming A given metric might be expressed in multiple equivalent ways. For example, one could write “A as a % of B” or “% A of B” or “A/B %”. All this variety can lead to the same metric erroneously entered multiple times in different namings, or else might make it difficult to find a metric given different namings. So, in order to ensure consistency as well as concision in the naming of metrics, a set of naming types was used to standardize metrics, detailed below. What principles guided these metric name types? A good metric name should provide the gist of the metric without having to resort to looking up further detail. For example, “# hours to fulfill service order” is better than “time to fulfill service order” since we do not need to look up the units. Also, “# hours” is self-explanatory, so we do not need to write “Number of hours”. Also, business objects and their states have been standardized as much as possible; so, we use “bills” instead of “invoices”, for example. Lists of standardize business objects, value drivers, and other expressions are presented later in this chapter. Some ratios are ratios of the same thing, where some subset based on attribute is in the numerator. This can be written very compactly. For example, “% bills issued electronically” means “% of bills that were issued electronically out of total bills”. A percentage of two different things in the same units can be written as “% opex, of revenue”, where the comma and “of” are used to indicate what the percent is based on. In the most general case, we have two different things in a ratio, like cost of fulfillment per employee. Such ratios are generally not written as percent. Prose metric names are written in title case. In addition to a Prose Name, a metric also has an “XLS Name”. This is driven by the need to have clear formulas for derived metrics. So, while older formulas exist in Prose Formula form, we have made sure that every formula can also be expressed an XLS Formula using the XLS Names.XLS Names are written in lowercase using the syntax for named ranges in Microsoft Excel1, or in other spreadsheet products that adopt the same naming conventions. To summarize, metric naming in BMS FX13 has been driven by several needs: <ul>
<li>To keep the name as short as possible.</li><li>To keep the name meaningful enough so that we reduce the need to reference the more complete definition in the documentation</li><li>To keep the structure of the names consistent</li><li>To keep the terms in the name consistent (see Business Object Taxonomy for BMS Measurand)</li></ul>
Below are the principle metric naming types used in Fx13: Type 1: Count – Basic <ul>
<li>Format: # #lt;business-objects#gt;</li><li>Example: # Bills</li><li>Format: # #lt;business-objects#gt;#lt;qualification#gt;</li><li>Example: # Bills Issued</li></ul>
<ul>
<li>Note: It is important to follow the metric format exactly. Thus, it is “# Bills”, not “# of Bills”, not “# Bills Total”, not “Number of Bills”, etc.</li></ul>
Type 2: Valuation <ul>
<li>Since currency is a count of dollars, it is similar to a count, except that the business object is not countable, but rather a financial concept that can be “evaluated”.</li><li>Format: $ #lt;financial-object#gt;</li><li>Example: $ Revenue</li></ul>
<ul>
<li>Note: A native financial concept (like cost or revenue) is easily understood to be evaluable. But if the item being evaluated is not a native financial concept, like the money value of bills issued, we precede the object with “value of”. Thus bills, which can either be counted or evaluated, require the expression “$ Value of Bills” to clearly indicate we mean the dollar value and not the count of bills.</li></ul>
Type 3: Ratio – Simple <ul>
<li>Format: % #lt;business-object#gt;#lt;qualification#gt;</li><li>Implication: % #lt;business-object#gt;#lt;qualification#gt; / #lt;business-object#gt; Total</li><li>Example: % Activation Failed </li><li>Implication: % Activations Failed / Activations Total</li></ul>
<ul>
<li>Note: This is the simplest kind of ratio to express since the same business object is referenced in the numerator and the denominator. Thus a % can be used to denote this type.</li></ul>
Type 4: Ratio – Common Units <ul>
<li>Format: % #lt;business-objects-1#gt; , of #lt;business-objects-2#gt;</li><li>Implication: % #lt;business-object#gt;#lt;qualification#gt; / #lt;business-object#gt;</li><li>Example: % Cost of Fulfillment, of Opex </li><li>Implication: % Cost of Fulfillment / Opex</li></ul>
<ul>
<li>Note: This is a ratio of two different business objects, but expressed in the same units. Because the units are the same, a % will also be used to denote this type.</li></ul>
Type 5: Ratio – General <ul>
<li>Format: # #lt;business-objects#gt; per #lt;business-object#gt;</li><li>Example: # Minutes per Service Problem Resolution </li></ul>
<ul>
<li>Note: This is the most general kind of ratio to express, since the numerator and denominator can be totally difference objects. The “per” is used to denote this type.</li></ul>
Type 6: Ratio – Over One <ul>
<li>Description: There are some BMS ratios that are really not ratios, but needed to be expressed as a numerator over denominator of 1 for historical modeling reasons. </li></ul>
Type 7: Special Name <ul>
<li>Description: Some of the metric names do not follow the above canonical rules. This occurs when a common industry name is preferred. Thus, rather than saying “% Customer Requests Resolved on First Call”, we prefer to use the common industry name “First Call Resolution (FCR)”. Since industry names exist independently of our naming convention, we denote these as Special Names.</li><li>Example: First Call Resolution (FCR).</li></ul>
Type 8: Suffix: Metric-Name-Type – By Dimension <ul>
<li>Description: Generally, a dimension should not be part of a metric definition intrinsically. Rather, the dimensions should be listed out in a separate field, like (Prose or XLS) Dimensions. But we permit this due to historical reasons: some of the metrics already in the Scorecard are pre-dimensioned. But going forward, we advocate that metric dimensions should be extracted from the definition proper, and kept in a Dimension field.</li><li>Format: #lt;metric-name-type#gt; , by #lt;dimension#gt; Type</li><li>Example: # Bills Issued, by Delivery Type</li></ul>
<ul>
<li>Note: A dimension phrase is always started with a comma and the word “by” and ends with the word “Type”. Other uses of the word “by” may exist, and these are not dimensioned metrics (like “% Problem Reports Resolved by Due Date”, which does not have the comma before “by”). Also note that a pre-dimensioned metric will have multiple values, distributed by the values of the dimension. So, the above example metric will have values for each of the different Delivery Types, like paper invoice, e-bill, etc.</li></ul>
Type 9: Suffix: Metric-Name-Type – By Dimension Value <ul>
<li>Description: A few metrics are explicitly dimensioned by just one-dimension value. For example, rather than metric by “Service Type”, it is a metric “By Voice Services” only. The construction is similar to the preceding, except there is no “Type” at the end.</li><li>Format: #lt;metric-name-type#gt; , by #lt;dimension-value#gt;</li><li>Example: % Revenue, by Voice Services</li></ul>
Other Patterns Note that other common patterns also recur within the text of the metric names: <ul>
<li>If a business object’s time between two states is being measured, expressions like “from #lt;state1#gt; to #lt;state2#gt;” appear, as in “# Hours for all Orders, from Ordering to Activation”.</li><li>Generic metric words that can be used everywhere (like Average, Mean or Total) are avoided. The word Mean is actually reserved for the benchmarking reports – as in the mean of a metric given that the sample data has come from many CSPs. Otherwise, without these conventions, we would end up having a “Mean of a Means”, which is a bit unwieldy.</li><li>To distinguish the time for one instance of a business object versus the aggregate time for all instances, the word “All” is used, as in “# Hours for All Orders” (an aggregate) versus “#Hours per Order” (an average or mean).</li></ul>
Business Object Taxonomy Many of the BMS metrics are about process, and as such, what we are measuring, the measurand, is process, or more broadly speaking, an occurrence. Some measurands are people or stakeholders (# customers, # FTE), while other are artifacts, tangible or informational (# problem reports, # data records). Of course, many are financial in nature ($ cost, $ revenue). A list of business architecture domains will help standardize these measurand categories, which will give us another way to organize, navigate, and analyze metrics. Measurands are substantives, or nouns in our business architecture depictions. Some people call them business objects. A taxonomy for measurands and other substantives will help limit and control the number of nouns (and ideally states and behaviors) that are used in the metric definitions. For this iteration, our focus has been on the substantives only. We are basing the top-level business architecture domains on the work of the OMG (Object Management Group) BASIG (Business Architecture Special Interest Group) – who have produced a domain model. The top-level measurand domains are: <ul>
<li>Capability</li></ul>
<ul>
<li>Stakeholder</li><li>Money</li><li>Artifact (a document, information record, or other business object, like a bill, a payment, a problem report, etc.)</li><li>Occurrence (an initiative, process, behavior or event, that takes place in time)</li><li>Resource (a tangible or intangible material used to deliver products/services to customers).</li></ul>
The list of measurands used in this release of the BMS metrics is as follows: <ul>
<li>Capability</li></ul>
 - assurance - repair (a subset of assurance) - repair and maintenance (a subset of assurance) - billing - channel (may overlap Marketing & Sales and CRM) - collections (a subset of billing) - customer management (referred to as CRM earlier) - fulfillment - sales (a subset of the Marketing & Sales capability discussed earlier). <ul>
<li>SLA management (a subset of assurance)</li></ul>
- Stakeholder - Customer Note: The term “customer” is preferred over “user” or “subscriber” - User Note: This term is retained for some special uses, such as its use in ARPU, where U stands for User - Employee (NOC FTE). <ul>
<li>Value</li></ul>
 - account receivables - asset - billing charge - bill - capex -- collectable debt written off -- cost -- operating income -- opex -- revenue -- sales. <ul>
<li>Artifact</li></ul>
 - bill - billing suspense file - data record - order - problem report - service problem - settlement report - SLA - XDR. <ul>
<li>Occurrence</li></ul>
- activation - bill processing fault - billing error - case of revenue recovery - customer call - customer contact - customer incident - customer payment - customer request - customer transaction - failure - fulfillment issue - installation - pricing change - service outage - SLA violation. <ul>
<li>Resource</li></ul>
- future infrastructure build - revenue system - service. Using this taxonomy, for example, the metric “Average Handle Time” has a measurand of “occurrence.customer_request”, since it represents the time to handle a customer request, which is a kind of business occurrence. The list of measurands will help keep the list of nouns used in metrics to a manageable level. Resource Performance Management Metrics Resource Performance Management metrics provides deep insights into the performance of the entire communication network resources and plays important role in assessing operational efficiency, evaluating customer’s experience and conducting operational troubleshooting. The scope of Metrics for Resource Performance management is very extensive as it includes a wide range of telecommunication domains, such as: Mobile, Fixed, Transmission, Cables, etc. and a large variety of technologies within each one of them. Equipment vendors may provide dozens to thousands of Metrics per a version of the equipment or per function. The list of provided Metrics per version is usually updated in each new version of the equipment. As a result, it is common for a Resource Performance Management system to be updated few times a year, either to keep up with frequent changes in Metrics or as a consequence of the evolving needs to monitor new equipment added. In modeling the world of resource performance metrics, a significant distinction to be made is between Metrics that are being measured (also known as Measurements) and Metrics that are being calculated by one of the OSS systems. Typically, a Measurement will be taken by a Network Element or a Probe. It will be then sent to a Management system (usually EMS, sometimes NMS) that may send it to another management system (most typically NMS). Each one of the management systems in the OSS layered architecture may add calculated Metrics to provide additional value and better insight of the network. Such calculated Metrics will be often termed as KPIs (Key Performance Indicators). When identifying a Resource Performance Metric, the scope of the Metric has to be determined by its dimensions. Network typical dimensions usually include: <ul>
<li>Domain/Main Technology: such as Mobile Core, Mobile RAN, Switching, Transport, etc.</li><li>Specific Technology: This may be a sub-category of the Domain/Main Technology. For example, for Mobile RAN domain, this may include: GSM, UMTS, LTE</li><li>Monitored Network Element/Function (Monitored Object Class level)</li><li>Additional technology specific dimensions</li></ul>
When a Resource Performance Metric is presented, a decision has to be made about the right level of the Metric as reflected in the possible values for the composing dimensions. Currently, different standard organizations defined standard Metrics for the different Domains/Main Technologies; therefore, it makes sense to separate the Metrics at this level. For Specific Technologies and Monitored Network Elements/Functions, these scoping decisions should be probably further examined. The following example may demonstrate the decisions that have to be taken when classifying a Metric. Let’s look at the “IP Throughput” Metric. We can certainly say the following: <ul>
<li>“IP Throughput” may be used for various network domains, so the option of defining it as a single multi-domain Metric may be raised. Yet, we will probably prefer to differentiate “IP Throughput” of a Transport equipment from the one of a Mobile E-Utran</li><li>A similar question can be raised about differentiating “IP Throughput” among various network elements of the same domain that use IP as their communication protocol.</li><li>For IP communication, it makes sense to differentiate between “Downlink IP Throughput” and “Uplink IP Throughput” so we may actually have another specific dimension in this case, based on the direction of the traffic.</li><li>Interestingly, “IP Throughput” can be calculated for specific “IP Links” or per groups of “IP Links” and not just for a network element object. Moreover, “IP Throughput” may be calculated for additional dimensions that are not even entirely network level such as: “Source Device/Application”, “Target Service/Application” and many more. These should be probably considered as different Metrics.</li></ul>
To continue this example, we may observe that we can look at average throughput, maximum Throughput, Minimum Throughput or a Peak Hour Throughput. Each one of these will be considered as a different Metric as it has a different meaning and may be used at a different context. We can therefore see that different ways of calculating a Metric may generate a different Metrics when the result is used in a different way and has a different meaning. In general, there are the basic properties that are common to all kinds of Metrics such as: Metric Name, Description, Metric Unit, etc. Some properties are specific to the way that Metrics that are Measurements are being measured. These may include a collection method and additional details on the collection. Calculated Metrics will have a set of properties that define the way a Metric is being calculated, a formula, its components, etc. A Resource Metric may be collected or calculated per different time granularities with a lot of diversity depending on their usage. For near real-time monitoring or for ongoing maintenance, the time window may start with a multi-second time window up to X Minutes, hourly or daily granularity, with most common 5 minutes or 15 minutes granularities. Metrics that are used for Resource planning may perform aggregative calculations that reach daily, weekly or monthly granularities. Resource Metrics can be classified by various ways. A useful way was introduced by ITU-T in E.800 document: Quality of telecommunication services: concepts, models, objectives and dependability planning – Terms and definitions related to the quality of telecommunication services. The document is presenting a classification of KPIS to Quality of Service categories. ITU suggested the following QoS characteristics: Speed Performance criterion that describes the time interval that is used to perform the function or the rate at which the function is performed. (The function may or may not be performed with the desired accuracy.) Accuracy Performance criterion that describes the degree of correctness with which the function is performed. (The function may or may not be performed with the desired speed.) Dependability Performance criterion that describes the degree of certainty (or surety) with which the function is performed regardless of speed or accuracy, but within a given observation interval. Availability Availability of an item to be in a state to perform a required function at a given instant of time or at any instant of time within a given time interval, assuming that the external resources, if required, are provided. Reliability The probability that an item can perform a required function under stated conditions for a given time interval. Simplicity Ease and lack of complexity in the benefit to the user of a function of the service. 3GPP has referred to the ITU adding some specific categories such as “Retainability”, “Mobility”, “Integrity” and “Accessibility” categorization (3GPP TS 21.905: “Vocabulary for 3GPP Specifications”.). Following are some examples for possible QoS classification: <ul>
<li>“Success Rate/Failure Rate” of sessions at different levels of protocol are considered as “Accessibility” Metrics</li><li>“Drop Rates” for Voice/Data/Video are considered as “Retainability” Metrics</li><li>“Latency”/”Delay” of messages are “Integrity” Metrics</li><li>“Mobile Handover Success Rate” is a Mobility Metric (This category is Mobile specific)</li><li>“Availability” (percentage) is an Availability Metric</li><li>“Throughput” is an “Integrity” Metric.</li></ul>
Additional categorizations other than by Quality of Service are of course possible. A distinction should be made between categories that define the Metric from descriptive categories that add details on the Metric. Characterization of Existing Metrics Models------------------------- 1http://office.microsoft.com/en-ca/excel-help/define-and-use-names-in-formulas-HA102749565.aspx#_Toc312073993