Coming clean on dirty data

It's pretty certain that most CSPs are mindful that their data is neither clean nor consistent across their operations. Anyone embarking in any form of transformation project finds data cleansing the first major hurdle. It is therefore good to see that Subex and Heavy Reading conducted some reasonably in-depth research into the problem by interviewing 95 operators worldwide.

Conducting research on probably what Subex, and most other vendors of RA and data cleansing technology already know, does highlight just how serious the problem is. After all, they can't exactly publicize their customer's issues, can they? Analysys Mason also conducted research earlier this year and determined the market for Data Integrity Management, the fancy name for cleaning up 'dirty' data, will reach US$137M by 2014 from US$98M in 2010.

For many years the preferred or default method of measuring data quality and  consistency was a series of manual ad hoc audits. With the introduction of revenue assurance processes and automated tools, CSPs became exposed to data anomalies and process shortfalls they had not expected. RA vendors grasped this information to repackage and enhance their products in order to capture part of the extended data integrity market, Subex with its aptly named ROC Data Integrity Management solution.

Part of any RA discovery is to determine that order to bill processes are complete and accurate and anybody that has been involved in one of these exercises will tell you that the order process, often only partly automated, is an area of prime concern. That was also one of the key findings of the report. It found that six out of 10 operators (of the 95 that were interviewed) experienced up to 30 percent order fallouts. Of those, 40 per cent admitted that over 10 per cent of the fallouts were due to inaccurate data.

Approximately 50 per cent of operators still use ad hoc manual audits to keep inventory data in sync with the network. The message must be sinking in because the survey found that a majority of operators plan to invest in discovery and reconciliation solutions within the next 12 months. Chief amongst the reasons for this is a large discrepancy between an operator’s inventory data and what’s actually on their network. The survey estimates that 40 per cent of operators have 20 per cent of their inventory out of synch with their network. That's a serious issue when you consider the cost of network elements like switches and routers and way too much capex investment to leave sitting there as 'stranded assets'.

As European Communications commented, aside from the obvious savings and improvements to customer service, there is a wider shift that sees data as perhaps the most important commodity in a connected world. What use is a heavy investment in data analytics and business intelligence gathering if the core data is rubbish?

Aside from the obvious savings and improvements to customer service, there is a wider shift that sees data as perhaps the most important commodity in a connected world. Operators have an opportunity to play a central role, but they will only be as strong as their data is clean.

The really surprising thing about dirty data, apart from the time it has taken for someone to research it adequately, is that every CSP knows it us an issue. Perhaps denial is part of our cultural DNA, just as the first RA proponents discovered. Time to get those heads out of the sand? Most definitely!