Data Migration

Status of the Data Migration Community Group

KennethLipnickey — Wed, 14 Sep 2011 17:36:11 GMT

Dear TM Forum Member,

Due to a lack of on-going activity, as of September 16, 2011, the Data Migration community group will be disabled and no longer publicly visible or accessible. Although the community group will no longer be accessible, we will archive and retain any information that is currently stored in the community group in the event it is needed in the future. This includes the project workspace, discussion areas and wikis.

We thank you for your support of TM Forum activities, and hope to see you participating in some of our other community groups in the future.

Regards,

Ken Lipnickey
Head of Project Management, Collaboration

Gartner November 2010 Magic Quadrant points to further consolidation in Data Integration

TonyS1 — Thu, 06 Jan 2011 17:10:29 GMT

We’re currently seeing much consolidation, and strong rumour of further consolidation, in the Data Integration (DI) market, with major players seeking to strengthen their position in MDM (Master Data Management), data integration and data migration. The recent Gartner Magic Quadrant highlights the importance placed on having a completeness of vision and an ability to execute. Oracle’s 2010 acquisition of GoldenGate and rumours of a bid for Informatica along with Talend’s acquisition of Sopera point to an industry that sees significant value in having the right tools for the job.

Gartner specifically calls out Data Migration and Synchronisation as key elements of the DI capability set, but interestingly omits Data Quality. They don’t strongly recognise (from our viewpoint) the emergence of cloud-based data management and the impact of software-as-a-service in this arena – for instance Jigsaw, the emerging cloud-based contact and client data management solution provider is not mentioned.

The risk for some traditional players in the market, such as the Systems Integrators (SIs), is that their labour intensive, bespoke approach to projects will increasingly count against them in a world of much greater sophistication and automation. Future oriented clients, particularly in the increasingly agile Communications, High Tech and Financial Services Sectors are looking more and more for solutions that bring control, performance and reliability. Those are elements that a bottom-up build often promises but rarely delivers.

There is also exposure for Software Vendors currently in the Data Integration sector if they are unable to rapidly bring comparative and competitive solutions to the table whilst their peers take the acquisition shortcuts.

Business models are clearly changing for Applications Vendors and Systems Integrators alike as Service Providers increasingly drive for greater agility and flexibility to meet competitive and financial pressures. The old world - where enterprises bought software licenses from the apps vendor and hired an SI to stand it up and manage the transition of business processes and population with transformed legacy data – is fast disappearing. Service Providers now ask that vendors (SI and Apps alike) take a much broader responsibility and include data integration and migration in the scope of their contract.

Vendors traditionally contracted at the point where the Service Provider provided a set of cleansed legacy data into a staging area ready for a big-bang load over a quiet weekend or so. That approach left the vast majority of the project and business outcome risk squarely with the Service provider. It also left a much bigger opportunity for the vendors to develop deeper customer engagement and revenues by taking on responsibility for the success of the business change being delivered. In fact a number of the major transformation contracts we have seen awarded over the past year have been moving substantially in this direction and away from the day-rate focus of the past where Change Control was a major revenue-generator for SIs and apps vendors. We expect the trend to continue strongly into 2011.

Cloud-based applications also threaten existing business models for DI vendors. There is a growing expectation that DI tools will be simply deployed and flexibly licensed, and that they will support Cloud-based apps and business processes that span not just internal business operations, but multi-party processes across the internet. This also raises the stakes for industry bodies to deliver standards for data models and interfacing that works out-of-the-box.

Businesses today are heavily reliant on interconnected applications, mash-ups and processes which are in turn utterly dependent on cross-and inter-enterprise integrated data. Integration happens continuously in a mix of real-time and background processing. In the old ‘application stack’ architecture invented in the 1990’s, the batch ETL (extract transform load) paradigm dominated and raw speed was king. OurService provider customers recognise that world no longer exists, and that flexibility, agility and control are the new table stakes. A gap still exists in mainstream offerings – one that delivers on those needs for enterprises in increasingly challenging conditions.

That Royal Wedding, Wikileaks, the snow and Data Migration Frameworks

TonyS1 — Mon, 20 Dec 2010 16:08:49 GMT

Well I hope the title got your attention at least. Of course there is no tangible linkage between William and Kate's forthcoming nuptuals, leaked diplomatic documents, the arctic weather and Data Migration, but all three have been enjoying airtime in my Twitter, e-mail and web traffic this week, so I thought I'd share my end-of-year thoughts under the banner.

I am, as it happens, still awaiting the arrival of my embossed, gold-leaf-on-vellum invitation for the joyful day in April next year, but I'm sure the delay is all down to the snow and Royal Mail getting stuck between Buckingham Palace and my house in Highgate - Ed Milliband lives just down the road and I expect his envelope is similarly delayed. Julian Assange was also apparently waiting on a special delivery but given Wikileaks divulging comments about members of the Royal Family he may be waiting a while.

The snow also managed to keep me and my fellow Easyjet passengers at Madrid Airport for around 5 hours longer than expected on Friday night travelling home to London after a couple of days talking enterprise Data Migration with clients and partners, but having listening to others who endured much longer delays I think I got away pretty lightly.

As we move towards the close of 2010 and the first decade of the 21st Century, I have to reflect on the growth in interest and understanding of the importance and value of Data Migration in our industry. The TM Forum has been very actively engaged for several years now on promoting the need for CSP's to transform operations and systems to achieve lean and agile businesses ready to take on the challenge of the new Web 2.0 economy and competition from the likes of Skype, Google, e-Bay and Amazon.

Smart Grids and Cloud services present equally compelling opportunities for CSP's to provide their customers with exciting new revenue-generating offerings and to leverage their unique experience and capabilities - scale, reliability, security and customer trust - but only if they can grasp the importance of managing the key issues around migrating product, services, people, processes and data from legacy to web 2.0 enabled platforms.

To take Smartgrids first, I attended a conference earlier this year organised by the Utilities sector to discuss the challenges of advanced meter data management (their version of MDM, quite different from the Master Data Managment we use the acronym to mean). I was frankly amazed at the naivity of the discussion. The majority of speakers were focused on deciding what kind of new network they would have to deploy to transport meter readings from smart meters to a billing platform, or on establishing data standards for defining a reading, or designing new hardware to manage secure home networks.

For those of you who have spent a few years in the Communications sector this will be surprising, since we know very well how to get data from a network into a billing system, (we call it Mediation and Rating) and there is a perfectly usable event record standard (IPDR) which will need no work at all to deliver what the Smart Grid world needs. The flakiest part of this environment is without doubt the home network, but as fast, reliable broadband ubiquity continues to grow and home area networks mature, this will soon become a non-issue.

The really interesting issues are around the impact of Smart Grids on the energy value chain. There is clearly a huge amount of protectionism going on within the current incumbent providers who really don't want anyone else muscling in on their customers and who are allowed to get away with taking several weeks to switch supply when requested, and who are also afraid of their customers getting to understand the value in the data describing their power consumption to drive switching.

This market reminds me very strongly of the early years of deregulation in the telecoms sector - and the opportunities for land-grab are just as clear. Could the CSPs take a share of this market? Absolutely - they have every piece of the armoury they need, and the regulator has laid open the gates. The ability to flexibly and accurately manage and migrate large volumes of data about tariffs and customer usage will be the key enablers.

Clouds also represent similar opportunities for CSPs to grow existing client revenues and win new market share. The TM Forum membership's growth areas in Defense, Financial Services and Utilites all reflect a clear thirst for understanding and belief that the Communications sector has a clear lead in this area. We in telecoms have been at the forefront of providing massive scale network services for many decades, and have developed a culture and deep knowledge and expertise doing that.

But Cloud based core enterprise applications are much more fundamental to our customers than simply providing enterprise e-mail or virtualsation services. These are absolutely mission-critical systems which cannot afford to be down for any time at all, the integrity of which must be uncompromised. Access and security are of paramount importance, and they are often vital to value chains that span many companies.

Setting these systems up in th cloud is not very different from the on-premise versions that came before - though they clearly need to be highly resilient, performant and secure, and usually support multi-tenancy. The really tricky challenge is making sure that each customer's data can be accurately loaded and maintained, and in ensuring that their business doesn't need to stop while the transition is made. And if it was important beofre that software and platform upgrades were made seamless, with multi-tenancy it becomes vital.

Pretty much everywhere I go now I hear a growing requirement for more options around enterprise application data migration. The old world of Big Bang, weekend downtime isn't quite dead, but its days are certainly numbered. My customers and partners are all looking for more strategic transformation options, and as the communications sector redifines itself over the coming decade to enable Clouds, Smart Grids and many other services, that demand can only grow.

So I'll continue to watch the post for the invitation from The Palace, and hope England can bounce back to win The Ashes from a resurgent Aussie cricket team and wish you all a great holiday and prosperous 2011.

Re: TM Forum Data MIgration Training Courses

TonyS1 — Mon, 20 Dec 2010 14:40:09 GMT

Hi Guilherme,

Yes, the course is being developed and is expected to be available through the TM Forum education framework during the first quarter of 2011. I will make sure this community is updated when we have agreed delivery dates.

Have a great Christmas and New Year.

Tony.

Re: TM Forum Data MIgration Training Courses

Guilherme Esser — Fri, 10 Dec 2010 18:24:46 GMT

Hello Tony,
as is the continuation of this course?
Any news?

Tks.

TM Forum Data MIgration Training Courses

TonyS1 — Tue, 05 Oct 2010 17:22:37 GMT

Exciting news - the community is now working on building Data Migration Training Courses to be run under the TM Forum education programme.

Draft Training topics include:

Data Migration Concepts

Data Migration Strategy and Planning

Data Migration Best Practice

Data Migration Procurement

Data Migration and the TM Forum Frameworx standards

Training will be delivered by a combination of TM Forum staff and other community members

Bids for further topics are very welcome - just submit via replies to this post.

We need your contributions

TonyS1 — Wed, 25 Aug 2010 17:18:23 GMT

There are now 88 members of the Data Migration Community - not bad.

That suggests to me that there is a significant interest in building shared understanding and common approaches to solving the problems we all face in delivering effective Data Migration in the modern world.

A number of you have asked how you can contribute your thoughts and experiences with a goal of creating Best Practice for all the TM Forum membership.

I hope my blog entries give some food for thought and sometimes entertain - I'm also pretty active on Twitter (@sceales) and find that's a great place to hit a wide and immediate audience.

So please send me your contributions through this community and let's get the discussion moving.

Data Migration is a massive issue for the TM Forum membership - that was shown by last year's survey and is regularly played back to me by my customers, partners and colleagues. So tell us your war stories, both good and bad, and help your colleagues in the community to learn from them.

Thanks,

Tony.

Smart Grids - it's all about the Data

TonyS1 — Wed, 25 Aug 2010 13:16:16 GMT

A huge amount of the Smart Grid/Metering discussion to data in the utilities, government and reglatory authorities has centred on what kind of a new network and technology stack is required, and on the sophisticated functionality of metering equipment.

I'm not sure how many of the folks in the Utilites sector have noticed, but over here in the Communitcations (the clue's in the name!) sector have already provided some quite effective networks that will meet pretty much all of the requirements the Smart grid creates.

There is a question of how deployed meters and monitoring points connect to existing networks, since many domestic and commercial supply points are not inherently well placed to connect to network termination points, hubs or whatever. The cupboard under the stairs is great for storing the vacuum cleaner and half-empty tins of paint, but your copper/DSL termination is likely to be some distance away. Running a cable, setting up a wireless or IP over powerline connection are all options, but all have challenges for domestic users. All have advantages and challenges, but ultimately resolving these will help us drive towards the Connected Home.

Connectivity in the commercial sector is much less demanding where there is an IT finction to help out. But having said that firewalls, staff availability and complexity can all cause delay and ultimately cost. The latter is very much an issue in the public sector.

So once everything is connected the next step is to worry about the information architecture, security and value-chain.

Information architecture for meter/monitor readings is pretty straightforward to those of us forttunate enough to have spent time pondering Call and IP Data Records, mediation and rating for telco billing. Esseentially you need an identifier to link the reading to an installation, a timestamp, and an accurate enough number of Kw/H or whatever you are measuring. If you really want to get sophisticated you can encrypt the data.

Now for the Data. As is the trend in almost every area of life, switching on Smart Metering/Monitoring will generate quite a lot of data we don't currently have. My utility company takes a reading (often estimated) perhaps 4 times a year. With a smart monitor readings could be taken every few seconds (six is common).

The information contained in the readings has huge potential value for the consumer, the energy retailer, the power generator and for the planet. For the first time it will be possible to accurately assess and manage energy usage - potentially down to device level, to align tariffs with time-of-day usage, and to encourage behaviours that are helpful to our global energy use.

For all of us, understanding our carbon and energy usage is a basic responsibility, and increasingly governements will introduce legislation to mandate reduced carbon behaviours. These schemes will publicly praise or pillory participants depending on where they appear on league tables, and financially incentivize enterprises and individuals to conform.

This is a natural place for the telecoms service providers and their technology supply chains to add real value, and to allow the utilities to continue to work on their own core businesses getting energy efficiently to the population. We should embace it, and I welcome the formation of this new community in the Forum.

What price your data?

TonyS1 — Mon, 21 Jun 2010 15:40:34 GMT

The ‘i-generation’, or ‘digItal natives’ (defined as those born since 1990) have grown up in an era where huge amounts of data about them as individuals is shared widely. But when I left school in 1977 there wasn’t a computer in the place and mobile phones were only available on Star Trek (a popular sci-fi programme of the day…). So my generation’s perception of how much of my data it is reasonable for a third party to hold is perhaps rather different from theirs.

My 17-year-old daughter told me last week she appeared in 1,800 photos on Facebook and I asked whether she felt that she might regret that someday (when she is famous?). She didn’t see it as a problem – Facebook tells her when someone tags her images and she can un-tag them if she doesn’t think they’re appropriate. But I suspect she may not be thinking long-term enough. Rather like the cute tattoo or body-piercing, it may look terrific when you are in your teens, but less appealing when you are 38 and looking for an important job.

Recent public humiliation of leading service providers from Google to AT&T has shown that our faith in the security of systems to contain and protect data is often over-trusting, and that the longer they hold it the more likely that at some point it will leak or be used inappropriately. Part of the problem is that over time context inevitably changes, and our initial intentions get blurred and distorted. Laws change, politics change, technologies change and we change.

There is nothing like being disconnected from your precious personal or business data for a period to make you understand its value to you. We recently had a moment of panic at home when an important spreadsheet got corrupted and looked like it was unrecoverable. Crucial to preparing accounts on a deadline, we reached for the backup version, only to find that too couldn’t be opened. Technical Support (i.e. me) was being blamed for the failings of both Microsoft and Iomega (my backup media provider) to make creating a regular guaranteed backup idiot-proof! After a great deal of sweat we tracked down a modestly priced software product that magically restored the sick spreadsheet to health and allowed life in our house to return to normality.

Our data is of critical value to all of us as individuals and collectively as organizations – without it we can’t track the value of assets, bill customers, file accounts, turn up at the right time and place for appointments – we are in short, completely reliant on it to manage our personal and business lives. And yet our willingness to take huge risks with key information seems extraordinarily strong when the consequences of not being able to rely on executing key business processes can be dire.

I often hear of business transformation programmes where transition of key business data is reliant on ad-hoc processes, methods or tools. Major banks, government departments, telecoms service providers and others put faith in scripts, spreadsheets and other tools built for one-time execution with little or no usable audit trail of actions taken, let alone business visibility or control. Nightmare stories abound in the press of such approaches failing spectacularly and causing major damage to enterprises employing them.

Modern IT is closely link to business needs, standards and drivers. Service Oriented Architectures deliver Business Services where we previously talked about Application Programming Interfaces (APIs) to get IT applications to execute functionality on behalf of the business. This is a profound change, and greatly improves the agility with which enterprises can act to meet competitive threat, introduce new services and to reduce costs of operation. Taking a Business Services approach to data migration, which the latest software products and methods do, substantially reduces risk, cost and timescales for transformation.

To summarise, our data (domestic and business) is a critically valuable resource. We all need to take a long-term view of how we need to protect and who we trust to look after it. Finally, our data is at its greatest risk during major transformation, so we need to take extra care to use the right tools, methods and people to guard against failure during phases of change.

Data is the new Oil

TonyS1 — Thu, 27 May 2010 12:18:05 GMT

Gerd Leonhard, Media Technology and Communications Futurist declared this week at the TM Forum Management World conference that ‘Data is the new Oil’. Data, he said, is very precious and that Communications Service providers have the opportunity, skills, experience and infrastructure to exploit the Tsunami of data, that Sally Davies (BT Wholesale’s CEO) told us is heading our way.

I have previously blogged on the Petabyte era we are entering, driven by the massive adoption of new networked devices, social networking, cloud computing and flat rate tariffs. Opportunity clearly exists for the Communications industry to get a very much clearer picture of the consumers and businesses we serve. Just look at the way retailers use loyalty card data to drive an intimate understanding of the way we shop – and when the likes of Tesco’s (UK retailer) is attracting 1 in 8 retail pounds in the UK, that is seriously effective.

Tal Givoly, Amdocs Chief Scientist also gave a super presentation highlighting the huge power of the individual that has arrived (much to the dismay of United Airlines when an unhappy customer posted the number 1 music video of all time to express just how unhappy he was). It will be hard for us in this industry to respond because in many cases we just don’t get how powerful Twitter, Facebook and others really are – we just don’t use them enough. But the children of the I-generation, the digital natives born after 1990 surely do.

Tal demonstrated clearly how easy it is to get at the data in social networking sites – all have open APIs we can, and must use to get in touch with our markets.

Finally, if data is truly the new oil, then (as every BP executive must know) it is really important that we can understand and control it’s flow into, through and out of our organizations. Our attention is often focused on what happens in business as usual, but it seems to me that we need to be equally concerned with what happens when big changes happen.

12 thoughts for Christmas

TonyS1 — Mon, 28 Dec 2009 15:28:46 GMT

12 thoughts for Christmas:

1. Growth in data remains on a very steep curve, driven by growth in both services and usage – smart devices create a lot of data and messages to service providers

2. Moore’s Law, Virtualization, Utility computing and effective DC management will mitigate, but can’t solve the problem

3. Increasing regulation, and attention to detail in compliance checking conspire to make Directors’ lives more risky

4. Service Providers in virtually all sectors are moving towards (if they aren’t already there) 24x7 operations

5. Front, Middle and Back office will converge into a single continuum extending from the end customer to the far end of the supply chain

6. New entrants will emerge in technology markets as angel investors look for new places to put their money

7. Financial Markets meltdown of 2008/9 driving scrutiny of risk positions, cash-conservation and M&A activity

8. IP-based Video (including HD and 3D), Social Networking, User-generated content and Cloud computing will all continue to drive further rapid growth in data

9. Cleantech/Smart Grid technologies will move every household and office from around 2 manual usage readings per annum to 30 messages per hour or more

10. Our relationship with Service Providers will evolve towards a real-time transactional one – periodic billing cycles will shift toward immediate event pricing and billing

11. Standards and Open Source deliver great value at key interfaces, but COTS applications will continue to rule for core process support

12. Evergreening (preserving to you and me) existing technology investments will make 30-year old applications continue to deliver value

So, 2010 will be all about managing complexity and risk as we move from Thriving to Surviving during the next 12 months.

The journey may not be a smooth one, and some will not survive it. My money is on the players who invest both in their vision of their destination and in making sure the transition is as free from risk as possible.

My best wishes to all for the coming year.

Clouds - Nirvana or the Hotel California?

TonyS1 — Fri, 23 Oct 2009 16:28:44 GMT

Martin Creaner (TM Forum CEO) blogging on the state of Cloud computing this week pointed out that pretty much all of us are already experiencing Cloud services already. If you use Salesforce.com, or iTunes or Google applications, then you are already a consumer of Cloud services. So what is the big difference between these very widely adopted services and moving a much larger proportion of your business processes outside your firewall?

A colleague joked that Cloud computing reminded him of the Eagles classic – “check into the Hotel California – it’s a lovely place, but you can never leave”. Vendor lock-in is certainly one of the biggest fears customers have about having a third party host their key applications and business data, and the idea of not being fully in control of their destiny is not one many would find comfortable. The Economist cover story this week outlined the battle of the Titans where Microsoft, Google, IBM, Apple and Yahoo are fast moving to win the right to provide both apps and data as services to consumers and enterprises alike.

For all of us who have been around for a few cycles of the network speed/availability versus processor speed/centralized versus distributed computing debate, this feels like pretty familiar territory. When I started my career at the beginning of the 1980’s, mainframes ruled the computing universe, and although Gates and Jobs were starting to shake things up on the desktop, if you wanted to run payroll or billing for a large enterprise, you had no choice but to head for mainframe computing in the datacenter. IBM still has a very strong mainframe business and completely dominates that market today - and if you get involved in financial services you discover plenty of systems written in COBOL 30 years ago.

Now centralization is back in vogue, and as with the old bureau services providers (remember Compuserve?), we’re all keen to see the benefits of off-loading the complexity of internally managed IT to specialist vendors. All the same issues were debated back in the 1980’s and that didn’t stop the bureau guys making a lot of money, so I don’t really see why we wouldn’t be able to deal with them now. The key is generally to resolve the contractual and operational issues satisfactorily, and the technical issues will follow. Not every application or service will work in the cloud, but I’d be surprised if most would not be OK.

Actually I anticipate a much bigger question being how enterprises transition business processes and applications to clouds – whether they be internal or externally provided. Organizations are now 24x7, and operate very complex, interconnected business processes that cross commercial, time-zone and geographic boundaries. Legacy sits alongside state-of-the-art systems, and compliance requirements to maintain business-level auditability of all actions have never been more demanding. Migration to the cloud will not be simple for core services, and new methods, strategies and tools are going to be needed.

Choice of Cloud vendor is clearly important – as with any supplier you need to know you can work with them on agreed terms and have recourse to remedies if things go wrong, including provision for how you can leave. Parting company with a Cloud provider will clearly be just as complex and risky as the process of migrating to them, but if you think it through at the beginning there should be less surprises if you need to do that. Is that any worse than dealing with your own internal IT department? I guess that depends on how good they are and what your relationship with them is – but I’ve experienced businesses with completely in-house IT and fully Cloud-based services – and I have to say the latter gives a lot less headaches.

On a parting note, do please go to the link at http://www.tmforum.org/community/groups/data_migration/default.aspx, join the TM Forum Data Migration Community and complete the Survey – we need your data.

Data Migration on the move

TonyS1 — Thu, 27 Aug 2009 16:40:00 GMT

As we all get back to work after a glorious summer (except inScotland where it rained solidly for the week I was there), plans arealready underway for an exciting next few months.

While Australia lost the Ashes to England, Telstra successfullycompleted their IP/MPLS network inventory using Celona’s nextgeneration Application Data Migration with zero downtime andnear-perfect data quality. Given the high level of complexity both inthe data and the way it affects business users and their processes thiswas a highly significant achievement.

So what’s happening in the world of data migration? TheTeleManagement Forum (TMF) has been leading an initiative to defineBest Practice in Data Migration across the Communications sector, andwill be producing a White Paper on the subject in time for the OrlandoManagement World Americas event in early December, where Celona willalso be presenting a Catalyst programme along with KPN and Logica. Thiswill build on and extend the the intiative presented in Nice during Maythis year.

The TMF is also running a survey of its membership to build a clearunderstanding of how data migration is being done today, and to give anopportunity for them to help steer future development in this key area.Given that data migration is a key enabler for successful businesstransformation, the importance of this work cannot be underestimated.The results of the survey will be published as part of the BestPractice White Paper.

I’d also like to highlight the up-coming Data Migration Matters 2event in London. This follows a highly successful conference last yearwhich gathered experts and end-users together for some excellentspeakers and enthusiastic debate, so I strongly recommend a visit to www.datamigrationmatters.orgto register early as places are limited. Johny Morris, author of theseminal ‘Practical Data Migration’ book will be speaking along withindustry leaders sharing their experiences and ideas.

Analysts are increasingly discussing data migration, and itsrelationship to data integration and other data managementapplications, and that reflects big name companies spending significantmoney on working out how to play in this fast-growing space. M&Aactivity continues as Oracle, IBM, SAP and others jockey for positionin dominating the world of enterprise data management, and datamigration is now on everyone’s agenda.

Hot items for debate are: how to safely migrate businessapplications and processes to clouds, how to achieve business processtransformation in-flight with zero downtime or disruption, howmigration can help meet green computing needs, and how to maintainregulatory compliance during major transformation. I’ll be discussingall these topics in more depth in my next few blogs

Are you really in Control of your Data Migration?

TonyS1 — Tue, 07 Jul 2009 16:50:00 GMT

There’s a lot of data migration going on - the vast majority of business change programmes involve moving large amounts of complex data around. It doesn’t matter whether these are implementing new or changed processes to drive more efficient operations, integrating customers from an acquired company, supporting new compliance reporting requirements or launching new products and services. With enterprise data storage spiraling ever larger (the average large business will own 8Tb of data by 2012*), a globally on-line supply-chain and customer base demanding 24x7 systems and process availability, it is clear that more options are needed for business to absorb and embrace change than have historically been available.

Research from Bloor indicates that quality of outcomes – 80% of projects delivered late, over-budget or failing outright – has been very poor, and that clients perceive poor understanding of scope, data quality and changing requirements are the major causes of project failure. I’d argue that these three items are never going to be easy to nail down at the outset, but that most methods and tools fail to anticipate their impact and deliver key capabilities needed to drive up customer experience of migration.

Data migration projects have historically over-emphasized the mechanics of data mapping and transformation, while under valuing the importance of stakeholder engagement, compliance and control. In a way, it’s not hard to see why - how could you economically build a fully integrated application product to solve a problem that was perceived as transient? Building the key features needed to deliver on engagement, compliance and control is non-trivial in an environment as variable as those faced by data migration. But with a productized approach, the costs of development and maintenance of these functions can be spread over a large user base.

Whilst the technology needs to deliver a new level of automation and control at lower cost and risk, new methods must also be employed to create strong connections with the business stakeholders at all levels of the engagement. You must also ensure understanding and insight is shared and migration progress maintained regardless of changes in scope and new data sources discovered. That extends into the way vendors and internal stakeholders collaborate and interwork. A progressive approach does not afford the simple, clear-cut phases and boundaries that characterized scripted big-bang migrations. So with more complex definition of responsibility, better governance is required. The outcome is clear adhesion between the migration and the business change being implemented, and in my view that is surely worth having.

Data Migration in the Petabyte Age

TonyS1 — Thu, 04 Jun 2009 14:06:00 GMT

Volumes of data continue to grow rapidly - we now frequently talk in terms of Petabytes – AT&T transfers 16Pb of data through its network daily, while Google processes 20Pb of data per day. When such companies upgrade their systems, reorganize or acquire new companies, they face a new scale of problem from previous generations of enterprises. And it isn’t simply a volume problem, because the data they use to manage their operations is frequently inter-related and shared across both processes and applications. They may not even directly own such data – it may belong to a partner or supplier company, or to their customers.

Best practice in data migration used to be all about speed – how fast can I load the target system database to minimize disruption to business users? Simplistically we used to ask the business what application downtime they could tolerate, then analyze the data to be migrated and divide the time available by the volume. That gave us a target load rate per object, and we then set about designing a technical strategy to achieve that rate. That often meant taking some short cuts on integrity by dropping database indexes and data validation routines, and sorting incoming data to fit physical storage algorithms. A number of market products developed to achieve massively parallel processing to maximize the speeds such approaches could achieve.

Datamigrationpro has published an excellent article by Dylan Jones on the Load Rate Bottleneck http://datamigrationpro.squarespace.com/data-migration-articles/2008/4/16/how-to-avoid-a-data-migration-traffic-jam-the-load-rate-bott.html in which he explains the challenge posed by proprietary software applications requiring data to be loaded via a standard gateway or API (Application Programming Interface). Whilst this guarantees the integrity of the data being loaded, it does so at the expense of speed. That’s because of two major factors, namely the validation of the incoming data against existing data already in the system to avoid duplication and ensure required predecessors exist, and also the overhead of working at the application layer rather than directly into the database management software. Together these can slow load rates up by several orders of magnitude relative to direct database storage.

Modern enterprises are no longer prepared to take risks on the integrity of their data – personal penalties to Directors for failing to safeguard private and commercial data ensure this. Neither do they have the luxury of downtime to load data – they run 24x7 operations and there is barely time to change a fuse in the datacenter, let alone perform a complex upgrade or migration. It must follow that new approaches are needed to support real-time migration and allow this to happen in parallel with normal operations.

In designing such approaches we need to be cognizant that the migration process needs to include a strategy for accessing the data to be migrated – commonly from multiple source applications, correlating and cleansing this and maintaining explicit and centralized control and visibility throughout. What looks like a perfect strategy for loading data often conflicts directly with the optimal design for obtaining the source data. Add to that the reluctance most enterprises feel about touching the source applications to allow access to the data – not surprisingly given the great age and fragility of many systems, and a single-event migration starts to look like a weak solution for many projects.

In considering volume we also need to distinguish between standing data and transactional data – the former generally qualifies how the latter will be processed, will be much lower in volume and has a much longer lifespan. It is particularly important therefore to take a very critical view of what the organization actually needs to take forward into the next generation architecture. Far too often requirements state that all current and historical data must be migrated, resulting in months of additional work to massage ancient data or customizing an otherwise pristine target application to cope with very little real business value. Statutory requirements to maintain financial records over several years can often be met with sensible off-line or summary solutions.

Nice - the aftermath

TonyS1 — Tue, 12 May 2009 17:21:00 GMT

Following the TMF Management World event in Nice last week, I take the view that the communications industry is entering a new phase.

The last few years have been dominated by news of major players planning massive transformations in networks, services and organization. But from the response to Trudy Norris-Grey’s question in her keynote speech (only 5 out of 150-odd attendees worked for companies actually engaged in such a transformation) it would appear there isn’t that much really going on.

Doug Zone (Metratech CTO) put together a good list of reasons why CIOs are reticent about replacing legacy systems which I will quote:

1. It doesn't do what I want, but no replacement will either...
2. I really don't know what it does now, so how can I possibly replace it...
3. I don't really know what the users want, and I am afraid to ask...
4. You can see the result of the last time I used a rigorous replacement selection process ...
5. If I replace one piece I will have to replace everything. With so many integration and mini (and not so mini) "workaround" apps, it will collapse like a house of cards.
6. I have implemented an SOA to allow easy application replacement; it has been about as successful as my EAI initiatve 10 years ago.
7. I know what I want, I know what I need, but I have neither the business analyst or solution architect talent to put it into place. And... I either can't afford or don't trust system integrators.
8. I am 52 years old and I think I can keep this thing going for another 8 years.
9. CIO's are supposed to be strategic. How can replacing a system be strategic? Maybe our our strategy is perfectly suited to our current systems.
10. Let my competition replace their legacy! I will be reeling in the customers when they do!

Whilst there is truth in every item on Doug’s list, my takeaway from Nice was a change in outlook from all the customers we spoke to - a real willingness to confront the underlying issues that have given rise to the inertia we have seen to date. The drivers behind this look pretty clear, and in contrast to Doug’s objections, they come from business rather than technology. As observed by Sanjiv Ahuja (Augere, ex-Orange), voice revenues will continue to decline to the point that service providers will finally be forced to find other ways to provide value to customers or be relegated to bit-pipes. This is inevitable and the pace is accelerating. The attraction of becoming global ICT providers has also seen some major challenges delaying the generation of new revenues from that source, while IPTV has also yet to prove itself a major new cash provider. Combine this with Apple, Google, Amazon and E-bay (Skype) all gathering at the gates, the imperative for change is becoming hard to ignore.

Taking on the Web 2.0 companies and winning the battle for wallet-share in the new post-crunch economy will be tough, and relies on mixing collaboration (Google and Amazon both have fantastic platform technology they share with partners) and revenue sharing (I-mode was pretty successful) with a radical reduction in running costs of operations with declining revenues. Customer knowledge, care and trust twinned with a service-led approach to technology delivery will also be very important differentiators.

The key message for me is that the industry has recognized that transformation is absolutely needed, and that a range of approaches are needed to achieve it. Given the speed at which markets, technology and competitive landscapes are changing, the complexity of our enterprises, and the demand for 24x7 operations, a one-size-fits-all approach to transformation cannot realistically be the answer. Moreover, the ability to flex and adopt different strategies to suit the phase and target business functions requiring transformation has become a critical need.

Overcoming the inertia characterized by Doug Zone’s list of ‘reasons why not’ by offering low-risk, fast ROI, reusable migration capability will underpin a shift in service providers to fully engage in a practical and pragmatic programme of transformation. This is now both commercially and technologically possible with the advent of Next-Generation migration products coming to market with proven capacity to implement in the challenging world we are in. At Celona we’re ready to engage with the new wave of CIOs and Directors of Transformation with a list of ‘reasons why we can’ – bring it on!

Application Data Migration Catalyst at Management World Nice

TonyS1 — Wed, 29 Apr 2009 18:42:00 GMT

Next week the world of Telecoms OSS/BSS professionals gathers in (hopefully) sunny Nice on the Cote d’Azure. We will spend time discussing how the world has changed since we last got together, and how events have affected trends and approaches to managing customers, products and complexity in a very dynamic environment.

The TM Forum has led in orchestrating the definition of models and standards for operation, and in fostering innovation in a sector renowned for commercial and technological creativity. This year the TM Forum has directed the spotlight within the Business Transformation stream onto Application Data Migration, and Celona will be leading this initiative with a new Catalyst showcase with KPN, the Dutch incumbent service provider, and Logica, global SI.

Details of the Catalyst can be found here and it would be great to welcome readers in person throughout the Management World event and see what we have done with KPN and Logica in a compelling demonstration of how Application Data Migration drives business value and transformation.

There is a growing wave of demand for new methods and solutions to the perennial challenge of managing transition between different organizational and technical states with zero downtime in servicing customers, selling products or running back-end processes. Traditional scripted Extract-Transform-Load simply doesn’t work in any but the most trivial of cases, and I’m keen to engage with the TM Forum membership to develop and extend the discussion on changing the game for Data Migration as an enabler for Business Transformation. So please stop by the Catalyst demonstrator in Nice and meet the team, including our KPN customer and Logica to find out how to benefit from a change in approach, and share your own experiences and challenges with us.

If you would like to discuss directly how you can participate in this new catalyst initiative which will cover many application areas from Network Inventory, to CRM, Billing and order management, you can email me at: datamigrationcatalyst@celona.com

Data Context - a key Transformation enabler?

TonyS1 — Mon, 27 Apr 2009 12:13:00 GMT

James Pullen’s OSSLine posting ‘The Top-5 OSS/BSS Integration Things’ http://www.ossline.com/2009/04/the-top-5-ossbss-integration-things.html makes a number of excellent points everyone involved in OSS or BSS software should take very seriously, including his point #4 on Data Without Context.

He highlights the issue that different applications and the business processes they support have different needs from data describing the same business entities, and that although you might yearn for a magical single truth view of a customer, product, service, transaction etc, it is both very hard to achieve and not necessarily that useful to try to hold it in one place. The Master Data Management (MDM) and Service Oriented Architecture (SOA) community have strong opinions on how to deliver consistent context-ready data to any application that needs it, and the literature is growing daily.

My particular interest is in thinking through how the data context question is impacted during migration and transformation activities. James asserts that 80% of the work in achieving correct understanding of data is in context and meaning, while only 20% is in delivering integration, basic format and content. Accurate records of relationships between data and time are particularly critical and are among the hardest elements to get right. That’s certainly true in ‘normal’ operations, but even more difficult when the business is moving to new processes and systems.

This does infer core requirements on solutions for transformation and migration – you can’t get away from the need to accurately plan and track the journey each object will take through migration and to access business logic that ensures you treat those objects correctly as they move from old application to new. Building in Common and State Models helps with the process of controlling the primary key relationships (correlation) between application views of the same or related objects, but (again I’m in agreement with James), don’t expect a TMF SID to be the answer there. SID and the NGOSS contracts help us to agree on an understanding of what the primary objects are and how they relate, but are never going to include sufficient granularity to support an application, especially not an OSS/BSS COTS product. They are, however, very helpful in creating common understanding around data interchange and integration and during migration, that is a key need.

Surveys from Gartner and Bloor suggest that over 70% of data migration is still done today using ad-hoc scripting, with most of the remainder using re-purposed data warehousing/ETL tooling, though new players are now emerging with built-for-purpose products in this space.

I’d be very interested to hear from customers, practitioners, SI’s and ISV’s how they overcome the Data Without Context issue in such complex migrations as are commonplace in the enterprise arena. Do you build controls as I’ve laid out every time, ignore the requirements for control, or do your tools include appropriate enterprise-grade capability in this key area? Building Best-Practice for Data Migration in the Communications Sector is a stated TMF goal – so let’s start by understanding what people are really doing today in the field.

Business-focused Data Migration – The Key to Next Generation Telecom?

TonyS1 — Tue, 21 Apr 2009 13:05:00 GMT

The process of moving data can speed up service provider transformation and allow them to realize the benefits of their transformation investment much faster.

Data migration is all about moving data from one environment to another in an optimal way, with the least amount of business disruption and risk; whether to consolidate applications or to change from one vendor, or version of an application to another, data migration is at the heart of the deployment process. In the Telecom market, OSS/BSS environments are being optimized to fit in with Next Generation strategies for driving down operating costs, improving efficiency and increasing customer focus. In addition, Mergers and Acquisitions have been prevalent in recent years, while the convergence of service networks and regulatory pressures have greatly increased the focus on business application deployment or upgrade and consequently application data migration.

This has been particularly true for the areas of Network Inventory Management, Billing, Customer Management and Web services. However, many such application decisions get deferred because of concerns over the data migration aspects of such projects. In other words, and especially for mission critical and 24x7 applications, the potential risks of time and cost overruns in data migration mean that many enterprises prefer to delay their application plans, ultimately to the detriment of the business. Undoubtedly, data migration projects have typically been prone to time and cost overruns and, in some cases, outright failures. However, this does not need to be the case anymore.

Data Migration that Supports Business Requirements
One of the major problems associated with data management in general is the tendency to treat data in isolation rather than as a part of a broader environment. That is, to treat data separately from applications. By doing this, the data is essentially separated from the business and can lose its integrity. When it comes to data migration, this is particularly true. In part this is because historical solutions (conventional ETL tools and scripts) were designed for a different purpose (moving data into a data warehouse), but in any case such tools have no understanding of the context within which data is used and therefore cannot ensure that the context is retained.

From the outset, it is key to de-risk the data migration process, to understand the context within which data is being used in the business, and to preserve that context throughout the migration process.

Telecom IT transformation, or rather optimization of OSS/BSS stacks, is a highly desired outcome. Current architectures in many cases are overly complex, not fully automated and lack flexibility. While the solutions landscape is littered with systems that are too expensive, unable to scale or cope with the demands being placed upon them, or are simply redundant or duplicated and need to be removed. None of this is news to the industry. We know that as pure organic growth goes out of maturing markets we need to control operational costs, focus on customer satisfaction and innovate to replace flattening revenues from commoditized services such as voice.

The problem, as always, has been in delivering against that vision. At a high level the TM Forum frameworks such as the Business Process Framework (eTOM) and Information Framework (SID) look both appealing and feasible. But on the ground we also know how much hard work is required to deliver against the vision. Of course some CSPs (Communications Service Providers) have done so – and the whole market salutes their determination, vision and industry – but for the rest the problem remains of how to move from where they are to where they want to be.

Buying a new back-end infrastructure is not like buying a new car. In the automotive business you go to a showroom, have a test drive, agree on a price and then select the extras you would like to have. A few weeks later your car, personalized to your requirements, is delivered for an agreed on-the-road price. The process is fairly predictable and customer-centric – they even take your old car away and you can trade it in to get a better price. Auto dealers certainly do not have to incorporate integral, vital parts of your old car into the new one.

The truth is that transformation or optimization begins after system selection. It is at that point that business process transformation, implementation, integration and data migration / transformation begins.

In fact the full value from a new billing, CRM or inventory management solution cannot be fully realized until a whole load of complex legacy, yet business critical data has been moved to the new solution. Operational efficiency objectives from switching off legacy solutions cannot be realized until all the necessary data has been moved to the new target system. And by even attempting to move this data, business exposure and risk is high.

So a desired outcome is to optimize, but many fear the transformation. It is important to understand that this is not a one-off process: back end transformation cannot be achieved in one hit for the average CSP. And neither does throwing money at the problem negate risk or guarantee data being delivered on time.

Time is the real enemy here: the longer it takes for a business to migrate its data, the more problematic the migration will become. Reduced migration cycle times are therefore key, as is predictability. Without predictability you cannot plan, and for large telecom operators migrating systems is something of a way of life. Not being able to deliver migrations predictably and reliably means they cannot control their ROI or cannot perform accurate business planning – impacting both shareholder and customer confidence.
What makes the process more problematic is that there is no time to migrate any more. Business applications operate 24×7. Applications cannot simply be switched off at the weekend and the data then moved. And, given the sheer volume of data involved, even if they could switch everything off for the weekend, they could not move it all in that time.

Business-centric Data Migration Opens the Door
Business-oriented application data migration enables quick, reliable, predictable data migration in a controlled fashion, all within a zero downtime environment. Now you can achieve that fast-track transformation or optimization that will deliver competitive advantage, shareholder and customer confidence, and lower operational costs. Modern purpose-built data migration platforms address many of the problems that CSPs have struggled with when using older data migration technologies and scripts.

Modern platforms enable a CSP to have much greater control and increased visibility of the migration process, which improves efficiency and lowers risk. They support bi-directional synchronisation of data held in legacy and new applications, and provide a zero downtime environment, a feature we pioneered at Celona. Furthermore, they enable CSPs to manage the movement of users, data, applications and processes independently, ensuring that the complexities of process-dependent data are fully managed.

Such platforms also support the concept of the business or common model, in which core business entities (Customers, Account, Products, etc.) and their complex relationships are defined in an application-neutral way that allows for much greater reuse and faster time to ROI from transformation projects.TM Forum’s Information Framework provides a ready-made starting point for the common model for the CSP.

From a CSP’s perspective the most important benefit is that these modern approaches and purpose-built tools enable both fast-track transformation and a faster time-to-value from that transformation. Data can be migrated incrementally in business-driven data sets, and CSPs can therefore gain value from their investment much faster.

The message from TM Forum and its membership is clear - now is not the time for CSPs to delay their transformation or optimization programs, but it is time that they raised their expectations about how these transformations can be delivered with certainty and plan accordingly for a successful outcome that fully embraces and enables the business goals.

The communications industry does now need to engage in the debate about how we should explore the new migration opportunity and build best practice, standards and adoption. TM Forum is fully supporting this initiative – I have started the ball rolling with a new blog on the TM Forum website, and we plan to build a data migration program around that including a Catalyst project to launch at Management World 2009 in Nice next month.

So do please contact me through the TM Forum Community or directly at tony.sceales@celona.com if you’d like to get involved.

The Need for Speed

TonyS1 — Mon, 30 Mar 2009 13:29:00 GMT

There is an excellent article on the DataMigrationPro website highlighting application data load rates as one of the issues facing enterprise data migration projects today.

To summarise, most modern applications don’t allow customers to load data directly to the database, protecting their software with an application programming interface (API) that checks that the data meets strict integrity rules before accepting it. Those API’s are generally orders of magnitude slower to run than raw SQL can load a database, especially when rules enforcing integrity are relaxed and indices removed. With complex applications load rates of 10 or less objects per second are quite normal, and are rarely faster than 100 or so. In my view this is a price worth paying to be able to guarantee the integrity of the data in the receiving system, and commercially the only viable path without contravening warrantees.

This is not a new problem though, but one which migration solution providers have spent a lot of development effort trying to circumvent. They often end up replicating all the business logic of the API in their migration routines – which seems to me a dangerous exercise and makes the transform very heavyweight. Then they remove the database integrity checking and indices to enable a fast table load, introducing significant risk of quite subtle data corruption in the process – I saw a ERP migration project a few years ago that looked to have completed fine on the Sunday evening and everyone went home. Two weeks went by before calls started arriving calls from customers who were finding odd data combinations on bills. It took another three weeks and a huge number of very long shifts before the mess was sorted out and the offending index settings rebuilt.

The ‘How fast can you load?’ question really misses the important question which is ‘How fast can the enterprise absorb the change?’. Consider the end-user training challenge – a thousand end-users will take more than a day to train on most major new applications – so why not plan the migration around the training plan, cutting over a team at a time over a longer period? Often the problem is compounded by the issue of dependent systems – no application is an island and taking out a slice of data from one to another could cause a dependent system to lose touch with transactions moved to the new platform. You have a couple of choices – either reconfigure the dependent system to expect to get some of its data from the new platform, or synchronize the old and new systems until you are ready to move the whole interface from old to new.

Deciding on the right cutover strategy is always difficult since no one approach is likely to meet the needs of all stakeholders. Previously big-bang, where everything happens in one major migration event overnight or a weekend has been favoured because it makes for a cleaner break between old and new operations, and releases the legacy licenses, platforms and equipment immediately when complete. But it’s also the highest risk unless the applications, data and connectivity have been pre-loaded and thoroughly tested in advance (and kept freshly synchronized with the operational source data), and the method least in keeping with current recommendations to renew processes and applications incrementally. So that is probably triggering the increasing numbers of requirements from customers looking for more options for cutover, and that is backed up by them prioritizing the importance of low risk and data integrity over very fast load rates.