Data Migration

Gartner November 2010 Magic Quadrant points to further consolidation in Data Integration

Tony Sceales — Thu, 06 Jan 2011 17:10:29 GMT

We’re currently seeing much consolidation, and strong rumour of further consolidation, in the Data Integration (DI) market, with major players seeking to strengthen their position in MDM (Master Data Management), data integration and data migration. The recent Gartner Magic Quadrant highlights the importance placed on having a completeness of vision and an ability to execute. Oracle’s 2010 acquisition of GoldenGate and rumours of a bid for Informatica along with Talend’s acquisition of Sopera point to an industry that sees significant value in having the right tools for the job.

Gartner specifically calls out Data Migration and Synchronisation as key elements of the DI capability set, but interestingly omits Data Quality. They don’t strongly recognise (from our viewpoint) the emergence of cloud-based data management and the impact of software-as-a-service in this arena – for instance Jigsaw, the emerging cloud-based contact and client data management solution provider is not mentioned.

The risk for some traditional players in the market, such as the Systems Integrators (SIs), is that their labour intensive, bespoke approach to projects will increasingly count against them in a world of much greater sophistication and automation. Future oriented clients, particularly in the increasingly agile Communications, High Tech and Financial Services Sectors are looking more and more for solutions that bring control, performance and reliability. Those are elements that a bottom-up build often promises but rarely delivers.

There is also exposure for Software Vendors currently in the Data Integration sector if they are unable to rapidly bring comparative and competitive solutions to the table whilst their peers take the acquisition shortcuts.

Business models are clearly changing for Applications Vendors and Systems Integrators alike as Service Providers increasingly drive for greater agility and flexibility to meet competitive and financial pressures. The old world - where enterprises bought software licenses from the apps vendor and hired an SI to stand it up and manage the transition of business processes and population with transformed legacy data – is fast disappearing. Service Providers now ask that vendors (SI and Apps alike) take a much broader responsibility and include data integration and migration in the scope of their contract.

Vendors traditionally contracted at the point where the Service Provider provided a set of cleansed legacy data into a staging area ready for a big-bang load over a quiet weekend or so. That approach left the vast majority of the project and business outcome risk squarely with the Service provider. It also left a much bigger opportunity for the vendors to develop deeper customer engagement and revenues by taking on responsibility for the success of the business change being delivered. In fact a number of the major transformation contracts we have seen awarded over the past year have been moving substantially in this direction and away from the day-rate focus of the past where Change Control was a major revenue-generator for SIs and apps vendors. We expect the trend to continue strongly into 2011.

Cloud-based applications also threaten existing business models for DI vendors. There is a growing expectation that DI tools will be simply deployed and flexibly licensed, and that they will support Cloud-based apps and business processes that span not just internal business operations, but multi-party processes across the internet. This also raises the stakes for industry bodies to deliver standards for data models and interfacing that works out-of-the-box.

Businesses today are heavily reliant on interconnected applications, mash-ups and processes which are in turn utterly dependent on cross-and inter-enterprise integrated data. Integration happens continuously in a mix of real-time and background processing. In the old ‘application stack’ architecture invented in the 1990’s, the batch ETL (extract transform load) paradigm dominated and raw speed was king. OurService provider customers recognise that world no longer exists, and that flexibility, agility and control are the new table stakes. A gap still exists in mainstream offerings – one that delivers on those needs for enterprises in increasingly challenging conditions.

That Royal Wedding, Wikileaks, the snow and Data Migration Frameworks

Tony Sceales — Mon, 20 Dec 2010 16:08:49 GMT

Well I hope the title got your attention at least. Of course there is no tangible linkage between William and Kate's forthcoming nuptuals, leaked diplomatic documents, the arctic weather and Data Migration, but all three have been enjoying airtime in my Twitter, e-mail and web traffic this week, so I thought I'd share my end-of-year thoughts under the banner.

I am, as it happens, still awaiting the arrival of my embossed, gold-leaf-on-vellum invitation for the joyful day in April next year, but I'm sure the delay is all down to the snow and Royal Mail getting stuck between Buckingham Palace and my house in Highgate - Ed Milliband lives just down the road and I expect his envelope is similarly delayed. Julian Assange was also apparently waiting on a special delivery but given Wikileaks divulging comments about members of the Royal Family he may be waiting a while.

The snow also managed to keep me and my fellow Easyjet passengers at Madrid Airport for around 5 hours longer than expected on Friday night travelling home to London after a couple of days talking enterprise Data Migration with clients and partners, but having listening to others who endured much longer delays I think I got away pretty lightly.

As we move towards the close of 2010 and the first decade of the 21st Century, I have to reflect on the growth in interest and understanding of the importance and value of Data Migration in our industry. The TM Forum has been very actively engaged for several years now on promoting the need for CSP's to transform operations and systems to achieve lean and agile businesses ready to take on the challenge of the new Web 2.0 economy and competition from the likes of Skype, Google, e-Bay and Amazon.

Smart Grids and Cloud services present equally compelling opportunities for CSP's to provide their customers with exciting new revenue-generating offerings and to leverage their unique experience and capabilities - scale, reliability, security and customer trust - but only if they can grasp the importance of managing the key issues around migrating product, services, people, processes and data from legacy to web 2.0 enabled platforms.

To take Smartgrids first, I attended a conference earlier this year organised by the Utilities sector to discuss the challenges of advanced meter data management (their version of MDM, quite different from the Master Data Managment we use the acronym to mean). I was frankly amazed at the naivity of the discussion. The majority of speakers were focused on deciding what kind of new network they would have to deploy to transport meter readings from smart meters to a billing platform, or on establishing data standards for defining a reading, or designing new hardware to manage secure home networks.

For those of you who have spent a few years in the Communications sector this will be surprising, since we know very well how to get data from a network into a billing system, (we call it Mediation and Rating) and there is a perfectly usable event record standard (IPDR) which will need no work at all to deliver what the Smart Grid world needs. The flakiest part of this environment is without doubt the home network, but as fast, reliable broadband ubiquity continues to grow and home area networks mature, this will soon become a non-issue.

The really interesting issues are around the impact of Smart Grids on the energy value chain. There is clearly a huge amount of protectionism going on within the current incumbent providers who really don't want anyone else muscling in on their customers and who are allowed to get away with taking several weeks to switch supply when requested, and who are also afraid of their customers getting to understand the value in the data describing their power consumption to drive switching.

This market reminds me very strongly of the early years of deregulation in the telecoms sector - and the opportunities for land-grab are just as clear. Could the CSPs take a share of this market? Absolutely - they have every piece of the armoury they need, and the regulator has laid open the gates. The ability to flexibly and accurately manage and migrate large volumes of data about tariffs and customer usage will be the key enablers.

Clouds also represent similar opportunities for CSPs to grow existing client revenues and win new market share. The TM Forum membership's growth areas in Defense, Financial Services and Utilites all reflect a clear thirst for understanding and belief that the Communications sector has a clear lead in this area. We in telecoms have been at the forefront of providing massive scale network services for many decades, and have developed a culture and deep knowledge and expertise doing that.

But Cloud based core enterprise applications are much more fundamental to our customers than simply providing enterprise e-mail or virtualsation services. These are absolutely mission-critical systems which cannot afford to be down for any time at all, the integrity of which must be uncompromised. Access and security are of paramount importance, and they are often vital to value chains that span many companies.

Setting these systems up in th cloud is not very different from the on-premise versions that came before - though they clearly need to be highly resilient, performant and secure, and usually support multi-tenancy. The really tricky challenge is making sure that each customer's data can be accurately loaded and maintained, and in ensuring that their business doesn't need to stop while the transition is made. And if it was important beofre that software and platform upgrades were made seamless, with multi-tenancy it becomes vital.

Pretty much everywhere I go now I hear a growing requirement for more options around enterprise application data migration. The old world of Big Bang, weekend downtime isn't quite dead, but its days are certainly numbered. My customers and partners are all looking for more strategic transformation options, and as the communications sector redifines itself over the coming decade to enable Clouds, Smart Grids and many other services, that demand can only grow.

So I'll continue to watch the post for the invitation from The Palace, and hope England can bounce back to win The Ashes from a resurgent Aussie cricket team and wish you all a great holiday and prosperous 2011.

Smart Grids - it's all about the Data

Tony Sceales — Wed, 25 Aug 2010 13:16:16 GMT

A huge amount of the Smart Grid/Metering discussion to data in the utilities, government and reglatory authorities has centred on what kind of a new network and technology stack is required, and on the sophisticated functionality of metering equipment.

I'm not sure how many of the folks in the Utilites sector have noticed, but over here in the Communitcations (the clue's in the name!) sector have already provided some quite effective networks that will meet pretty much all of the requirements the Smart grid creates.

There is a question of how deployed meters and monitoring points connect to existing networks, since many domestic and commercial supply points are not inherently well placed to connect to network termination points, hubs or whatever. The cupboard under the stairs is great for storing the vacuum cleaner and half-empty tins of paint, but your copper/DSL termination is likely to be some distance away. Running a cable, setting up a wireless or IP over powerline connection are all options, but all have challenges for domestic users. All have advantages and challenges, but ultimately resolving these will help us drive towards the Connected Home.

Connectivity in the commercial sector is much less demanding where there is an IT finction to help out. But having said that firewalls, staff availability and complexity can all cause delay and ultimately cost. The latter is very much an issue in the public sector.

So once everything is connected the next step is to worry about the information architecture, security and value-chain.

Information architecture for meter/monitor readings is pretty straightforward to those of us forttunate enough to have spent time pondering Call and IP Data Records, mediation and rating for telco billing. Esseentially you need an identifier to link the reading to an installation, a timestamp, and an accurate enough number of Kw/H or whatever you are measuring. If you really want to get sophisticated you can encrypt the data.

Now for the Data. As is the trend in almost every area of life, switching on Smart Metering/Monitoring will generate quite a lot of data we don't currently have. My utility company takes a reading (often estimated) perhaps 4 times a year. With a smart monitor readings could be taken every few seconds (six is common).

The information contained in the readings has huge potential value for the consumer, the energy retailer, the power generator and for the planet. For the first time it will be possible to accurately assess and manage energy usage - potentially down to device level, to align tariffs with time-of-day usage, and to encourage behaviours that are helpful to our global energy use.

For all of us, understanding our carbon and energy usage is a basic responsibility, and increasingly governements will introduce legislation to mandate reduced carbon behaviours. These schemes will publicly praise or pillory participants depending on where they appear on league tables, and financially incentivize enterprises and individuals to conform.

This is a natural place for the telecoms service providers and their technology supply chains to add real value, and to allow the utilities to continue to work on their own core businesses getting energy efficiently to the population. We should embace it, and I welcome the formation of this new community in the Forum.

What price your data?

Tony Sceales — Mon, 21 Jun 2010 15:40:34 GMT

The ‘i-generation’, or ‘digItal natives’ (defined as those born since 1990) have grown up in an era where huge amounts of data about them as individuals is shared widely. But when I left school in 1977 there wasn’t a computer in the place and mobile phones were only available on Star Trek (a popular sci-fi programme of the day…). So my generation’s perception of how much of my data it is reasonable for a third party to hold is perhaps rather different from theirs.

My 17-year-old daughter told me last week she appeared in 1,800 photos on Facebook and I asked whether she felt that she might regret that someday (when she is famous?). She didn’t see it as a problem – Facebook tells her when someone tags her images and she can un-tag them if she doesn’t think they’re appropriate. But I suspect she may not be thinking long-term enough. Rather like the cute tattoo or body-piercing, it may look terrific when you are in your teens, but less appealing when you are 38 and looking for an important job.

Recent public humiliation of leading service providers from Google to AT&T has shown that our faith in the security of systems to contain and protect data is often over-trusting, and that the longer they hold it the more likely that at some point it will leak or be used inappropriately. Part of the problem is that over time context inevitably changes, and our initial intentions get blurred and distorted. Laws change, politics change, technologies change and we change.

There is nothing like being disconnected from your precious personal or business data for a period to make you understand its value to you. We recently had a moment of panic at home when an important spreadsheet got corrupted and looked like it was unrecoverable. Crucial to preparing accounts on a deadline, we reached for the backup version, only to find that too couldn’t be opened. Technical Support (i.e. me) was being blamed for the failings of both Microsoft and Iomega (my backup media provider) to make creating a regular guaranteed backup idiot-proof! After a great deal of sweat we tracked down a modestly priced software product that magically restored the sick spreadsheet to health and allowed life in our house to return to normality.

Our data is of critical value to all of us as individuals and collectively as organizations – without it we can’t track the value of assets, bill customers, file accounts, turn up at the right time and place for appointments – we are in short, completely reliant on it to manage our personal and business lives. And yet our willingness to take huge risks with key information seems extraordinarily strong when the consequences of not being able to rely on executing key business processes can be dire.

I often hear of business transformation programmes where transition of key business data is reliant on ad-hoc processes, methods or tools. Major banks, government departments, telecoms service providers and others put faith in scripts, spreadsheets and other tools built for one-time execution with little or no usable audit trail of actions taken, let alone business visibility or control. Nightmare stories abound in the press of such approaches failing spectacularly and causing major damage to enterprises employing them.

Modern IT is closely link to business needs, standards and drivers. Service Oriented Architectures deliver Business Services where we previously talked about Application Programming Interfaces (APIs) to get IT applications to execute functionality on behalf of the business. This is a profound change, and greatly improves the agility with which enterprises can act to meet competitive threat, introduce new services and to reduce costs of operation. Taking a Business Services approach to data migration, which the latest software products and methods do, substantially reduces risk, cost and timescales for transformation.

To summarise, our data (domestic and business) is a critically valuable resource. We all need to take a long-term view of how we need to protect and who we trust to look after it. Finally, our data is at its greatest risk during major transformation, so we need to take extra care to use the right tools, methods and people to guard against failure during phases of change.

Data is the new Oil

Tony Sceales — Thu, 27 May 2010 12:18:05 GMT

Gerd Leonhard, Media Technology and Communications Futurist declared this week at the TM Forum Management World conference that ‘Data is the new Oil’. Data, he said, is very precious and that Communications Service providers have the opportunity, skills, experience and infrastructure to exploit the Tsunami of data, that Sally Davies (BT Wholesale’s CEO) told us is heading our way.

I have previously blogged on the Petabyte era we are entering, driven by the massive adoption of new networked devices, social networking, cloud computing and flat rate tariffs. Opportunity clearly exists for the Communications industry to get a very much clearer picture of the consumers and businesses we serve. Just look at the way retailers use loyalty card data to drive an intimate understanding of the way we shop – and when the likes of Tesco’s (UK retailer) is attracting 1 in 8 retail pounds in the UK, that is seriously effective.

Tal Givoly, Amdocs Chief Scientist also gave a super presentation highlighting the huge power of the individual that has arrived (much to the dismay of United Airlines when an unhappy customer posted the number 1 music video of all time to express just how unhappy he was). It will be hard for us in this industry to respond because in many cases we just don’t get how powerful Twitter, Facebook and others really are – we just don’t use them enough. But the children of the I-generation, the digital natives born after 1990 surely do.

Tal demonstrated clearly how easy it is to get at the data in social networking sites – all have open APIs we can, and must use to get in touch with our markets.

Finally, if data is truly the new oil, then (as every BP executive must know) it is really important that we can understand and control it’s flow into, through and out of our organizations. Our attention is often focused on what happens in business as usual, but it seems to me that we need to be equally concerned with what happens when big changes happen.

12 thoughts for Christmas

Tony Sceales — Mon, 28 Dec 2009 15:28:46 GMT

12 thoughts for Christmas:

1. Growth in data remains on a very steep curve, driven by growth in both services and usage – smart devices create a lot of data and messages to service providers

2. Moore’s Law, Virtualization, Utility computing and effective DC management will mitigate, but can’t solve the problem

3. Increasing regulation, and attention to detail in compliance checking conspire to make Directors’ lives more risky

4. Service Providers in virtually all sectors are moving towards (if they aren’t already there) 24x7 operations

5. Front, Middle and Back office will converge into a single continuum extending from the end customer to the far end of the supply chain

6. New entrants will emerge in technology markets as angel investors look for new places to put their money

7. Financial Markets meltdown of 2008/9 driving scrutiny of risk positions, cash-conservation and M&A activity

8. IP-based Video (including HD and 3D), Social Networking, User-generated content and Cloud computing will all continue to drive further rapid growth in data

9. Cleantech/Smart Grid technologies will move every household and office from around 2 manual usage readings per annum to 30 messages per hour or more

10. Our relationship with Service Providers will evolve towards a real-time transactional one – periodic billing cycles will shift toward immediate event pricing and billing

11. Standards and Open Source deliver great value at key interfaces, but COTS applications will continue to rule for core process support

12. Evergreening (preserving to you and me) existing technology investments will make 30-year old applications continue to deliver value

So, 2010 will be all about managing complexity and risk as we move from Thriving to Surviving during the next 12 months.

The journey may not be a smooth one, and some will not survive it. My money is on the players who invest both in their vision of their destination and in making sure the transition is as free from risk as possible.

My best wishes to all for the coming year.

Clouds - Nirvana or the Hotel California?

Tony Sceales — Fri, 23 Oct 2009 16:28:44 GMT

Martin Creaner (TM Forum CEO) blogging on the state of Cloud computing this week pointed out that pretty much all of us are already experiencing Cloud services already. If you use Salesforce.com, or iTunes or Google applications, then you are already a consumer of Cloud services. So what is the big difference between these very widely adopted services and moving a much larger proportion of your business processes outside your firewall?

A colleague joked that Cloud computing reminded him of the Eagles classic – “check into the Hotel California – it’s a lovely place, but you can never leave”. Vendor lock-in is certainly one of the biggest fears customers have about having a third party host their key applications and business data, and the idea of not being fully in control of their destiny is not one many would find comfortable. The Economist cover story this week outlined the battle of the Titans where Microsoft, Google, IBM, Apple and Yahoo are fast moving to win the right to provide both apps and data as services to consumers and enterprises alike.

For all of us who have been around for a few cycles of the network speed/availability versus processor speed/centralized versus distributed computing debate, this feels like pretty familiar territory. When I started my career at the beginning of the 1980’s, mainframes ruled the computing universe, and although Gates and Jobs were starting to shake things up on the desktop, if you wanted to run payroll or billing for a large enterprise, you had no choice but to head for mainframe computing in the datacenter. IBM still has a very strong mainframe business and completely dominates that market today - and if you get involved in financial services you discover plenty of systems written in COBOL 30 years ago.

Now centralization is back in vogue, and as with the old bureau services providers (remember Compuserve?), we’re all keen to see the benefits of off-loading the complexity of internally managed IT to specialist vendors. All the same issues were debated back in the 1980’s and that didn’t stop the bureau guys making a lot of money, so I don’t really see why we wouldn’t be able to deal with them now. The key is generally to resolve the contractual and operational issues satisfactorily, and the technical issues will follow. Not every application or service will work in the cloud, but I’d be surprised if most would not be OK.

Actually I anticipate a much bigger question being how enterprises transition business processes and applications to clouds – whether they be internal or externally provided. Organizations are now 24x7, and operate very complex, interconnected business processes that cross commercial, time-zone and geographic boundaries. Legacy sits alongside state-of-the-art systems, and compliance requirements to maintain business-level auditability of all actions have never been more demanding. Migration to the cloud will not be simple for core services, and new methods, strategies and tools are going to be needed.

Choice of Cloud vendor is clearly important – as with any supplier you need to know you can work with them on agreed terms and have recourse to remedies if things go wrong, including provision for how you can leave. Parting company with a Cloud provider will clearly be just as complex and risky as the process of migrating to them, but if you think it through at the beginning there should be less surprises if you need to do that. Is that any worse than dealing with your own internal IT department? I guess that depends on how good they are and what your relationship with them is – but I’ve experienced businesses with completely in-house IT and fully Cloud-based services – and I have to say the latter gives a lot less headaches.

On a parting note, do please go to the link at http://www.tmforum.org/community/groups/data_migration/default.aspx, join the TM Forum Data Migration Community and complete the Survey – we need your data.

Data Migration on the move

Tony Sceales — Thu, 27 Aug 2009 16:40:00 GMT

As we all get back to work after a glorious summer (except inScotland where it rained solidly for the week I was there), plans arealready underway for an exciting next few months.

While Australia lost the Ashes to England, Telstra successfullycompleted their IP/MPLS network inventory using Celona’s nextgeneration Application Data Migration with zero downtime andnear-perfect data quality. Given the high level of complexity both inthe data and the way it affects business users and their processes thiswas a highly significant achievement.

So what’s happening in the world of data migration? TheTeleManagement Forum (TMF) has been leading an initiative to defineBest Practice in Data Migration across the Communications sector, andwill be producing a White Paper on the subject in time for the OrlandoManagement World Americas event in early December, where Celona willalso be presenting a Catalyst programme along with KPN and Logica. Thiswill build on and extend the the intiative presented in Nice during Maythis year.

The TMF is also running a survey of its membership to build a clearunderstanding of how data migration is being done today, and to give anopportunity for them to help steer future development in this key area.Given that data migration is a key enabler for successful businesstransformation, the importance of this work cannot be underestimated.The results of the survey will be published as part of the BestPractice White Paper.

I’d also like to highlight the up-coming Data Migration Matters 2event in London. This follows a highly successful conference last yearwhich gathered experts and end-users together for some excellentspeakers and enthusiastic debate, so I strongly recommend a visit to www.datamigrationmatters.orgto register early as places are limited. Johny Morris, author of theseminal ‘Practical Data Migration’ book will be speaking along withindustry leaders sharing their experiences and ideas.

Analysts are increasingly discussing data migration, and itsrelationship to data integration and other data managementapplications, and that reflects big name companies spending significantmoney on working out how to play in this fast-growing space. M&Aactivity continues as Oracle, IBM, SAP and others jockey for positionin dominating the world of enterprise data management, and datamigration is now on everyone’s agenda.

Hot items for debate are: how to safely migrate businessapplications and processes to clouds, how to achieve business processtransformation in-flight with zero downtime or disruption, howmigration can help meet green computing needs, and how to maintainregulatory compliance during major transformation. I’ll be discussingall these topics in more depth in my next few blogs

Are you really in Control of your Data Migration?

Tony Sceales — Tue, 07 Jul 2009 16:50:00 GMT

There’s a lot of data migration going on - the vast majority of business change programmes involve moving large amounts of complex data around. It doesn’t matter whether these are implementing new or changed processes to drive more efficient operations, integrating customers from an acquired company, supporting new compliance reporting requirements or launching new products and services. With enterprise data storage spiraling ever larger (the average large business will own 8Tb of data by 2012*), a globally on-line supply-chain and customer base demanding 24x7 systems and process availability, it is clear that more options are needed for business to absorb and embrace change than have historically been available.

Research from Bloor indicates that quality of outcomes – 80% of projects delivered late, over-budget or failing outright – has been very poor, and that clients perceive poor understanding of scope, data quality and changing requirements are the major causes of project failure. I’d argue that these three items are never going to be easy to nail down at the outset, but that most methods and tools fail to anticipate their impact and deliver key capabilities needed to drive up customer experience of migration.

Data migration projects have historically over-emphasized the mechanics of data mapping and transformation, while under valuing the importance of stakeholder engagement, compliance and control. In a way, it’s not hard to see why - how could you economically build a fully integrated application product to solve a problem that was perceived as transient? Building the key features needed to deliver on engagement, compliance and control is non-trivial in an environment as variable as those faced by data migration. But with a productized approach, the costs of development and maintenance of these functions can be spread over a large user base.

Whilst the technology needs to deliver a new level of automation and control at lower cost and risk, new methods must also be employed to create strong connections with the business stakeholders at all levels of the engagement. You must also ensure understanding and insight is shared and migration progress maintained regardless of changes in scope and new data sources discovered. That extends into the way vendors and internal stakeholders collaborate and interwork. A progressive approach does not afford the simple, clear-cut phases and boundaries that characterized scripted big-bang migrations. So with more complex definition of responsibility, better governance is required. The outcome is clear adhesion between the migration and the business change being implemented, and in my view that is surely worth having.

Data Migration in the Petabyte Age

Tony Sceales — Thu, 04 Jun 2009 14:06:00 GMT

Volumes of data continue to grow rapidly - we now frequently talk in terms of Petabytes – AT&T transfers 16Pb of data through its network daily, while Google processes 20Pb of data per day. When such companies upgrade their systems, reorganize or acquire new companies, they face a new scale of problem from previous generations of enterprises. And it isn’t simply a volume problem, because the data they use to manage their operations is frequently inter-related and shared across both processes and applications. They may not even directly own such data – it may belong to a partner or supplier company, or to their customers.

Best practice in data migration used to be all about speed – how fast can I load the target system database to minimize disruption to business users? Simplistically we used to ask the business what application downtime they could tolerate, then analyze the data to be migrated and divide the time available by the volume. That gave us a target load rate per object, and we then set about designing a technical strategy to achieve that rate. That often meant taking some short cuts on integrity by dropping database indexes and data validation routines, and sorting incoming data to fit physical storage algorithms. A number of market products developed to achieve massively parallel processing to maximize the speeds such approaches could achieve.

Datamigrationpro has published an excellent article by Dylan Jones on the Load Rate Bottleneck http://datamigrationpro.squarespace.com/data-migration-articles/2008/4/16/how-to-avoid-a-data-migration-traffic-jam-the-load-rate-bott.html in which he explains the challenge posed by proprietary software applications requiring data to be loaded via a standard gateway or API (Application Programming Interface). Whilst this guarantees the integrity of the data being loaded, it does so at the expense of speed. That’s because of two major factors, namely the validation of the incoming data against existing data already in the system to avoid duplication and ensure required predecessors exist, and also the overhead of working at the application layer rather than directly into the database management software. Together these can slow load rates up by several orders of magnitude relative to direct database storage.

Modern enterprises are no longer prepared to take risks on the integrity of their data – personal penalties to Directors for failing to safeguard private and commercial data ensure this. Neither do they have the luxury of downtime to load data – they run 24x7 operations and there is barely time to change a fuse in the datacenter, let alone perform a complex upgrade or migration. It must follow that new approaches are needed to support real-time migration and allow this to happen in parallel with normal operations.

In designing such approaches we need to be cognizant that the migration process needs to include a strategy for accessing the data to be migrated – commonly from multiple source applications, correlating and cleansing this and maintaining explicit and centralized control and visibility throughout. What looks like a perfect strategy for loading data often conflicts directly with the optimal design for obtaining the source data. Add to that the reluctance most enterprises feel about touching the source applications to allow access to the data – not surprisingly given the great age and fragility of many systems, and a single-event migration starts to look like a weak solution for many projects.

In considering volume we also need to distinguish between standing data and transactional data – the former generally qualifies how the latter will be processed, will be much lower in volume and has a much longer lifespan. It is particularly important therefore to take a very critical view of what the organization actually needs to take forward into the next generation architecture. Far too often requirements state that all current and historical data must be migrated, resulting in months of additional work to massage ancient data or customizing an otherwise pristine target application to cope with very little real business value. Statutory requirements to maintain financial records over several years can often be met with sensible off-line or summary solutions.