Data integration and data migration are, at their core, quite different beasts.

Data migration is typically a one-stop affair. You get the business and data into shape, plan your migration strategy and if you've done a decent job move your data effortlessly into that shiny new target system.

Data integration is an important aspect of data migration projects but historically data integration tools have really spun out of the warehouse building phases of the nineties. As a result they were mostly designed to accomplish repetitive tasks, very quickly in an effort to load a static target.

There are now a few tools coming into the marketplace that are purely aimed at migration. Some of the bigger integration vendors (eg. Informatica) are starting to focus their messaging, licensing and product bundling on data migration too so if the big guns are moving their sights that could be a warning to the other integration vendors out there who may be missing a trick.

Data migration requires a great deal of error management, synchronisation, operational control, forecasting and planning. You need to bring together a myriad of workgroups and provide great reporting, not always the remit of ETL or data integration tools.

So, does the industry need a magic quadrant solely for data migration tools to reflect this new industry that has always been there but is now slowly coming out of the shadows?

Welcome your thoughts on the subject.

I'd vote for no - a good data integration tool needs to be judged on its ability to do DW, migration and real time and breaking it out into sub categories could be confusing. Most data warehouse projects come with an initial migration so any tool that populates a data warehouse also needs to be a good migration tool.

John Platten says...
Posted 21 April 2008
I half agree with Vincent - Based on an economic argument.

Businesses want value for money from their staffing and software tool investments, so that:

It is possible in a major corporate environment that investment in migration specifically will pay off time and again if they have a constant inflow of such work e.g. through mergers and acquisitions or ongoing system replacement and innovation.

Turning to software houses, integrators and consultancies it is even easier to see how they can specialise in migration specific tools and people as their inflow of projects comes from a broader client base, each with it's own ebb and flow of projects a number of which will feature migration rather than integration.

This leaves the largest sector, the other companies in which there are only two viable economic arguments: The tools and people must either do double service on integration and migration to be worth purchasing outright or the migration should be outsourced and the tools and expertise rented in for discreet periods.

Time will tell how large each of these three sectors (corporate, consultancy, other) proves to be and which sort of tool vendor offerings they will support economically. Traditionally the more ground a tool can cover the more momentum it will gather and this may actively drive the vendors towards the dual purpose tools that Vincent identifies.

We can see a parallel in data warehousing where Business Objects is popular precisely because it is not as specialised as other OLAP tools and can be thrown across almost any shape of database. This becames both its strength and its weakness, so that a place remained for specialist tools that for instance mine data better or drill deeper on the fly. Business Objects did not in any way wipe out these other tools and they continue to coexist and prosper with clients who want more from their information. I foresee the migration specialist vs. integration/migration dual tools market potentially going the same way.

A lesson that can also be drawn from the history of data warehousing is that the ability of specialist tools to prosper will largely be determined by their reception amongst Data Migration Professionals specifically and particularly those who are respected independents or consultancy based opinion formers. Without one Ralph Kimball I dare say we might be looking at Business Objects with everything in DW and BI.

So will our market also support both: Wide coverage popular tools and specialist migration-only toolsets that do the job "super-properly"?

The one thing I can be fairly certain about is that the next few years are the period in which the question will be answered. Ask us again in two years time Dylan, and I am sure we will all know the answer!

(Posted 27 May 2008)

OK, so coming from Celona Technologies, a Data Migration specialist company, you'll all expect me to argue for a DM specific quadrant; so I shalln't disappoint you.

Of course it doesn't make sense to create new markets (and hence quadrants) for every problem that exists. That road leads to anarchy. The time to separate is when the problem becomes big or frequent enough and when there is clear water between capabilities/requirements; and I think both conditions are required to be satisfied.

I think we are there now:
1) Market size
Bloor estimates DM is a $5 Bn market, only counting projects over $1m. Looking at the part of the estimate that I know in detail its clearly under well the real size. If this error is projected across the rest of the sizing then we're looking at something much bigger.

2) Distinctive Requirements
This obviously requires longer to discuss in full, so I'll just put one case down and leave it for another day.

Data Migration in complex environments needs a range of strategies to be successful, often within a single project. For example sometimes a "don't migrate" (just sync key data) is right at first, event-driven is sometimes the most suitable and then Celona, for example, also classifies, incremental, bulk-load and big-bang. Without these approaches projects will often fail, we argue. And they do - Bloor has this at over 80% of projects, but an example, I recently got first hand knowledge of, was of the business users preventing any more "successful" migrations being delivered by IT with the old strategy of big-bang.

Therefore I conclude (albeit briefly) that DM is big enough and distinct enough to warrant its own market quadrant. But I would say that, wouldn't I!

(Posted 24 June 2008)

Many thanks to Bloor research and in particular Philip Howard for recognising the industry need and producing the first independent view of the "Data Migration" and "Content Migration" (as in unstructured data) industry.

Published on the Bloor site but will also be at http://www.celona.com very shortly.