Promoting DQ Tools for Data Migration Projects

One of our members recently had a tough time convincing her peers and sponsors of the need for a DQ tool on their forthcoming data migration project.

She had been pitching the benefits primarily of cleanse, citing the fact their migration tool had limited DQ capabilities so an additional toolset to manage the more complex transformations was required.

The source systems were largely consumer addresses, their business being B2C in the financial sector. The argument she came up against was that they already had a profiling tool so why couldn't that help solve the problem?

Here are some of the additional areas that DQ tools can help increase the success rate of any data migration project.

 

Cost and Scope Analysis

If you work in the data migration industry then you are no doubt aware that most migrations are poorly forecasted both in terms of cost and duration.

In one case I witnessed an organisation using function points as a measure for effort and cost. This was without even looking at the data! Such practices are common where there is a poor understanding of the entire migration process.

Having a DQ tool that provides data discovery, profiling and cleanse early in the project means that you can perform some really essential activities:

• Reality check on the actual number of business objects versus the count of variations in the database (for example, one business thought it would need to build 7200 equipment classifications, de-duplication and cleansing discovered they actually only 750, this meant a 10x reduction in analysis and development)
• Better metrics for load volumes, understanding exactly how many objects to load is critical for determining your go-live strategy and window of opportunity for migration, this needs to be understood at the very outset of the project
• Improved resource allocation by understanding what type of data defects will require automated, project team or business user intervention you have a much clearer idea of when these need to be deployed, critical for giving the business prior warning

Migration Simulation

One of the most useful techniques I have practiced in recent years is to perform a "virtual migration" of the legacy data to the target environment. This approach effectively de-risks the migration well in advance of committing a major spend.

The key to simulating the migration is to understand your key business building blocks, those business objects that are pivotal to success. Customer, location, order, equipment - define these in a simple conceptual and logical model first. Then define their physical relationships by using data discovery which underpins the entire migration process. Using our DQ platform we can then simulate the likely issues when consolidating disparate data in our legacy world and performing basic transformations before loading into a dummy target environment.

Using this simple technique prevented one organisation moving ahead with a migration that was simply not viable under traditional approaches, it therefore provides excellent intelligence gathering.

Data Quality Rules Management

Regular readers will have seen the many discussions and tutorials we have published on the subject of managing data quality so it goes without saying that a basic data profiler is insufficient to manage the full spectrum of data quality rules. You will require additional cleansing and transformation capabilities plus a means of managing a library of possibly hundreds of different data quality rules as they are developed.

Migration Execution Sequencing

How do you know which data will be eligible for migration? DQ tools provide a vital cog in the intelligence gathering machinery because they can report on the health of data prior to migration. The alternative is to run the migration and handle the fallout of records in error. Not advisable so make use of those DQ tools to report on data that needs to be postponed or handled as a special case.

Independent Data Migration Validation

How do you know if your data migration has successfully moved data across to the target environment? A DQ tool is a great way to independently vet that what was migrated correctly correlates to the legacy environment.

Ongoing Data Quality Assurance

At this point you should have a vast armoury of data quality rules and metrics that were created during the project to help you validate both the legacy and target environment. Why not benefit from this resource and launch an ongoing data quality assurance programme?

There will probably never be a time where such detailed knowledge about the application, database, data and functionality is freely available.

What other uses can you present for benefiting from DQ tools during a data migration project? Why not add your comments below.

 

Useful Resources

 

Data migration project checklist - a template for more effective data migration planning

What is landscape analysis and why is it important to your data migration? (Part 1 - Overview)

How to create a data quality rules management repository (Part 1 - What are data quality rules?)

How to create a data quality rules management repository (Part 2 - Creating the application)

Ensuring Data Quality in Data Conversion

Comments

[John Platten]

I tried to post a reference but it didn't quite work - so here is the link direct:

Why use DQ Tools? (a management justification)

In this brief article I look at DQ tool justifications that appeal to the Programme Manager rather than the data migration specialists. ...They after all are doing the buying!

Reproduced below but lacks a small diagram. Hope it helps!
----------------------------------------------------------------------------------------------------

I am often asked by line of business managers and programme managers why the use of data quality tools, now increasingly considered best practice, is important: And particularly so when they are asked to fund investment in these platforms!

The heart of this problem is that many otherwise excellent data migration leads tend to talk in technical terms when it’s a simple business case that is required. It is not necessary to explain the need in Data Quality terms at all, as the benefits can be explained purely in terms of risk management and planning:

Traditional Data Migration can be visualised in terms of a project plan, in which we would on-board the team, progress with setup, start moving small items of data around, begin gradually to integrate these into larger sets and then finally move to a fuller picture encompassing the whole starting data set and also the target. However we notice when we look at this plan that any integration problems would only be realised a significant way into the project, when a lot of money has been spent and promises have been made on the size of the team, the budget and even the feasibility.

It’s tempting when budgeting to look at the data movement as an excercise unto itself, but if the data does not arrive, the system stays empty and an entire business change progamme may founder. Workflows will have been created, hardware has been ordered, training has been undertaken, managers have been moved and backfilled, and other strategic business objectives have been put aside - all for nought. Worst still, if the data is flawed and the business trust to luck instead of calling a “no-go”, they may find themselves stranded on an unworkable platform, unable to operate processes effectively and in danger of damaging customer and investor confidence directly.

So that:

Data Migration is a critical dependency of wider change and implementaton programmes that has a tendency to generate long thin plans with a high liklihood of trouble towards their end-points.

As managers what are we going to do about that? I don’t mean the Data Migration Lead, I mean you, the key sponsor with his/her career on the line.

With any other such problem you would seek to bring the problem area forwards in the timeline, work on it in parallel, investigate the extent of the risk and from there form an opinion sooner rather than later so that countermeasures can be deployed and mitigation put in place, even to the point of cancellation if teh endeavour appears too risky for the organisation to procede. …This is Project management bread and butter.

This is what DQ Tools allow your Migration Lead - the means of performing that transformation of project time and risk; Snapping the problem elements away from the troublesome end section and bringing them forwards in the timeline, closer to programme initiation and before significant damage can occur.

[See original article for illustrative image]

The technical detail of how this works is far less important than the impact in risk, planning and certainty, allowing the project to forge ahead with confidence rather than bravado.

(c) John Platten and DataMigrationPro

Vivamex Limited
SapDataMigration.co.uk

Tel: 0794 109 5082

[Deenadayalan Venkatasamy]

What type of DQ tools are available in the market? Any names?

Thanks,
Daya

[Dylan Jones (Founder)]

Hi Daya

We have a list on our sister site:

http://www.dataqualitypro.com/data-quality-technology/

Plus take a look at the Bloor Market updates we posted on Data Quality Pro, that also lists the products by type.

Any more information required, just let me know.

Dylan (Editor)