Last week John Morris posted a detailed account of how to rescue a sinking data migration project, that post provided some excellent stablising techniques John has developed over many years.

This week John closes out the series with a look at how we can re-plan a failing project and successfully "mop-up" on those outstanding tasks that are so important but often neglected.

How to rescue a failing data migration project [Part 2 - Plan and Mop-up]

Planning

Carrying on from our earlier post, the next task we need to address is buying some time to create a new plan.

With your checklist in hand you should now have a feel for where the programme is failing and will not be confusing causes with symptoms. I recognise that buying time is easier for us in iergo because we are outsiders and are only brought in when management is prepared to acknowledge they have a programme that is failing.

We also have the benefit of having a tightly integrated set of criteria, the general outline of which is above, linked to industry measures so we can show degrees of risk and likely outcomes in terms of financial and time over-runs. We also don’t own the problem but have the impenetrable sheen of disinterested professionals.

If you are reading this, however and you have been involved in the programme from the beginning then requesting extensions and sudden changes of tack might be suicidally counter-productive. I’m afraid I leave it to you to decide how far you share your insights with your senior management.

However, you now have a detailed analysis of the shortcomings. I don’t suppose you really need this piece of advice but you have to stop chasing the programme and get back in charge.

The following are the classic steps that we find most failing programmes need to correct:

• Data Architecture - Get the list of Legacy Data Stores under change control. Does everyone on the programme know where to go to see a full list of the data sources? Is it easy to see which data sources in the plethora that are available to you contain which of your Conceptual Entities? Is it easy to find out which data sources will be expected to have matching Conceptual Entities for the migration (e.g. can you see which application within billing, has to have a matching customer reference to which application with sales)? Do you have an agreed Legacy Data Store Model against which each candidate legacy data store can be measured for consistency? Are you performing formal Landscape Analysis or are you going for the "Poke And Hope" approach (i.e. throw stuff at the target and see what fails)?
• Data Quality Rules - Get the list of data issues under change control and get them into your Data Quality Rules process. In panic mode the number of issues can escalate rapidly but what we need at this stage is focus and prioritisation. Most projects only track the really big issues on an issue register leaving hundreds of smaller issues to go unnoticed. These issues can rapidly eat up project hours. Get each data issue logged as a data quality rule so that the relevant Technical Systems Expert and Business Domain Expert are closely involved and working together for a solution.
• Key Data Stakeholders - These people are essential and are often lacking on a failing migration. These people can help you prioritise and resolve the data issues and you must tactfully convince them to take responsibility for the data in their area. This is not easy if they have been used to being consulted but not having any responsibility to the programme for delivery.
• Delivery - Focus on consulting, not fixing. In panic mode there is often the temptation to fix every issue that arises. Stop. Accept that you have missed important data gaps and begin to educate sponsors and stakeholders that you will be delivering the project iteratively from this point on using a prioritised process.
• Do the do-able – Your analysis will tell you where you are going wrong but at this point you can’t hope to fix everything. It is just not technically or politically feasible. Concentrate on iterative increments. Accept that knowing who the Key Data Stakeholders are and getting them to accept their responsibilities are two different things. Using the Data Quality Rules process will bring them into the project. You will not at this point in time be allowed to perform a formal SRP creation process, but be alert to Reasons To Say No as they come out in conversation. Never let one slide by. Always ask what it would take to close it out. Record it. Build up trust.

Most data migration projects fail because they lack a cohesive data migration methodology that is converted into an effective plan. Create a checklist based on the pointers in this post and the previous post to see how close you are to industry best practice then amend your plan accordingly.

Working in a far more controlled manner after implementing our stabilisation phase we can start to plan releases in an iterative fashion that deliver measurable improvements to the current situation and give benefits back into the business.

Here are some final activities in this phase:

• Prioritisation - start prioritising faults and create realistic estimates of how long these will take to resolve. Data quality prioritisation is also critical, we always say "No enterprise wants, needs or will pay for perfect quality data" however in the rush to recover the project the temptation to load data will be great. If your data is sub-standard, how will it be corrected in a later iteration? How will it affect the load process? Have these risks been communicated to the Data Stakeholders? Your project may not necessarily need perfect quality data but you do need to understand and communicate the risks
• Change control - tighten up change control and configuration management procedures. The fire-fighting mentality of failing projects has to be transformed into a stakeholder-driven release plan with clearly defined deliverables that the business can plan for

Mop-up

In a well-managed migration there are a number of activities that would have been completed as a matter of course but when in failure mode can be forgotten. 

Our mop-up phase ensures we focus on the critical migration tasks but are mindful of our post-migration house-keeping duties. At this stage there are probably a wealth of best-practices that have been omitted such as key data stakeholder management, data quality rules, legacy data store management and many more.

Here we introduce 3 other key project activities that are typically lacking but still need to be addressed:

• System Retirement Policies. We cover these in great deal in the book (ed: this 25 tips for system retirement article is also useful) but in most struggling projects they will have been omitted and this can be a real challenge. The best solution is to implement a system retirement policy from the outset and get this logged on the programme issue log so it has full attention. If you have no system retirement policies in place then you will certainly need to exercise some political sensitivity as system owners may be antagonistic if they've not been involved.
• Key Business Data Areas. Chances are you've now got a lack of knowledge on exactly where your consistency problems exist between data stores. It's unlikely you will be given sufficient time to completely re-plan and do this correctly but certainly look to address consistency issues across your legacy data areas in subsequent iterations
• Reality Check. Again, it's unlikely you will have sufficient time to analyse the accuracy of your migrated data compared to business reality. However, that doesn't mean it should be forgotten. Use the legacy data stores and business experts you found in the stabilisation phase to help you corroborate the data. Ensure the data stakeholders are responsible for prioritising remedial work post implementation and start to transition your data quality rules framework (set up in the previous phases) over to the enterprise for ongoing data quality management

In Summary

The lesson here is is a rather obvious one that by adopting data migration best-practices there is simply no need for your project to slip into failure mode. Use the checklists in this series to see how close your project is to best-practice even if you are yet to experience slippage.

By following the advice on this site, in my book and on the iergo and Data Migration Pro websites you should be able to assess whether your project has the correct approach and resources to be successful.

One of the main causes of failure is when the business takes a back-seat in the data migration.

Is your project IT-heavy?

Are the right stakeholders, domain experts and business sponsors engaged and active in your project?

Integrating the enterprise into the project is probably the most important driver for migration success so my advice on a failing project is to at the very least focus on this critical area.

Resources

How to rescue a failing data migration project [Part 1 - Stabilise]

Iergo Free Data Migration Training Centre

Practical Data Migration by John Morris

Presentation that includes the 4 golden rules of data migration

Data migration project checklist - a template for more effective data migration planning

The data migration go-live strategy - what is it and why does it matter?

Does your data migration strategy have an effective plan for systems retirement?