Data Management Platform Migration: Reporting Considerations and Guide

Introduction

Migrating to a new data management platform is a complex endeavor that involves not only the transfer of data to a new system, but also the migration of business processes and reports. For this reason, it is essential that proper consideration is given to reporting during the migration process.

After recently helping to support such a migration, I found my team’s expertise in reporting made us well positioned to take ownership of assessing and migrating existing reports. Our work allowed us to certify that the new data management platform met the reporting requirements of the business.

The guide that follows will provide an overview of steps we took and other learnings from the migration process with respect to reporting.

Understand the Existing Report Landscape

Catalog and Audit All Legacy Reports

While tedious, creating a report catalog assists greatly in planning and later executing the report migration. This is also an opportunity to audit the existing report landscape and identify reports that need enhancements or that can be deprecated.

Start by making a list of all existing reports and identifying their dependencies. It can also be helpful to group them into functional topics such as sales, web, marketing, etc. Then, assess each report’s criticality and determine their prioritization in the migration timeline based on their value to the business and their usage statistics.

Below are some points to consider as you create your catalog and complete your report audit:

• Where is the report hosted or generated (BI and Reporting tools such as Tableau, Salesforce, Excel or other software)?
• Who are the report owners and users?
• How often are the reports used/accessed?
• At a high level, what are the use cases for each report? What business questions do they answer?
• How often are the reports and underlying data sources refreshed?
• How many months or years of historical data are needed in the report?
• What are the data sources used in each report?
• What other dependencies does each report have (such as automated or manual tasks)?

Analyze the Report Data Sources

Once you have your catalog of reports and have identified their data sources, it’s time to dig deeper into the underlying data.

Create documentation of every data source and the specific tables and fields used in each report. This will be essential to later mapping your legacy data to the target data in the new platform. It is also helpful to measure the volume of the data used in each report to help the engineers working on the migration understand resource requirements and mitigate performance issues or other challenges related to the data size.

Below are some points to consider as you document each report’s data source:

• For every field in the data source, what is the corresponding name in the report?
• What is the query behind each data source? How is the data source created?
• What dependencies do the data sources in the report have (such as other data sources, intermediate tables, scheduled queries or jobs, data transfers)?
• What critical data transformations are being done and in which step of the workflow are they being done?
• How many rows and columns does each table have?  What is the table’s size in bytes?

Taking the time to document the reports and their data sources will help equip you to work together with the engineers to make sure report data requirements are met in the new platform and decide on which reports to prioritize in the migration.

Map the Legacy Report Data Sources to the Target Data Sources

Create a cross-reference document

As data becomes available in the new platform, you will want to create a cross-reference document or map that matches the fields from the legacy report data sources to their counterparts in the new platform.

Include the following for both the legacy data and target data sources:

• Legacy Table and Target Table
• Legacy Field Name and Target Field Name
• Example data for the Legacy and Target fields
• Definitions of any calculations or logic that need to be applied to each field for use in the report
• Summary of issues for each field such as missing fields, incorrect formatting, incorrect or no data returned, duplicates, anomalies, or any other issues encountered in the new platform

This exercise will help you confirm that the required report data is available in the new platform and uncover any gaps in the migration.

QA and Validation

Validation of the migrated data

The migration process is likely to be dirty and it is expected that issues with the data quality and quantity will arise.

Below are some considerations as you validate the migrated report data:

• Is there missing data (for a particular time period, blank records, missing fields)?
• Are the number of rows and columns returned as expected?
• Are the data types and formats as expected?
• Do aggregated metrics totals match the totals in the legacy data?

There are some differences between the legacy and migrated data which may be acceptable while others will need to be addressed further. In either situation, make sure you are able to explain the differences.

It will also be essential to engage with report end-users and stakeholders to validate that the migrated data and new reports meet their requirements. Their feedback will help further identify any gaps in reporting or business logic.

Maintain an Issue and Change Log

Due to the complexity of a migration, it can be easy to lose track of issues and changes that occur without keeping a log. Do yourself a favor and document all issues and changes that are made. Include status updates where applicable and note the date that the fix or change was implemented. This will help you make sure that all issues that arise in the report migration are addressed and also prevent redundancy in the future.

Conclusion

Reporting considerations are an important component of a data management platform migration effort that should not be overlooked. By using the information in this guide, you can perform a thorough report assessment and contribute to the overall success of a broader data management platform migration effort. Good luck!