Archiving Strategy: Data Relevance - SmartData Collective

We often think of the relevance of data when we want to include or exclude it from analysis or process. However, are you thinking about relavence as part of your data quality effort?

Just as you focus data quality efforts to clean existing information, there are invariably records that can’t be cleansed or enhanced. They have no value in either business analytics or business process. They are noise, similar to the noise you have when there is bad data. To save and maintain them in your database can affect your ability to accurately analyze information, continue to deflate confidence in data, and if a significant percentage of your database, will cause problems in performance and added maintenance. Developing an archival strategy as part of your data quality practice is a significant component that should not be overlooked.

Benefits of Data Relevance

Trust in data
Enables process
Accuracy of analysis
Supports decisions
Database optimization

We often think of the relevance of data when we want to include or exclude it from analysis or process. However, are you thinking about relevance as part of your data quality effort?

Benefits of Data Relevance

Trust in data
Enables process
Accuracy of analysis
Supports decisions
Database optimization

It can be tempting to simply delete records from your databases. Though, this can have a detrimental affect due to data dependencies within your databases as well as causing non-compliance in regulated environments. Instead, it is best to formulate a strategy that flags non-relevant data removing or suppressing it from user interfaces and analytics.

Components of Archiving Strategy

Data decay rates – Attributes of records that loose relevance over time. This component is a good guide on the frequency at which you will focus cleansing efforts. It also provides an indicator on when data is approaching a horizon when a record will lose its relevance. Age of the data and activity related to a record, even if a record is complete, can signify whether the data is relavant and open to archiving.
Minimum requirements of record viability – Records should continually be assessed to determine if they meet the minimum standards of use. Failure to meet minimum requirements is a leading indicator that the record is a candidate for archiving.
Relevance of record to analysis, process, decisions – If a record is not going to be used in analysis, process, or decision making, there is not need to keep it in use. This may be the case if processes have been optimized and certain information is no longer needed. Or, it could be that it was a candidate for archiving due to decay rates and minimum data requirements. Additionally, relevance may be determined when integrating systems where old records with old transaction history is not relevant to the existing or new business.
Regulatory compliance – In highly regulated environments like health care, there are standards on what you can and cannot remove. Records may not be useful in existing process, analysis, and decision making, but might be required in certification or other compliance related activities. Archiving ensures that information is not deleted from primary systems. Although, you may have to provide a mechanism that provides adequate access to data for compliance.

An archiving strategy is a critical component of data quality best practices. It will continually help you focus on improving and refining your data quality projects as well as thinking strategically about how you use and manage your data on a daily basis. Establish an archiving strategy at the forefront of your data quality initiatives and you start your efforts off on the right foot.