Monday, 4 January 2010

How Clean is Your Data?

In a recent client engagement, I was amazed at the quality of data that had apparently been cleaned and verified.

In preparing the data for import to the platform, it was very clear that the companies that had cleansed the data, did nothing to prepare the data in a manner that could be easily managed in a proper database. They had merely validated details.

If you have old data that you want to re-qualify and re-use, having an understanding of some simple data principles will guide you to a correct choice of provider for your data cleansing. Ask to see a sample of the format of data to be returned, and if necessary specify the field structure you want the data returned in. Remember, as soon as the data is recorded it starts to age, so you want to be able to import and use that data ASAP and start to make it deliver value to your organisation.

Some Simple Actions Will Deliver Great Value

You should take care of your data and the way in which it is structured. When data is managed correctly it can be manipulated and analysed to deliver tangible and reliable results that enable sound decisions and actions to be taken. Many organisations spend a small fortune on solutions to record customer information but where they fail is in training users in the basic principles of databases and data entry.

Here is my take on some of the fundamental principles you and your staff should understand about data.
  1. Decide on the purpose of each piece of data to be recorded. Why do you need it and what do you want to be able to do with it? (Think - billing, segmentation, forecasting, performance - to name a few.)
  2. How do you want to record that data? It is fixed or variable? This will increase the accuracy of your data. For example, there are only so many salutations (Mr, Ms, Miss, Mrs, Dr, etc..), so a fixed picklist list of all the potential variables can be used, whereas, last names will be different, so a variable field would be used to record this data.
  3. Do you need mandatory data? Some data is useless unless other data is also recorded - there is no use in just recording a first name and not a last name. (Have you ever had the classic sticky note 'call John' with no other detail?). If you need certain data to be mandatory, justify it to the users either in training or in help scripting. Conversely, too much mandatory data requirements will affect user adoption and usage. (You can complete a mandatory field just by putting in a full stop!)
  4. Data ages from the moment it is recorded. Adopt a policy with users to continuously update data as a matter of habit, not as a matter of frequency. If a system is used continuously with information updated as it is discovered in the course of doing business, it will continuously deliver value. (e.g. - update a job title if someone gets promoted, or change their company name if they have changed jobs).
  5. Purge your data with a vengeance! Sometimes we need to record data that is very specific and time related for a campaign. Once you have finished with data recorded for an old campaign, etc. delete it out of your pick lists and product tables. (e.g. - Product interest in Vancouver 2010 Winter Olympics apparel, will not be relevant in March 2010 necessarily)
So, to your users, engender them to understand the purpose of each data field and how it should be completed. (Don't cram an entire address into the first field. This will prevent you searching and segmenting your data accurately). Get them to understand that even when they are creating a simple adhoc data collection spreadsheet, to record the data in a structured format, so that it can be manipulated and used easily for other purposes.

Whether you are managing a spreadsheet database or a major ERP or CRM application, data accuracy is king. Keep it clean and keep it tidy - it's everyone's responsibility, not the IT or database manager's.

/* Use this with templates/template-twocol.html */