Companies are relying on data more than ever before, especially following the AI boom. Industries have integrated AI into their services to improve how they gather and share information. However, whatever data we feed into information systems is what AI and other data-sharing systems send back to us. This use case stresses the need for good-quality data and how bad data can significantly impact decisions.
A 2020 Gartner research report estimated that organizations lose up to $12.9 million each year due to poor data quality, emphasizing the need for an effective data governance process that ultimately improves data quality. This article discusses bad data, what it looks like, how it creeps up on systems, the cost to businesses, and some preventive measures organizations can take to tackle it.
What Makes Data Bad?
Bad data refers to any data in a dataset that is inaccurate, inconsistent, duplicated, incomplete, or outdated. The direct relationship between data quality and quality of business intelligence means that using bad data leads to severe business consequences like reduced revenue from missed marketing opportunities, reputational damage, reduced customer loyalty, and, in some cases, legal fees from non-compliance with data standards.
Types of Bad Data
Bad data constitutes any data that can compromise the quality of decision-making. They fall into any of these types:
- Inaccurate data: This data is wrong and results from faulty data collection systems or data entry errors. For example, replacing/missing a letter in a customer name during entry.
- Inconsistent data: Data in this category are often the same, but inconsistent across formats or units. For example, the logistics team stores weights in kilograms, while the marketing team stores them in pounds. Inconsistent data often arises in siloed systems with poor communication, where there is a lack of standardized policies guarding the collection and usage of data. This inconsistency causes deadlocks when it’s time to process and use this data, reducing operational inefficiency.
- Incomplete data: This data lacks values that offer a complete picture of an attribute, and hence cannot be trusted to present a comprehensive view of the dataset. For example, including customer profiles that are missing an age and location value cannot be trusted to create buyer personas when creating targeted marketing strategies.
- Duplicated data: This type appears when a value appears more than once in a dataset, which can skew the view of the entire dataset.
- Unstructured data: This is data in its raw, unprocessed format. Data in this pure state is unusable and requires the right tools and processes to process it before usage.
- Irrelevant data: This kind of data is not valuable in the dataset, and when processed with other types of data, it increases processing time and wastes resources.
- Outdated data: Data that is old and doesn’t present an updated view of an attribute. For example, in fast-paced industries, finance traders in high-frequency and algorithmic trades rely on immediate, automatic updates on data, as outdated data presents a stale view of the markets, leading to inaccurate plays and sometimes significant losses.
What Causes Bad Data?
Bad data often arrives in systems as part of raw data during the data sourcing process, which is why businesses employ different processing techniques to remove this bad data. However, bad data can still persist and make its way to business use cases — sometimes via human oversights or use of ineffective technology.
Human Errors
Repetitive, cumbersome tasks like data entry are common ways for bad data to enter your system. For example, a staff member misspelling names and/or addresses when entering customer information into a spreadsheet.
System Failures
With new data sources cropping up each day and the impact of data automation on data collection, finding tools to handle these large volumes and varying data types becomes essential. Sometimes, systems collating and processing this data may be faulty, thus recording inaccurate data points. For example, an ETL pipeline that ingests data from multiple sources may perform below capacity when data exceeds a specific volume, resulting in bad data moving to the next step of the cycle.
No Established Processes
Data governance policies establish standardized processes and dictate how data is handled as it moves within an organization. Without establishing these policies that guide how and what tools to use to process data, storage, and delivery, individuals treat data as personal, applying their individual methods in handling the data, thus creating inconsistent, bad data.
Inadequate Checks During Data Processing
Data processing employs a series of steps to transform raw data into a clean, usable format for analytics or other operations. However, without checks at each processing step to validate the data received, bad data can make its way to analytics.
Your Systems and Processes are Outdated
As the need for data increases, so do data sources and volumes, which calls for tools and technology that can handle this variety and volume. Using outdated tools that are unable to fetch and process data inevitably corrupts your data.
Cost of Bad Data Quality
With data being the foundation on which strategy planning, business intelligence, and marketing tactics are built, using bad data often comes with detrimental effects, like increased costs from changing strategies to fit, operational inefficiencies from inconsistencies in data, and in some cases, lawsuits.
Missed Business Opportunities
When bad data makes its way into delivery mediums like analytics, the quality of insights becomes inaccurate and often results in missed opportunities. For example, sending your latest promotional newsletter to a customer database filled with bad data means a majority of your emails will return as bounced or unopened.
Operations Become Frustrating
Businesses relying on data to keep daily operations functional and efficient will face problems if bad data is used. Picture a logistics company that relies on frequent updates to its database system to keep its users informed on the progress of their orders out for delivery. Incorrect or incomplete data means customers calling for updates or clarification, meaning the need for more customer support and increased frustrations.
Financial Losses
High-frequency, fast-paced industries like the stock market heavily rely on data to make split-second decisions on trades. Using stale/outdated data for cases like this provides an incorrect picture of the market at the time, which can lead to financial losses to the tune of millions.
Loss of Customer Trust and Loyalty
Interacting with users based on information from bad data causes a gap in communication, confusion, and a gradual loss of trust in your business. For example:
- Sending newsletters to customers about a new offer, but addressing them with the wrong name.
- Wrong updates on the progress of an order, causing delays and confusion.
Repeated errors like these damage brand reputation and slowly reduce customer engagement over time.
Regulatory and Compliance Issues
Heavily regulated industries like health and finance deal with personally identifiable information (PII) and are subject to data protection laws set by standardized bodies like HIPAA and GDPR. For example, the GDPR data minimization law restricts the collection and use of personal data to that for which it was collected and nothing more, hence including irrelevant data may violate this law, attracting high legal fees, penalties and damage to business reputation.
Real Life Examples of How Bad Data Affects Systems
So, when bad data finds its way to business decisions, what does the business impact look like? Let’s dive a bit into two real-life cases, how they happened and the effects on these businesses.
Uber’s $45 Million Commission Payments
In 2017, reports emerged that Uber made their commission deductions from their drivers’ gross fare, instead of the net fare, thereby taking over 2.6% more than its terms and conditions allowed. Uber made provisions to pay these backlogs, plus over 9% in annual interest, and with over thousands of drivers affected, the Wall Street Journal estimated the costs to be at least $45 million.
Equifax Inaccurate Credit Score
Between March 17 and April 6, 2022, a coding issue on Equifax’s legacy on-premises server resulted in the issuance of inaccurate credit scores to over 300,000 customers. This score was off by 20 points or more, enough to have loan applications rejected or affect interest rates.
An exposé by The Wall Street Journal on this issue led to a 5% fall in Equifax’s stock price, along with a class-action lawsuit led by one of the affected customers, who was denied an auto loan.
How to Prevent Bad Data
Data enables business intelligence; hence, the adverse effects of using bad data mean that businesses must be willing and ready to invest in and implement practices to prevent it from being used in making decisions. Investing in efficient tools and constant monitoring are some common methods used by data systems.
Constant Monitoring
Constant monitoring of data as it moves between systems enables monitoring and alerting systems to catch any inconsistencies or bad data before its usage. Having validation checks at each step ensures data passes a set of predetermined criteria, which helps filter out potential bad data before it passes on to the next step.
Invest in Your Talent
Investing in training skilled talent on the importance of data quality and best practices to follow when handling data can reduce the occurrence of bad data in your systems.
Invest in the Right Tools and Technology
Allocate budget and resources into modern data management tools designed to handle large datasets while ensuring high data quality. However, before investing in these tools, it is essential to explore their functionalities to confirm they meet your data use case.
Implement Proper Data Governance
Establishing good data governance policies creates a standard that defines how data is collected, cleaned and integrated into systems, which eliminates inconsistencies and variations that can cause bad data.
Automatically Clean Data
Data cleansing is the first step after data collection, and helps to remove inconsistencies in data that affect its quality. Integrating and automating data cleaning ensures that bad data is detected and removed before entering the next stage of data processing.
Always Review and Refine Your Data Management Process
Data management process is never a one-time and done process, but requires constant review to refine and update processes, when needed.
Conclusion
The garbage-in, garbage-out principle highly applies to how bad data affects the quality of your business decisions. Bad data includes any data that is incorrect, irrelevant, inconsistent or outdated and using this data for business intelligence can lead to millions in losses, reputation damage, or legal cases in some situations, as seen with Uber and Equifax.
Key takeaways:
- Bad data is incomplete, inconsistent, duplicated, incorrect, and severely impacts business outcomes.
- Using bad data for analytics leads to incorrect strategies that negatively affect revenue and customer trust.
- Using old systems, inadequate checks, and human entry errors are some ways bad data can find its way into your data systems.
- Employing the right tools, training your staff on data processing best practices, and constant monitoring are some ways to prevent bad data.
Bad data can occur due to human errors, using old systems and practices, and not implementing validation checks to ensure data meets the needed requirements before being marked safe for use. Following best practices like using trained talent and the right technology, automatically cleaning data after ingestion while employing constant monitoring and checks, can help catch and remove bad data, ensuring that only high-quality data is available for business purposes.
Frequently Asked Questions
When is data considered bad?
Data is considered bad when it reduces the quality of business decision-making, like using the wrong user names or addresses.
How to identify bad data?
Some telltale signs you are using bad data include error rates in emails, operational friction, and constant incorrect market predictions.
How do I fix bad data?
You can fix bad data by ensuring data is thoroughly processed with the right tools and talent, and constantly monitoring how data flows within the system, from ingestion to delivery.
