Data is a new buzzword. Big Data, data science, data analytics, etc. are words that surround us every day. With the abundance of data, the challenges of data quality and accessibility become more prevalent and relevant to organizations that want to use data to support decisions and create value. One question about data quality is “can we trust the data we have?” No matter what analysis we perform, it’s “garbage in, garbage out.”
This is one reason that Measurement System Analysis (MSA) is included in all Six Sigma training. Because Six Sigma is a data-driven business improvement methodology, data is used in every step of the problem-solving process, commonly following the Define-Measure-Analyze-Improve-Control (or DMAIC) framework. The goal of MSA is to ensure that the measurement system is adequate for the intended purpose. For example, a typical MSA evaluates the accuracy and precision of the data.
In science and engineering, much more comprehensive and rigorous studies of a measurement system are performed for specific purposes. For example, the US Food and Drug Administration (FDA) publishes a guidance document: Analytical Procedures and Methods Validation for Drugs and Biologics, which states
“Data must be available to establish that the analytical procedures used in testing meet proper standards of accuracy, sensitivity, specificity, and reproducibility and are suitable for their intended purpose.”
While the basic principles and methods have been available for decades, most organizations lack the expertise to apply them properly. In spite of good intentions to improve data quality, many make the mistake of sending newly trained Six Sigma Green Belts (GB’s) or Black Belts (BB’s) to conduct MSA and fix measurement system problems. The typical Six Sigma training material in MSA (even at the BB level) is severely insufficient if the trainees are not already proficient in science, statistical methods, and business management. Most GB’s and BB’s are ill-prepared to address data quality issues.
Here are just a few examples of improper use of MSA associated with Six Sigma projects.
- Starting Six Sigma projects to improve operational metrics (such as cycle time and productivity) without a general assessment of the associated measurement systems. If the business metrics are used routinely in decision making by the management, it should not be a GB’s job to question the quality of these data in their projects. It is management’s responsibility to ensure the data are collected and analyzed properly before trying to improve any metric.
- A GB is expected to conduct an MSA on a data source before a business reason or goal is specified. Is it the accuracy or precision that is of most concern and why? How accurate or precise do we want to be? MSA is not a check-box exercise and consumes organization’s time and money. The key question is “is the data or measurement system good enough for the specific purpose or question?”
- Asking a GB to conduct an MSA in the Measure phase and expecting him/her to fix any inadequacy as a part of a Six Sigma project. In most cases, changing the measurement system is a project by itself. It is out of scope of the Six Sigma project. Unless the system is so poor that it invalidates the project, the GB should pass the result from the MSA to someone responsible for the system and move on with his/her project.
- A BB tries to conduct a Gage Repeatability & Reproducibility (R&R) study on production data when a full analytical method validation is required. A typical Gage R&R only includes a few operators to study measurement variation, whereas in many processes there are far more sources of variation in the system, which requires a much more comprehensive study. This happens when the BB lacks domain expertise and advanced training in statistical methods.
To avoid such common mistakes, organizations should consider the following simple steps.
- Identify critical data and assign their respective owners
- Understand how the data are used, by whom, and for what purpose
- Decide the approach to validate the measurement systems and identify gaps
- Develop and execute plans to improve the systems
- Use data to drive continuous improvement, e.g. using Six Sigma projects
Data brings us opportunities. Is your organization ready?