Conceptually, process capability is simple.  If a process makes products that meet the customer requirements all the time (i.e. 100%), it has a high process capability.  If the process does it only 80% of the time, it is not very capable.

For quality attributes measured as continuous or variable data, many organizations use Process Capability Index (Cpk) or Process Performance Index (Ppk) as the metric for evaluation.  In my consulting work, I often observe confusion and mistakes applying the concept and associated tools, even by Quality and CI professionals.  For example,

• Mix-up of Cpk and Ppk
• Unclear whether or when process stability is a prerequisite
• Using the wrong data (sampling) or calculation
• Misinterpretation of process capability results
• Difficulty evaluating processes with non-normal data, discrete data, or binary outcomes

The root cause of this gap between this simple concept and its effective application in the real world, in my opinion, is lack of fundamental understanding of statistics by the practitioners.

Statistics

First, a process capability metric, such as Cpk, is a statistic (which is, by definition, simply a function of data).  The function is typically given as a mathematical formula.  For example, mean (or the arithmetic average) is a statistic and is the sum of all values divided by the number of values in the data set.   

The confusion between Cpk and Ppk often comes from their apparently identical formulas, with the only difference being the standard deviation used.  Cpk uses the within-subgroup variation, whereas Ppk uses the overall variation in the data.  Which index should one use in each situation?

It is important to understand that any function of data can be a statistic – whether it has any useful meaning is a different thing.  The formula itself of a statistic does not produce the meaning.  Plugging whatever existing data into a formula rarely gives the answer we want. 

To derive useful meaning from a statistic, we must first define our question or purpose and state assumptions and constraints.  Then we can identify the best statistic, gather suitable data, calculate and interpret the result. 

Enumerative and Analytic Studies

Enumerative and analytic studies¹ have two distinct purposes. 

• An enumerative (or descriptive) study is aimed to estimate some quantity in the population of interest, for example, how many defective parts are in this particular lot of product? 
• An analytic (or comparative) study tries to understand the underlying cause-system or process that generates the result, for example, why does the process generate so many defective parts?

If the goal is to decide if a particular lot of product should be accepted or rejected based on the number of defective parts, then it is appropriate to conduct an enumerative study, e.g. estimating the proportion of defectives based on inspection of a sample from the lot.  A relevant consideration is sample size vs. economic cost – more precise estimates require larger samples and therefore cost more.  In fact, a 100% inspection will give us a definite answer.  In this case, we are not concerned with why there are so many defectives, just how many.

If the goal is to determine if a process is able to produce a new lot of product at a specified quality level, it is an analytic problem because we first have to understand why (i.e. under what conditions) it produces more or fewer defectives.  Methods used in enumerative studies are inadequate to answer this question even if we measured all parts produced so far.  In contrast, control charts are a powerful analytic method that uses carefully designed samples (rational subgroups) over time to isolate the sources of variation in the process, i.e. understanding the underlying causes of the process outcome.  This understanding allows us to determine if the process is capable or needs improvement.

Cpk versus Ppk

If our goal is to understand the performance of the process in a specific period (i.e. an enumerative study), we are only concerned with the products already made, not the inherent, potential capability of the process to produce quality products in the future.  In this case, demonstration of process stability (by using control charts) is not required, and Ppk using a standard deviation that represents the overall variability from the period is appropriate.  

If our goal is to plan for production, which involves estimating product quality in future lots, the process capability analysis is an analytic study.  Because we cannot predict what a process will produce with confidence if it is not stable, demonstration of process stability is required before estimating process capability. 

If the process is stable, there is no difference between within-subgroup variation (which is used for Cpk) and overall variation (which is used for Ppk), except estimation errors. Therefore, Cpk and Ppk are equivalent.

If the process is not stable, the overall standard deviation is greater than the within-subgroup variation — Ppk is less than Cpk as expected.  However, Ppk is not a reliable measure of future performance because an unstable process is unpredictable.  If (a big IF) the subgroup is properly designed, the within-subgroup variation is stable and Cpk can be interpreted as the potential process capability if all special causes are eliminated.  In practice, the subgroup is often not designed or specified thoughtfully, making interpreting Cpk difficult.

In summary, process capability analysis requires good understanding of statistical concepts and clearly defined goals.  Interested practitioners should peruse many books and articles on this topic.  I hope the brief discussion here helps clarify some concepts. 

1. Deming included a chapter “Distinction between Enumerative and Analytic Studies” in his book Some Theory of Sampling (1950).

What makes a goal achievable?  In my work as a Continuous Improvement (CI) coach and consultant, I have seen some common practices setting a numerical goal using, for example

1. A target set by management, e.g. a productivity standard for the site
2. Customer requirements, e.g. a minimum process capability
3. Some benchmark value from a similar process
4. A number with sufficient business benefit, e.g. 10% improvement

At the first glance, these methods seem reasonable.  In practice, they are problematic for two reasons.

First, the goals are based on what is desirable, not sound understanding of the opportunity using data.  How do we know if a desirable goal is achievable?   In many organizations, a numerical goal is “set in stone” when the project starts; failing to achieve the goal can have potential career repercussions.  While management tends to aim for aggressive targets, the project leaders are more concerned with the risk of failing to achieve them.  They prefer a more “realistic” target that can be met or even exceeded and negotiate with the sponsors to make the desirable target a “stretch” goal.  In the end, no one knows what the real improvement opportunity is.

Secondly, the practices create a mindset and behavior inconducive to the CI culture.  I have seen too many organizations’ Lean, Six Sigma, or other CI initiatives focus only on training and project execution.  They fail to build CI into their daily decisions, operations, and organization’s culture.  Quality improvement cannot be accomplished by projects alone – numerous incremental improvement opportunities exist in routine activities outside any project.  Projects, by their nature, are of a limited duration and are merely one mechanism or component of continuous improvement. Most improvement does not require a project.  Depending on projects to improve a process is a misunderstanding of CI, reinforces reactive (firefighting) behavior, and sends a wrong message to the organization that improvement is achieved through projects, and even worse, by specialists.

Creating a project with only a desired target leads to high uncertainty in project scope, resources, and timelines – a lot of waste. 

To be effective, a CI project should have a specific opportunity identified based on systematic analysis of the process.  Furthermore, the opportunity is realized through a project only if it requires additional and/or specialized resources; otherwise, the improvement should be carried out within routine activities by the responsible people in collaboration. 

What kind of systematic analysis should we perform to identify the opportunities?

One powerful analysis is related to process stability.  It requires our understanding of the nature and sources of variation in a process or system.  In a stable process, there is only common cause variation – its performance is predictable.  If a process is not stable, there exists special cause variation — its performance is not predictable.  Depending on process stability, the opportunity for improvement and the approach are distinct. 

The first question I ask about the goal of any improvement project is “Is the current performance unexpected?”  In other words, is the process performing as predicted?  No project should start without answering this question satisfactorily in terms of process stability.  Most often the answer is something like “We don’t really know but we want something better.”  If you don’t know where you are, how do you get to where you want to be?  This is a typical symptom of a project driven by the desirability rather than a specific opportunity based on analysis.  If the process stability was examined, most likely the first step toward improvement would be to understand and reduce process variation, which does not need a project.

For people familiar with Deming’s 14 Points for Management, I have said nothing new.  I merely touched point 11 “Eliminate management by numbers, numerical goals.”  His original words¹ are illustrative.

“If you have a stable system, then there is no use to specify a goal.  You will get whatever the system will deliver.  A goal beyond the capability of the system will not be reached.”

“If you have not a stable system, then there is again no point in setting a goal.  There is no way to know what the system will produce: it has no capability.”

A goal statement that sounds SMART does not make a project smart.  A project devoid of true improvement opportunity achieves nothing but waste.  But if we follow the path shown by Deming, opportunities abound and improvement continues. 

1. Deming, W. Edwards. Out of the Crisis : Quality, Productivity, and Competitive Position. Cambridge, Mass.: Massachusetts Institute of Technology, Center for Advanced Engineering Study, 1986.

This is one reason that Measurement System Analysis (MSA) is included in all Six Sigma training.  Because Six Sigma is a data-driven business improvement methodology, data is used in every step of the problem-solving process, commonly following the Define-Measure-Analyze-Improve-Control (or DMAIC) framework.  The goal of MSA is to ensure that the measurement system is adequate for the intended purpose.   For example, a typical MSA evaluates the accuracy and precision of the data. 

In science and engineering, much more comprehensive and rigorous studies of a measurement system are performed for specific purposes.  For example, the US Food and Drug Administration (FDA) publishes a guidance document: Analytical Procedures and Methods Validation for Drugs and Biologics, which states

“Data must be available to establish that the analytical procedures used in testing meet proper standards of accuracy, sensitivity, specificity, and reproducibility and are suitable for their intended purpose.”

While the basic principles and methods have been available for decades, most organizations lack the expertise to apply them properly.  In spite of good intentions to improve data quality, many make the mistake of sending newly trained Six Sigma Green Belts (GB’s) or Black Belts (BB’s) to conduct MSA and fix measurement system problems.  The typical Six Sigma training material in MSA (even at the BB level) is severely insufficient if the trainees are not already proficient in science, statistical methods, and business management.  Most GB’s and BB’s are ill-prepared to address data quality issues.

Here are just a few examples of improper use of MSA associated with Six Sigma projects.

• Starting Six Sigma projects to improve operational metrics (such as cycle time and productivity) without a general assessment of the associated measurement systems.  If the business metrics are used routinely in decision making by the management, it should not be a GB’s job to question the quality of these data in their projects.  It is management’s responsibility to ensure the data are collected and analyzed properly before trying to improve any metric.
• A GB is expected to conduct an MSA on a data source before a business reason or goal is specified.  Is it the accuracy or precision that is of most concern and why? How accurate or precise do we want to be?  MSA is not a check-box exercise and consumes organization’s time and money.  The key question is “is the data or measurement system good enough for the specific purpose or question?”
• Asking a GB to conduct an MSA in the Measure phase and expecting him/her to fix any inadequacy as a part of a Six Sigma project.  In most cases, changing the measurement system is a project by itself.  It is out of scope of the Six Sigma project.  Unless the system is so poor that it invalidates the project, the GB should pass the result from the MSA to someone responsible for the system and move on with his/her project.
• A BB tries to conduct a Gage Repeatability & Reproducibility (R&R) study on production data when a full analytical method validation is required.  A typical Gage R&R only includes a few operators to study measurement variation, whereas in many processes there are far more sources of variation in the system, which requires a much more comprehensive study.  This happens when the BB lacks domain expertise and advanced training in statistical methods.

To avoid such common mistakes, organizations should consider the following simple steps.

1. Identify critical data and assign their respective owners
2. Understand how the data are used, by whom, and for what purpose
3. Decide the approach to validate the measurement systems and identify gaps
4. Develop and execute plans to improve the systems
5. Use data to drive continuous improvement, e.g. using Six Sigma projects

Data brings us opportunities.  Is your organization ready?

As a management consultant specializing in quality and continuous improvement, I see opportunities everywhere to improve customer satisfaction.   Most of them don’t require significant resources or efforts.  Despite the rhetoric about being customer-focused, most businesses fail to see and act upon such opportunities.  

Here is an example from my personal experience.

On a Sunday evening in March this year, I flew from Raleigh/Durham to Phoenix to speak at the American Society for Quality (ASQ) Lean & Six Sigma Conference.  Upon arrival at the airport, I went to the rental car center to pick up my reserved car.  Since it was my first time visiting Phoenix, it took me a while before finding the rental company in a giant parking deck. There was a single service agent, who was sitting comfortably inside a booth.  

He finished the paperwork and pointed to the direction in the garage where I could find my car.  It was a good few minutes of walking carrying my bags before I found it.  Throwing my bags in the car and sitting down, the first thing I noticed was a large horizontal crack across the windshield!  Wait, how could anyone have missed it! Pulling out my bags and walking all the way back to the service booth, I complained to the service agent.  He didn’t seem surprised or even care – he simply gave me the paperwork for another car and pointing to me where it was. 

I walked back to a sea of vehicles and found the replacement, a hatchback.  It had a large board (some interior part of the vehicle) sitting across in the back where the luggage goes.  The backseats were folded down forward.  Was the car just returned by someone and was it prepared for new customers?

I sat down and immediately smelled cigarette smoke in the car.  It was supposed to be a smoke-free vehicle.  I am highly sensitive to cigarette smell, which gives me headache. But it was late (past 10pm where I live), after 5 hours of sitting in the airplane, I hadn’t had dinner, it was a good distance to the conference hotel, and I had a presentation first thing in the morning.  I didn’t want to go back asking for another car – what was the chance that I would get a better one?  I was hoping (a big mistake!) that the smell would go away after a while.  But it didn’t, even after full blowing fans and open windows.  

Before exiting the garage, I noticed that the rental agreement listed the current mileage as 4858 whereas the odometer showed 48588.  It shouldn’t matter as the rental came with unlimited miles.  But I mentioned it to the lady at the exit gate anyway when she checked my paperwork.  She couldn’t care less.

When I returned the car a few days later, I told the agent about the smoke and the need to get the car cleaned and ready before renting to customers, he told me that the car was clean.  Really?  I didn’t hear a sorry or any apologies. 

He then gave me a wrong receipt — $20 more than what I reserved for.  It was good that I checked it.  I told him it was wrong.  He then blamed me for not returning with the tank full when in fact he looked at the gauge already, which showed full. I topped off the tank before I returned the car and had the receipt to prove it.  Luckily, I was prepared and brought a copy of my reservation showing the correct full price.   He finally produced a correct receipt.

What’s comical about this experience was that the day after returning home, I received an email message from the rental car company with “We Value Your Opinion!” in the subject line.  

I am familiar with the Net Promoter Score (NPS) and a long-time fan of Frederick Reichheld and his books The Loyalty Effect, Loyalty Rules, and The Ultimate Question.  I knew what it meant when asked “how likely are you to recommend [a business] to a friend or colleague?”  So I promptly filled out the survey.   In the survey, it asked me what they could do to make me rate them a 10.  Here was my response.

Do not worry about getting a 10 when you deserve a zero.  Do the basics to get back to 6, 7, or 8 by giving customers a clean vehicle and a correct receipt, right the first time.  Train your people to care about their jobs and do them competently.  Treat customers with respect and appreciation.  Know that the most profitable customer segments have the most options when it comes to renting a car.

Businesses need to learn the right way to understand customers and improve their satisfaction. Unfortunately, but not surprisingly, the NPS has been seriously misused and abused by many businesses, who don’t understand that it is not the score, but the quality of the products and services, that matters.  Customers don’t care about the number. 

Stop asking customers for their feedback if you cannot meet their basic, obvious requirements.

Customers give us plenty of feedback through their normal interactions with the business.  A survey may complement our knowledge about customer satisfaction but cannot replace proactive learning and improvement by every employee every day.

So if you want to improve quality and customer satisfaction, first things first — go back to the basics and get them right.