There is hardly a day gone by without seeing an article about a business or organization using data analytics, machine learning, or artificial intelligence to solve tough problems or even disrupt the industry. With wide availability of computers and other digital devices, capturing and storing data becomes easier. This represents unprecedented opportunities to gain knowledge and insight from data.
However, turning the opportunities into fruitful results can be a bumpy journey. I expect organizations to encounter greater difficulty than implementing Six Sigma, a data-driven business improvement methodology.
Since the 1980’s, many organizations have implemented Six Sigma (often along with other methodologies) to improve their performance. Some were able to transform the entire organization’s culture and capabilities to achieve sustained improvement, while many were only able to achieve isolated and/or temporary gains. There is no question that change leadership and organizational change management capability played a critical role. Implementing data analytics is no exception.
In addition, Six Sigma and Big Data analytics share some unique challenges, one of which is the requirement for data and the expertise in extracting insight from the data. I have seen countless Six Sigma projects fail to deliver the promise because of poor data availability or quality and/or lack of skilled resources. Unable to achieve quick and significant improvement, some organizations have given up on Six Sigma and shifted more effort to Lean or Agile. But the underlying causes of deficiencies in data and analytics capabilities are not addressed and will inevitably impede implementation of data analytics initiatives.
Therefore, organizations considering investing in data analytics should seriously assess these two risk areas.
Poor data quality
I use “quality” here loosely to mean two things, usefulness and absence of defects.
Not all data are equally useful and can help us develop insight or solve problems. What data should be captured, stored, and processed? Data that is readily available may not be useful to the problem we try to solve, whereas potentially useful data can be costly to collect. Who can help decide and prioritize what data to collect?
It is a known fact but may be surprising to some people that data scientists spend more time cleaning up data than analyzing it. Useful data rarely come in a complete, accurate, and consistent format. A Forbes article reports that data scientists spend about 80% of their time on collecting, cleaning, and organizing data. I concur that data cleaning is the most time-consuming and least enjoyable task. No business wants their scarce and highly paid resources to spend the majority of the time on non-value added activities. What can they do about it?
Lack of resources with analytics and subject matter expertise
To solve business problems, analytics experts need computer science and statistical skills but also general operations and business knowledge. Ideally, they also have subject matter expertise. But such talent is the exception rather than the norm. The iterative process of collecting, cleaning, modeling, and interpreting data requires close collaboration among analytics, subject matter experts, and management. My observation has been that most subject matter experts are not familiar with even the basic concepts of computer science and statistics, the backbone of analytics. Simply hiring a few Black Belts never worked for Six Sigma; acquiring data scientists is not enough if the rest of the organization is ill prepared. A Center of Excellence model tends to centralize the analytics expertise and delay broad engagement and ownership across the organization.
These areas are but two important considerations as leaders develop a comprehensive approach to mitigate risks in analytics programs. Leaders should follow a strategy development process and resist one-off efforts, such as technology installation or talent acquisition. By evaluating all aspects of the current operating model with respect to their vision of digital capabilities for the future organization, they can develop a cohesive plan for a smoother ride to the destination.