Data Integrity: The Foundation of Effective Performance Benchmarking

February 21, 2018

Performance benchmarking is essential to sustained operational improvement in asset-intensive industries such as oil & gas, refining, and chemicals.

For nearly 40 years, Solomon has been helping clients in these industries understand competitive position and gaps, as well as the detailed factors that drive them. Our unique method of benchmarking is called Comparative Performance Analysis™ (CPA®). While we have pioneered many innovative metrics and insightful comparisons to maximize business impact for our clients, it is our enduring commitment to input data integrity that forms the foundation of effective performance benchmarking. Solomon invests significant resources toward ensuring the integrity of participant-submitted data—before that data is applied to the benchmarking process.

What Types of Data do Participants Submit?

Solomon collects a wide range of performance data from its participants as part of its regular benchmarking studies. Participants are asked to submit data for measurements encompassing capacity, raw material and product mix, conversion performance, energy consumption, maintenance expenses, reliability losses, capital expenditures, staffing, and other drivers of cost structure and margins. This data is then leveraged to develop relevant performance comparisons across a variety of indicators and metrics. For these performance comparisons to be valid and meaningful, it is essential that the underlying participant-submitted data be both consistently developed and accurately represented.

The Goal: Zero Errors in Participant-Submitted Data

Solomon’s goal with respect to participant-submitted data is simple: zero errors. Achieving this goal is important not just to each individual participant but to all of the participants in a benchmarking study. That being said, errors can happen during the participant data collection and submission process. Sometimes these errors are as simple as an improper unit of measurement or a misplaced decimal point. At other times they can arise from misinterpretation of requested data or inconsistencies between a participant’s various data sources and/or functions. Regardless of the cause, it is imperative that these errors be both detected and communicated to participants as early as possible so that they can make appropriate corrections, preferably before data is even submitted to Solomon.

How Does Solomon Ensure Integrity of Participant-Submitted Data?

Solomon deploys multiple overlapping layers of protection to ensure the integrity of participant-submitted data to the benchmarking process. The first layer is education on the data submission process itself. To that end, Solomon regularly performs both public and private seminars to train participants on data collection and submission. It also offers online help modules on specific topics that can be referenced during the data collection process.

The next layer of protection is comprised of the “smart” input forms in which data is actually entered. These “smart forms” perform a myriad of data quality and consistency analyses in real time, providing instantaneous feedback to the participant when potential errors are detected during the data entry process. For example, these forms will alert the participant if data is outside of the expected range for a given parameter or type of asset. The forms will also provide warnings if current data is materially inconsistent with prior data submissions, prompting participants to either verify current data or explain legitimate differences. Moreover, these “smart forms” perform multiple cross-category analyses to identify potential inconsistencies between related performance categories, such as between maintenance labor expenses and reported work hours. Most importantly, these “smart forms” provide feedback to participants on potential inconsistencies in the data prior to submitting data to Solomon, enabling early review and correction.

The third layer of protection is applied after Solomon receives participant data submissions. This layer consists of a series of rigorous review procedures performed in-house on each participant’s input data by highly experienced Solomon consultants. These Solomon consultants examine in detail both the raw data and the resulting calculations. If suspect or inconsistent data is detected via these in-house analyses, the participant is contacted to verify the data, provide further explanation, or submit corrected data.

The final layer of protection involves outlier and anomaly analysis of the entire study participant pool and relevant participant peer groups. If suspect or irregular data is detected for a particular participant during this outlier analysis, that participant will be contacted to verify the data, provide further explanation, or submit corrected data.

The Bottom Line

Benchmarking studies rely on participant-submitted data that is both consistently developed and accurately represented. Ensuring the integrity of that input data is therefore of paramount importance to effective performance benchmarking. Solomon remains committed to input data integrity via its systems, practices, and protective measures, both now and in the future, to ensure that our clients are able to trust the study results and use them confidently to manage their performance improvement efforts.