Using a Reliability Scorecard to Measure if Your Project is Delivering Results

By John Cray, Life Cycle Engineering

Let’s assume you have the necessary sponsorship and formal approvals to invest in improving reliability. Remembering Stephen Covey’s second habit (begin with the end in mind), how can you measure if your reliability improvement project is delivering results that align with business objectives?

Begin by Evaluating Your Current State

It’s likely that some aspect of your operation, maintenance, or equipment is not meeting expectations. You’ll want to establish a current-state baseline so that you can track how reliability improves and the business benefits. Begin your assessment by collecting data to assemble the hard facts. Here are good areas to measure: 

• Overall equipment effectiveness (OEE) 
• Production volume performance (use a statistical process control chart to highlight variability)
• Historical maintenance spend trend
• Level of reactive maintenance (labor hours spent on emergent and urgent work vs. total labor hours)
• Overtime for production and maintenance
• Top 5 – high-repeat failure equipment, production process issues, cost areas
• Maintenance backlog
• Schedule compliance 

Next, interview a small cross-section of employees to collect the ‘soft’ facts. Make sure to interview people closest to the work, namely operators, tradespeople, and storeroom staff. Ask open-ended questions to gain their full perspective on reliability. I always conclude 45-60 minute interviews with two questions:

• If you were plant manager for a day and had magical powers, what one aspect of the operation would you improve?
• Do you have any questions for me?

Asking people their opinion shows appreciation and generates buy-in. And, of course, remember to thank them for their time invested.

Define the Project Scope and Plan, and Create the Reliability Scorecard

With a good baseline of both ‘hard’ and ‘soft’ facts, including a diagnosis of which areas need the most improvement, define your problem statement and the project scope. It is important to have a well-defined scope, timeline, and realistic business objective(s) so that leaders can manage towards a plan and schedule. Leaders must understand the gaps and the business benefit of closing them; this will allow them to answer questions from their organization.

The scope of the reliability improvement and associated benefits allows the project leader to create a reliability scorecard. The scorecard should change over time as the organization transitions, completes the project, and matures to sustaining new levels of performance.

The reliability scorecard is critical to capturing project benefits and should include the following types of measures:

Transition – These are temporary project measures to indicate if reliability changes are taking place. When these measures reach their objectives and level off, you can remove them from the scorecard and add new transition measures. Examples include:

• Percent of labor hours recorded on a work order
• Number of assets with a criticality rank
• Number of employees trained on a given topic

Leading – These are measures of a process, also called dashboard measures, because they are forward looking, like driving an automobile. Process measures indicate how well an organization is executing a particular process. Trends indicate where you are heading. Examples include:

• Backlog in crew-weeks
• Weekly schedule compliance
• Number of critical assets with a maintenance strategy

Lagging – Also known as rear-view-mirror measures, lagging measures typically measure completed actions, such as volume produced. These measures are like looking in a car’s rear-view mirror and indicate past performance. Examples include:

• Number or weekly ‘widgets’ produced
• Maintenance monthly spending
• Finished goods inventory

The scope and business objectives of your improvement project are the basis for which measures you select. The point to remember when developing a scorecard is to keep a balance of all three metric types during the project. At a minimum, include the business result measure(s). For example, if the objective is more 'pounds' of production, measure in pounds, then translate into agreed-upon financial terms – revenue, conversion cost, or profit.

Report Progress and Results, Not Just Activities

It is easy for any improvement project to fall into a trap of measuring and reporting progress based on measuring activities. An activity can be anything from the number of new assets loaded in a CMMS or the number of employees trained in root cause analysis, to the number of work orders with a job plan and an estimate, and countless other important, but transitional, project activity metrics. These are important steps towards improving reliability. 

There is an excellent article in Harvard Business Review, Jan / Feb 1992, titled “Successful Change Programs Begin with Results” by Robert H. Schaffer and Harvey A. Thomson. Their research in the early days of quality improvement programs indicated that two thirds of organizations focused on measuring and reporting activities, not results! Throughout my career, I continually reference this article. I find it as timely today as it was 25 years ago. 

Establishing the right effective measures for your reliability scorecard at the beginning of a project, will first establish the use of ‘leading’ and ‘lagging’ measures which can be new for the organization. Second, by including the business objective measure(s) in the scorecard, you are creating visibility and reinforcing to leadership and the organization as a whole, the importance of the project to the business. Lastly, when updating the scorecard, reporting to the organization, and to senior leadership, you are providing a balance of project execution, process performance, and most important, reinforcing the business objective.

John Cray, CMRP, is a Principal Consultant with Life Cycle Engineering. John has more than 35 years of experience with demonstrated results delivered in maintenance management, reliability engineering, small capital projects, manufacturing leadership, corporate reliability leadership, and consulting roles. He also has extensive technical, safety, CMMS (MAXIMO, SAP) and organizational change management knowledge. You can contact him at jcray@LCE.com.