Manager's Briefcase
Building and Implementing an Effective Court Performance Measurement System
by Ingo Keilitz

Editor’s Note: This article is an adaptation of several postings in Made2Measure, a blog in which the author explores emerging issues related to performance measurement in courts and justice systems in the US and other countries.

Performance Measures Drive Success
The ability to measure performance is a critical enabler for getting results and achieving goals. Knowing what and how to measure makes a complicated world less so. Because they are unambiguous and actionable, performance measures drive success. Effectively used, they serve both as incentives and as practical tools for justice system improvement. An effective court performance measurement system (CPMS) enables court leaders and managers to:

• Translate vision, mission and broad goals into clear performance targets
• Communicate progress and success succinctly in the language of performance measures and indicators
• Respond to legislative and executive branch representatives’ and the public’s demand for accountability
• Formulate and justify budget requests
• Respond quickly to performance downturns (corrections) and upturns (celebrations) in performance
• Provide incentives and motivate court staff to make improvements in programs and services
• Make resource allocation decisions
• Set future performance expectations based on past and current performance levels
• Insulate the court from inappropriate performance audits and appraisals imposed by external agencies or groups

Around the globe, the use performance measurement has spread dramatically in recent years at all levels of government, as well as in private and nonprofit organizations. [In the courts community, performance measurement increasingly is seen not only as the best way to improve the quality of programs and services but also to drive major policy reform and organizational transformation.

A Court Performance Measurement System (CPMS)
The key to the successful use of performance measurement is to identify and to develop the right measures -- measures that are aligned with vision, goals and success factors -- and then making sure that the measures get into the hands of those who can make good use of them at the right time. This requires a Court Performance Measurement System (CPMS). A CPMS is the routine collection, analysis, synthesis, delivery and display of performance measures and other information about the work and accomplishment of a court as an organization. It is an ordered and comprehensive assembly of parts -- interrelated data, principles, standards, methods, processes and procedures -- forming a unitary whole. All the elements of a CPMS -- the performance measures, data collection methods, analysis and interpretation, and information distribution and display -- operate toward the common purpose of running a court effectively and efficiently.

A Six-Step Design Process
Over the last five years, by benchmarking the design processes used successfully in private sector and public organizations, and by extensive field study of courts throughout the world, CourtMetrics and colleagues have developed a best practice design process of six overlapping steps and sub-steps (tasks). This design process introduces discipline, conceptual clarity and method to the myriad conceptual, methodological, analytic issues associated with the building a CPMS.

• Step 1. Assessing the performance measures currently used.
• Step 2. Identifying and defining the performance measures needed to help achieve goals.
• Step 3. Developing hierarchies or families of measures.
• Step 4. Testing the measures.
• Step 5. Creating data collection and distribution methods that ensure timeliness and utility.
• Step 6. Building useful performance measurement displays.

Although the steps are laid out in a linear sequence, taking these steps is iterative in practice. The design team may rethink the results of steps several times before reaching a consensus and final decision.

Each of the essential building steps addresses critical questions: How does the court currently measure its performance? How are the current measures distributed across inputs, outputs, outcomes, key success areas, perspectives, and core performance areas? What required or desired performance information currently is not available to the court? What specific performance measures would provide that information? How does the court select a vital few, instead of a trivial many, performance measures? How should the court go about developing the desired performance measures? To what degree should the selected measures be tried and demonstrated before implementation? When and how should the performance data be collected? To whom and how often should it be distributed? In what format and on what schedule should the performance information be conveyed and displayed?

Step 1. Assessing Current Performance Measures
Purpose: What performance measures are currently used in your court and by your court’s justice system partners? Are they sufficient in terms of number, type, and balance of perspectives? Which of the measures seem more important than others? By what methods are the measures taken and by whom? Taking a critical look at the performance measures your court and justice system partners are using today is a practical starting point for getting to the fundamental question of what are the right measures for your court. Rather than trying to tackle this daunting question with a blank slate, this first step draws on knowledge already available to the court. It produces results in a relatively short time. Even if a court postpones or abandons the building of a CPMS, both the process and the results of this first step can serve as valuable references for planning and development of various court improvement projects.

Tasks: The products of this first step are an inventory of input, output and outcome measures currently used by the court and its divisions, departments and units, and an evaluation of the adequacy of the measures as a basis for building a CPMS. Three tasks are required to complete this step: (1) cataloguing the court’s performance measures and indicators, as well those of the court’s justice system partners; (2) categorizing the measures by whether they are input (resources), output (activities) or outcome (impact) measures; and (3) assessing their completeness and balance.

Step 2 – Developing Desired Performance Measures
Purpose: The second step places a premium on conceptual clarity and the operational definition of values, principles, and fundamental success factors that are at the heart of a court’s purpose –access and fairness, efficiency and effectiveness, public trust and confidence. It requires the court to identify its key performance areas, determine the types of performance measures to be included, select a limited number core measures, and define the selected measures operationally. People must sign on to the purpose of a performance measure, the key outcome it indicates, not just the metric. Although there is no ideal number of measures that should be identified, it is better to have a few meaningful performance measures than many poor ones. It is also preferable to select measures that indicate the desired outcomes of the court's programs and services rather than measures of the resources (input) or completed activities (outputs) used to produce those outcomes.

Tasks: The four tasks of this second step proceed from the general to the specific, from broad conceptualization of performance areas to the operational definition of desired measures. The step produces a synopsis of seven to twelve core performance measures, operationally defined and linked to key court performance areas.

Step 3 – Creating Measurement Hierarchies
Purpose: Using the results of the previous two steps –an inventory all the performance measures currently used at all levels of the court a set of high-level core performance measures – the third step of building a CPMS requires the creation of hierarchies (families) of related performance measures. The value of measurement hierarchies is that they define the connection between high-level strategic goals and performance measures with lower-level departmental or divisional objectives and measures. Measurement hierarchies in which lower-level subordinate measures cascade down from core measures help align the overall goals of the court with the goals and objectives of its divisions, units, and programs. They help make it clear to all court employees precisely how their actions help fulfill the court’s mission and strategic goals.

Tasks: Step 3 requires three overlapping tasks: (1) breaking out (disaggregating) core measures; (2) identifying and defining other subordinate measures aligned with core measures that are not mathematical breakouts of core measures; and (3) constructing a hierarchy of performance measures for each of the core measure.

Step 4 – Testing, Demonstrating and Developing Measures
Purpose: Once steps have been taken to identify and define the desired performance measures, and to construct hierarchical relationships among them, simply issuing an edict to "go forth and measure" is likely to invite failure. It is one thing to identify and define performance measures, and quite another to demonstrate that a court can actually take the measures given its current operating structures, systems and procedures. Court staff should be provided detailed directions, as well as encouragement, to take the prescribed measures. Procedures for planning and preparations for the measures, data sources, data collection methods, analysis, and distribution and use of the measures should be carefully prescribed. The measures should then be field tested, demonstrated, and modified as necessary.

Tasks: This step requires three tasks. The first task is to describe the measures in sufficient detail to allow the testing and demonstration of the measures, which is the second task. The third task is to develop and refine the measures based on their test and demonstration.

Step 5 – Developing Data Collection and Reporting Timeframes
Purpose: Once you have taken the first four steps of building a CPMS -- identified the desired core performance measures based on assessment of your courts current performance measures, created measurement hierarchies of related measures, tested and demonstrated the measures -- it is time to make sure that the right performance data gets into the hands of the right people at the right time. How frequently should the data for each measure be collected and displayed? Who needs the performance data provided by the core measures and subordinate measures in the allied hierarchies? How will they use that data? When do they need it and how often will they use it?

Tasks: This step requires two discrete tasks. The first is to determine the ideal timeframe for availability (e.g., data may already available in real-time in automated databases) and use of the performance data for each core performance measure. The second task is to consider feasibility and costs of data collection and display and to adjust the ideal timeframes accordingly. The result of the step is a timeframe for collection and distribution of each core and subordinate measure in all of the measurement hierarchies.

Step 6 – Building Performance Measurement Displays
Purpose: Courts today are drowning in data and will succeed to the extent that they are able to harness performance information to make better, more informed and quicker decisions. Collecting and assembling critical performance data are useless if the data are not delivered to the right people at the right time. Although standardized or special reports are useful for many users, they can not meet the needs of users who need more rapid and flexible data access. An effective display of a CPMS is one that users can readily access, that is easily read and understood, that is organized for easy navigation among core and subordinate measures, and one that provides a “line of sight” that conveys to everyone what the drivers of success are and provides them with the concrete knowledge of how they contribute to that success.

Tasks: The sixth and final step of building a CPMS is the design, development, and implementation of performance measurement displays – presentations that allow end users to access, to view and to use the performance data. The first task is to decide how the display should look and function. A good place to start the design of a display system is with a study of the functionality of computer software products collectively referred to as performance management solutions or “business intelligence” (BI) offered by an increasing number of companies. The task is complete with a decision to buy or to build a computer-based performance display system. The second task of this step includes not only the actual writing of computer code but also the preparation of requirements and objectives, the design of what is to be coded, as well as testing and confirmation that what is developed has met the objectives. It should proceed through successive phases – from analysis of software requirements through system integration and testing --that are familiar to computer software engineers.

Implementing Performance Measurement
In the 1989 film Field of Dreams, an Iowa corn farmer (Kevin Costner) hears voices that tell him, “If you build it, they will come.” He interprets this message as a command to build a baseball field on his farm. He does and they -- Shoeless Joe Jackson and the other seven Chicago White Sox players banned from the game for throwing the 1919 World Series – come. This works in the movies but it does not work for court performance measurement. It is one thing to build a court performance measurement system (CPMS) and quite another to get the CPMS to be used effectively.

A Pile of Stethoscopes
Kevin Baum, a performance management consultant who works in government outside of the courts, warns us in a recent edition of Perform (Special Edition, Government, no date) that we make a fatal mistake when we declare victory too soon, that is, immediately after we have built a CPMS (see the October 15, 2005, Posting, “Six-Step Process for Building an Effective Court Performance Measurement System”). Baum makes his point with this anecdote:

You’re in a board room waiting for the executive staff meeting to begin when in walks the Director. Under his arm you see 20 stethoscopes, and you think, “What’s up with that?” The stethoscopes aren’t all the same though – they are different sizes, shapes, colors and brands – but yes they are still stethoscopes. The Director, with a broad grin and a tad of flair, tosses all the scopes on the boardroom table and declares in a proud and booming voice, “Look how well we are managed. We are truly a performance-informed organization and I’d like to thank all of you for your efforts.

Baum’s point is that a CPMS, like a stethoscope, is only a tool. Nothing more, nothing less. Courts should not declare victory once the CPMS has been built. Like a pile of stethoscopes, a CPMS is essentially meaningless until we get it into the hands of people who can put it to use, understand what it is telling us, and apply what we are learning.

Performance measurement can fundamentally change the way courts do business. For this to happen, however, a CPMS has to be integrated with a court’s key business processes and day-to-day management. Until the measures are actually used, they will never begin to work for us as tools to improve performance. Worse, warns Kevin Baum, an idle CPMS will alienate the court’s workforce by burdening it with yet another management initiative that suffers from no apparent follow-through.

Using a CPMS
Even before a CPMS is fully built and developed, courts should begin to consider the following two general strategies: Train court managers and staff on the use of the performance measures of the CPMS. The value of a performance measure lies not in the measure itself but rather in the questions it forces us to ask and how we learn and grow as a result. What is the current or initial performance level? What are the changes over time? What are the acceptable upper and lower boundaries of the particular measure? What are the problems identified by the measure? Given what we know about the measurement, what performance expectations should we have in the future? Managers and court staff need to be thoroughly familiar with the functions of the performance measures that these questions highlight – establishing baselines and benchmarking, control, trend spotting, problem diagnosis, and operational and strategic planning – for the court as a whole and, importantly, for their area of responsibility.

Integrate performance measurement with the court’s key operations and management processes. Yes, performance measurement can fundamentally change the way a court does business, but it will not happen by itself. They will not come into the field of dreams simply because the CPMS is rolled out. If a court’s leadership and management are to become truly performance-based, performance measurement has to become hard-wired into the very DNA of the court’s organizational culture.

Some of this is simple and straightforward. For example, automated performance measurement displays can identify specific court staff as “owners” of the performance measures who can be queried (“Why was there a downturn in trial certainty this month even though we tightened our continuation policies?”) with an email function linked to the measure on the display. Court leaders can decide to make the results of core performance measures a standing item on executive meeting agendas. Only those measures that require action – i.e., those that fall outside of the lower and upper control boundaries -- are discussed. Falling below the lower controls will stimulate improvement actions and exceeding upper controls (goals) will be cause for recognition and, perhaps, celebration.

Aligning performance measurement with other key management processes like strategic planning, budgeting, quality improvement, and human resource management may be more demanding but not necessarily difficult or complex. For example, a court’s ability to develop measurable performance objectives is critical to success of its strategic planning process. A strategic goal like maintain a high-performance workplace is made useful only if it is translated into a measurable objective such as workforce strength, commitment and engagement exceeds 80% as measured by a quarterly survey of court employees. A CPMS that includes a measure of workforce strength not only facilitates development of a strategic plan by giving definition to goals and objectives but also establishes the mechanism by which the strategic plan is put into action.

Ingo Keilitz, a long-time member of NASJE, is a frequent speaker, writer, and consultant to public and nonprofit organizations. He heads Sherwood Consulting/CourtMetrics, a management consulting firm in Williamsburg, Virginia, specializing in performance measurement and management, organizational development, and strategic planning. He is Of Counsel Consultant in performance measurement at the National Center for State Courts (NCSC). Ingo has helped shape the landscape of court administration as a major contributor to the development of court performance standards and measures, including the Trial Court Performance Standards (1995), the Family Court Performance Standards and Measures (1999), Quality Counts: A Manual of Family Court Performance Measurement (2001), the National Probate Court Standards (1993), the Guidelines for State Court Decision Making in Life-Sustaining Medical Treatment Cases (1992), and the Guidelines for Involuntary Civil Commitment (1986). He has helped courts in the United States and other countries build court performance measurement systems (CPMS); he is the chief architect of CourtMetrix, an automated CPMS produced by ACS, and a major contributor to CourTools, a set of ten performance measures developed by the NCSC. He is the author of over 120 articles, monographs, book chapters and books on planning, leadership, performance standards and measures, justice administration, mental health and the law, and education. He received the NCSC’s Distinguished Service Award in 1989 and was inducted into the Warren E. Burger Society in 2002. Ingo can be reached through his blog Made2Measure or by email at ikeilitz@cox.net.

Return to Top

Copyright 1999-2010, National Association of State Judicial Educators
Contact Web Developer

This Website is updated quarterly by NASJE.
The opinions expressed herein do not necessarily reflect the views of the National Association of State Judicial Educators.