| NASJE DB | NASJE Member Materials | NASJE Publications | NASJE Curriculum Designs |
Please consider joining NASJE to unlock additional features and information - Annual membership is $150.
0.0 Judicial Education Needs Assessment and Program Evaluation (132pg/14MB)
National Association of State Judicial Educators (NASJE)
Notes:
Description:
Judicial Education Needs Assessment and Program Evaluation
PREFACE
The orientation throughout this JERITT monograph is to treat needs assessment and evaluation in an interrelated Formal In selecting from the breadth of material Available on needs assessment and evaluation, we have been guided by the desire to discuss formats which are relatively compatible for doing both; that is. we selected formats that would produce complementary needs assessment and evaluative information.
And we focus on those approaches that seem most compatible with the realities faced by continuing judicial education organizations. Among the more important of these realities is that needs assessment and program evaluation need to be undertaken in a cost conscious way, recognizing that there must be balance in how scarce resources arc allocated between actually providing a program and the assessment and evaluation of it. Therefore, this monograph is not a general survey or compendium of all alternative approaches to or features of needs assessment or evaluation as found in the literature. A bibliography at the end of the monograph provides the interested reader with sources for pursuing these topics further.
For the new judicial educator, especially one without previous experience in adult continuing education, having to confront the processes of needs assessment, program planning and program evaluation can be daunting. We believe, however, that most of the ideas presented in this monograph are relatively straight forward and should not pose inordinate difficulty. For the experienced judicial educator. we hope that this monograph provides a few new ideas. and perhaps some reminders that will serve to freshen the approach and commitment to needs assessment and program evaluation.
The approaches and example provided in this monograph arc based on the author's experiences over the last 15 years in advising and implementing needs assessment and evaluation strategies in judicial education organizations throughout the country. The results of specific associations with several State and national judicial education organizations are represented in these materials, including especially procedures and materials developed for use in Michigan and Illinois and for the Institute for Court Management of the National Center for State Couns.
Chapter 1: A Context for Needs Assessment and Evaluation
Why Do Needs Assessment and Evaluation?
Pressures To Do Neither Needs Assessment Nor Evaluation
Organization of This Monograph
Chapter 2: Continuing Judicial Education Needs Assessment
The Origins of Program Ideas
What is Educational and Training Needs Assessment?
Needs Assessment Planning Questions
Generic Issues and Options for Data Collection
Kinds of Data
Kinds of People Involved in Needs Assessment Data Collection
Using These People in Groups
Kinds of Data Collection Methods
Document Search
Observation
Focus Groups and Committees
Surveys: Questionnaires and Interviews
Sampling Issues
Connecting Needs Assessment to Program Design
Chapter 3: Evaluating Judicial Education Programs
The Contributions and limits of Evaluation
Objectivity, Subjectivity, and Mixed Evidence
Purposes for Evaluating Judicial Education Programs
Necessary Tradeoffs
Alternative Criteria for Evaluating Judicial Education Programs
The Program Monitoring Criteria
The Learning-Behavior Criteria
Recognizing Constraints Inherent in Doing Evaluation
Money and Time
Fear of Results
Goal Ambiguity and Complexity
Research Design Problems and Unavailability of Data
An Approach to Judicial Education Program Evaluation
Underlying Features of the Recommended Approach
Combining the Program-Monitoring and Behavior-Change Criteria
Questions for Planning the Evaluation
Measuring Effort and Efficiency
Participant Reactions
Learning
Job-Behavior Change
Intentions to Change Behaviors
Measuring Actual Behavioral Change
Results and a Return to Needs Assessment
Designing, Doing and Using an Evaluation
Alerting People and Getting Their Cooperation
Linking Evaluation to Needs Assessment
Chapter 1 A Context for Needs Assessment and Evaluation
The recently completed "Principles and Standards of Continuing Judicial Education" produced by the National Association of State Judicial Educators (NASJE) states that the goal of continuing judicial education is:
“to maintain and improve the professional competency of all persons performing judicial functions, thereby enhancing the performance of the judicial system as a whole.”
Enhancing professional competency and system performance through education requires both knowledge and commitment It requires knowledge of the substance and work of the judicial system; it requires an understanding of the principles of adult learning and an effective translation of those principles to continuing judicial education settings. Success also depends on whether the judicial profession is receptive to educational programming, particularly whether it will support and participate in it. Finally, success depends on having tile means to deliver continuing judicial education, and increasingly this has come to mean having an organization and a staff who will deliver the educational programming goods.
Why Do Needs Assessment and Evaluation?
Successful continuing judicial education programming links educational opportunity to the job-related needs of judges and court employees, ultimately improving their performance and that of the system. The core objective is not, therefore, education for tile sake of education, although some continuing judicial education programming may seek to develop general intellectual and interpersonal skills. But in all cases tile final objective of judicial education programming is job-relevant individual and system improvements.
When thinking about continuing education programming, the most common vision that comes to mind is a room filled with people in which instruction and learning are taking place. But ultimately the connectedness of what happens in that room to some tiling real in tile work environment depends on a set of steps which occur long before and long after anyone comes to the classroom. Connectedness depends on needs analysis and evaluation and on both of these fitting into the rest of the many steps associated with continuing education.
The multi-stepped nature of continuing judicial education and its relationship to professional competency and system performance can be portrayed in many ways; one such view is offered by Figure 1·1.
Data drives both the substance and the pedagogy of an effective continuing judicial education program. At the front end of continuing judicial education, a prospective assessment (commonly called needs assessment) identifies existing and future problems of individual and system performance and anticipates how education might help address them. Needs assessment may also help to identify opportunities for improvements through education. In short, needs assessment helps identify what should be taught to whom, when, how and why. In a world of limited resources, needs assessment also helps set educational priorities determining which problems or opportunities require attention now, and which can wait.
Without some systematic understanding of educational need, t11ere is the risk of severe mismatch between what the profession needs and what is delivered. We risk that the programming offered is at the whim of a powerful administrator or a few key individuals whose ideas are not firmly rooted in a broad-based and objective understanding of system needs. We risk programming priorities being set based on what first comes to mind, what seems easily done, or on highly personalized views of what is important (what is considered important by a highly influential judge or administrator may not be in anyone else's view, or supported by objective data).
Although prospective assessment ought to identify field-connected needs, there is no guarantee that what is delivered meets the need. Therefore, connectedness requires retrospective assessment as well. Evaluation is the retrospective step; it may be summative only, that is, merely retrospective with no intention to impact future programming. Or, evaluation may be formative, the intention being to collect information about past efforts which can be used to improve future efforts.
Figure 1-1 can too easily leave one with the mistaken impression that needs assessment and program evaluation are at opposite ends of a set of linear steps, connected to each other only through the planning and offering of programs. Yet, as conveyed in Figure 1-2, the two are really conceptually and temporally connected. The conceptual connection is that needs assessment and evaluation are essentially data collection and analysis activities, the primary purpose of each being to improve what actually happens in the educational program.
When educational programming is ongoing, evaluation provides data directly to the next round of needs assessment and program planning. Thus, the continuing judicial education process is a cyclical rather than a linear one. Information from the effective evaluation of prior programming becomes an important ingredient to assessing the need for future programming. And also, needs assessment is inherently evaluative in the sense that it measures discrepancy between the existing and the preferred level of knowledge, skill or ability required by the profession.
Finally, it is worth pointing out that needs assessments and evaluation data provide potentially objective and powerful arguments which support Ute importance and value of continuing judicial education. Especially in times of tight money, policy and decision makers increasingly look for evidence of payoff in setting allocation priorities. Verifying needs, demonstrating impact and related value following programming, and showing that there is a conscious effort to be concerned about value-formoney, establishes a level of credibility that enhances rather than retards the ability of judicial education organizations to compete for funds.
Pressures To Do Neither Needs Assessment Nor Evaluation
The line drawn between the top and bottom halves of Figure 1-2 separate what may be viewed as the more analytical from the more action oriented components of a continuing judicial education delivery system. The bottom half, because it focuses on planning concretely what is to be taught to whom and how and on actually providing the instruction, is more appealing to the professional who wants to "get on with doing it." This rush to do it rather than to waste time thinking about it is abetted in fields where the focal recipient of eliminating education is an already highly trained professional (e.g., judges and court administrators).
Occasioned by their intimacy with the actual work of tile field, there is a naturally occurring belief among many of these professionals that they already know what is needed. The logical conclusion drawn from this assumption is that you do not have to engage in complex problem assessment and educational needs assessment; rather, a few simple questions put to the "right" people will provide all that is needed. Although, as discussed in Chapter 2, there are times when this approach to needs assessment works quite well, there are many more times when it does not, and precious educational programming funds, if not wasted, are at least grossly underutilized.
There is also a natural temptation to avoid retrospective assessment of continuing judicial education programming, especially if it is we who are the subjects of the evaluation. From a purely practical point of view, once the educational program is over and people have gone home, the natural inclination for the judicial educator who must get on with the next program is not to spend more energy on a program that is already history. Also, ev01luation is a bit like the annual or biannual physical: everyone says it ought to be done, but no one likes having it done to them.
The top half of Figure 1-2 is labeled "data gathering and analysis" and does not sound very action oriented. It provides, however, the critical data and information which permit continuing educational programming to be strategic rather than merely short-run and event-specific in orientation. The top hal! also comprises that set of activities which provide the greatest chance of maximizing the degree to which programming connects to and "improves the professional competency of all persons performing judicial functions, thereby enhancing the performance of the judicial system as a whole." In other words, the top half connects continuing judicial education programming to the profession it is supposed to service in concrete ways.
Organization Of This Monograph
As Warren K. Benne points out, the worlds of the behavioral scientist and the practitioner collide over how many resources and how much time should be devoted to discovering knowledge and information. In Chapters 2 and 3 which deal, respectively, with approaches to needs assessment and evaluation, we present an "ideal" model for doing each. These ideal approaches, although offering benchmarks for maximizing our knowledge about needs and about how well we have done, are beyond Ute time, resources and technical capacities of all, or all but a very few judicial education organizations. Then, why include them at all? Besides providing a standard, educators may elect to use pieces of them as time, money and technology permit.
The central point is that with limited resources there must always be some trade-off between Ule activities encompassed in the top half of Figure 1-2 and those in the bottom half. Just as it would be silly to devote all educational resources to needs assessment and evaluation, and none to programming, it is just as silly to do the opposite. The policy question to be answered by the judicial educator is how much of the budget should be devoted to each.
What criteria exist for the judicial educator to make this policy decision? Some emanate from the requirements of the scientific method; that is, certain methods of data collection produce only certain kinds of needs assessment and evaluative information, and with varying degrees of validity and reliability. The question is, what do we want to know and with what degree of accuracy. Chapter 2 presents a number of alternative approaches to needs assessment, providing for each an overview of the kinds of information produced and their uses and validity. Chapter 3, following a similar format, focuses on alternative approaches to evaluation and the kinds of information provided.
Other criteria for determining how much and how far to carry needs assessment and evaluation are situational. These largely are concerned with the political, organizational and budgetary constraints faced by the judicial educator. For example, there may not be time for meaningful needs assessment because the state supreme court wants the program held next month; or, we do not have enough money to hold the program and engage in the needs assessment that we should; or, those in charge do not want to risk challenging the status quo which a formal evaluation certainly would; or, we have neither the funds nor the access to find out whether programming has really led to performance improvements.
These and other realities of practitioner life force trade-offs between what we ought to do as dictated by the rules of inquiry and what we can do as dictated by the constraints of organizational life. ln addition to presenting the ideal and recommended models, Chapters 2 and 3 consider also the constraints that must be faced, offering a means for educators to weigh the alternatives.
Two appendices accompany the monograph. Appendix A contains all the figures referenced in Chapters 1, 2 and 3. These figures are procedural checklists, forms and other examples for collecting needs assessment and evaluation information. Almost all of the forms will require modification to fit the particular circumstances of individual organizations and programs. However, we grouped them in a single Appendix to ease copying if that is desired.
A bibliography is presented in Appendix B dealing with educational needs assessment and educational program evaluation. These references may be consulted by the judicial educator who wishes to read more on both issues.
Authored by
JOHN K. HUDZIK
With Assistance From
NANZHENG CHENG
DENNIS W. CA TUN
MAUREEN E. CONNER
RENEE D. ROBINSON
1991 Reprint 1999
The Judicial Education Reference, Information and Technical Transfer (JERRITT) Project
Appendix A: Figures and Forms
Figure 1-1 Steps in the Continuing Judicial Education Process
Figure 1-2 The Continuing Education Cycle
Figure 2-1 A Model of the Principal Components and Steps for an Educational Needs Assessment
Figure 2-2 Example of an Open-Ended Call for Educational Programming Needs
Figure 2-3 Example of an Open-Ended Opinion Survey Focusing on Future Conditions Having Educational Implications
Figure 2-4 Planning Worksheet for Identifying Prospective Members of a Needs Assessment Task Force
Figure 2-5 Sample Format for Guiding Experts in Conducting Literature Searches for Relevant Documents
Figure 2-6 General Rules and Procedures for the Non-Systematic Recounting of Behavioral Observations During Needs Assessments
Figure 2-7 Nominal Group Technique
Figure 2-8 Delphi Technique
Figure 2-9 Example: Opinion Survey Measuring Education Need with a Modified Delhi Approach
Figure 2-10 Linking the Management of Problems and Goals
Figure 2-11 Thinking Problems Through to Educational Programming Solutions
Figure 2-12 Examples of Scales Used in Closed-Ended Opinion Surveying Judicial Education Needs Assessment and Program Evaluation
Figure 2-13 Examples of Collecting Job Task Information By Survey and Relating it to Educational Needs Assessment
Figure 2-14 Example: Using A Sell-Assessment Survey to Measure Discrepancies Between Required and Needed Task Proficiencies
Figure 2-15 Example: Survey Measuring Task Importance with a Modified Delphi Approach
Figure 3-1 Summary Program Cost Data
Figure 3-2 Sampling Participant Reactions to Programs
Figure 3-2a Program in Behavioral Strategies for Managing Difficult Situations
Figure 3-2b Program Evaluation
Figure 3-2c Conference Evaluation
Figure 3-2d Judicial Writing Seminar Program Evaluation
Figure 3-2e Program Evaluation
Figure 3-3 Learning Assessment (Examples of Questions)
Figure 3-4a Action Planning Guide (Behavioral Intentions)
Figure 3-4b Action Planning -Behavioral Intentions
Figure 3-5 Action Planning Worksheet
Figure 3-6a Assessing Job-Behavior Changes
Figure 3-6b Three Month Follow-Up Assessment
Appendix B:
A Bibliography for Further Reading