December 2010 // Volume 48 // Number 6 // Tools of the Trade // 6TOT1
Viewing Bennett's Hierarchy from a Different Lens: Implications for Extension Program Evaluation
This article describes the use of Bennett's hierarchy in Extension program evaluation over the last 35 years. The authors discuss how the hierarchy can be linked to assess program outcomes and related costs for each level of the hierarchy and conclude that as one moves up the hierarchy, the evidence of program impact is stronger. The authors suggest a four-step framework for identifying costs for each type of outcomes—short, intermediate, and long term.
Bennett's hierarchy has been used for almost 35 years in Cooperative Extension. Educators continue to relate well to this hierarchy in evaluating their Extension programs. Bennett's hierarchy contains seven sequential steps (input, activities, participation, reaction, knowledge, skills, opinions, aspirations-KASA, practice change, and end results/social, economic, environmental conditions-SEEC) (Figure 1). The first four steps focus around process evaluation, while the last three steps focus on outcome/impact evaluation. Modifications were made to the hierarchy by Bennett and Rockwell in1995 and in 2000 by adding a continuum linking program evaluation and program development (Figure 2). This revision helped educators understand that evaluation should be considered upfront in the design or planning phase of a program, not as an after-program activity.
Bennett and Rockwell's TOPs Model
Today, in a time of continued reduction in government funding, Extension professionals are challenged more than ever before to document outcomes of programs and address stakeholder demands for accountability. This article provides a framework for linking Bennett's hierarchy to program outcomes and costs. Extension professionals could use this framework to link program outcomes and costs associated with such outcomes.
Examining Bennett's hierarchy from a different lens provides some insights to link the hierarchy with outcomes and costs associated with evaluating a program. A positive association between the seven steps of hierarchy and outcome type (short, intermediate, and long term) is proposed (Figure 3). As one moves up the hierarchy, the evidence of program impact gets stronger. Collecting evidence to assess impact of programs at the higher levels of the hierarchy becomes costly and time consuming and requires more skill (Figure 3). For example, one may use simple pre-post measures to assess short-term outcomes. On the other hand, to assess behavior/practice change, follow-up of participants is required, which adds to the cost of evaluating a program. This needs to be communicated to field-based educators so that they can plan early on what is needed in terms of costs, time, skills (data collection, analysis, and interpretation), and resources needed to evaluate an Extension program (Table 1).
Linking Bennett's Hierarchy to Program Outcomes and Costs
Based on the information presented in Figure 3, a framework for linking Bennett's hierarchy to program outcomes and costs was developed (Table 1). As shown in Table 1, evaluating programs at the lower levels (input, participation, activities, and reactions) may require little effort and are less expensive. However, evaluating at the lower levels will help program staff to assess ongoing program activities to make adjustments as the program progresses and to see whether or not the program is being implemented as planned. On the other hand, documenting and/or collecting evidence at higher levels of the hierarchy (KASA change, behavior change, and SEEC) requires skills relative to questionnaire development, data collection and analysis, interpretation, and reporting. In addition, it may also require understanding of evaluation designs, data collection at multiple points, sophisticated statistical analyses such as analysis of covariance, general equation modeling, use of covariates, etc. If these are planned in advance and properly done, the potential for program impact is stronger.
|Process Evaluation||Outcome Evaluation|
|X = Low cost, effort, and evidence; XX—requires questionnaire development, data collection and analysis skills; XXX—requires understanding of evaluation designs, multiple data collection, additional analysis, skills, interpretation; XXXX—all of the above, time, increased costs, potentially resulting in stronger evidences of program impact.|
Steps for Using the Framework
First, decide on the level of evaluation you want to conduct, that is, process evaluation (lower levels of Bennett's) or outcome evaluation (higher levels of Bennett's) or both. In this step, develop a concept of linking program objectives to evaluation questions to outcomes (Radhakrishna & Relado, 2009).
Second, identify key indicators for the evaluation. Make sure that the indicators you identified are measureable and relevant to the program.
Third, once you decide on the type of evaluation and key indicators, consider all the costs that you might incur to conduct the evaluation (Figure 3 and Table 1). For example, one of your objectives is to assess practice/behavior change (higher levels of Bennett's) as a result of participation in a financial management program. This implies that you will be collecting data at least three times (pre, post, and delayed) to determine practice/behavior change. Perhaps you may also want to add a comparison group (non program participants). Including a comparison group will help make meaningful comparisons of a particular Extension program's strengths and weaknesses. In addition, you need to be cognizant of the time, labor, and other resources you need to have to carry out the evaluation at higher levels of the hierarchy.
Fourth, design a matrix to document all the costs you incur to evaluate the program. Use of spreadsheets (Microsoft Excel) or other technology to document the costs for each level would be a good start.
Following these steps will help you to estimate the costs involved in evaluating an Extension program. In addition, these steps will also help guide the development of a budget for evaluation, especially if you are writing a grant for funding from a private or a public agency.
Documenting program/project outcomes will continue to challenge program managers and educators, especially in the accountability era. Early identification of costs associated with documenting short, intermediate, and long-term outcomes will go a long way in assessing the costs and benefits associated with evaluation of an Extension program. Extension specialists and program managers should use these steps when they conduct training/workshops relative to costs and benefits associated with evaluating an Extension program. In addition, the process of linking costs to program outcomes should be communicated to all educators who evaluate Extension programs. Such communication will help link evaluation questions to outcomes and costs, and ultimately justify the value of Extension programs to public good.
Bennett, C. (1975). Up the hierarchy. Journal of Extension [On-line], 13(2). Available at: http://www.joe.org/joe/1975march/75-2-a1.pdf
Bennett, C., & Rockwell, K. (1995, December). Targeting outcomes of programs (TOP): An integrated approach to planning and evaluation. Unpublished manuscript. Lincoln, NE: University of Nebraska.
Radhakrishna, R. B., & Relado, R. Z. (2009). A framework to link evaluation questions to program outcomes. Journal of Extension [On-line], 47(3) Article 3TOT2. Available at: http://www.joe.org/joe/2009june/tt2.php