Volume 35 Number 1
|
February 1997 Volume 35 Number 1 |
|
|
Editor's PageDear Reader, Evaluation, accountability, needs assessment are all current buzz words in the Extension System and this issue of the Journal of Extension is no exception. You'll find a lot of information about evaluation, accountability, and program planning using a needs assessment approach. At the same time, there's a reminder that basic Extension education approaches still apply to finding and working with new audiences. A this time, we want to welcome four new members to the editorial committee. Committee members are the folks who review articles for publication in the Journal. New members are Fred Herndon, Virginia; Mary Ann Sward, Oregon; Judy Winn, Texas; and Sue Buck, Oklahoma. We look forward to working with them. It's also time to salute the retiring members of the committee, most of whom went through the transition from a print publication to an on-line operations. We will miss working with Greg Hutchins, Wisconsin; Barbara Hunter, New Hampshire; Kathleen Parrott, Virginia; Kathy Prochaska-Cue, Nebraska; Bill Umscheid, Kentucky; and Barbara White, USDA. Their efforts are much appreciated. The Journal is also interested in people who would like to serve on the review committee. It would be desirable to have a bank of candidates for the committee from which to select when vacancies occur. In the meantime, keep those manuscripts coming. Len Calvert, editor
Extension Journal, Inc.Extension Journal, Inc. is a quasi-official body of the National Association of State Universities and Land-Grant Colleges and the Extension Committee on Organization and Policy (ECOP). It is a nonprofit corporation organized for the purpose of publishing a journal for professional Extension staff, adult educators, and community developers. Board of Directors:
Judith Jones, President, Virginia, Southern Directors Ex-officio:
Leonard Calvert, Oregon, Editor Editorial Committee:
Joyce Alves, Arizona
Competencies: A New Language for our WorkBarbara Boltes StoneAssociate Professor and Extension Specialist for Planning and Performance Texas Agricultural Extension Service Texas A & M University College Station, Texas Internet address: bv-stone@tamu.edu Sally Bieber 4-H Home Economist Louisiana State University Agricultural Center, St. Landry Parish Opelousas, Louisiana Facing an environment of increased accountability, the Cooperative Extension System (CES) is sharpening its focus on how to succeed in today's changing environment and how we can better communicate those successes to the public. Bob Robinson, Administrator, Cooperative State Research and Extension Service, has suggested that the terms relevance, usefulness and quality are essential when preparing for and communicating our successes. Another term that should be added--competency modeling--signifies a new language about work and has the potential to redefine the Cooperative Extension System so that the educational work we do can continue. Competencies are the application of knowledge, technical skills and personal characteristics leading to outstanding performance. Competency models are designed around the skills individuals and groups need to be effective in the future and are used for making human resources decisions. To be truly effective, competency models must have strong and irrevocable ties to the strategic issues of the Extension organization. One of the most critical strategic issues facing the CES may be how to create an infrastructure that promotes innovation and continuous learning. Identifying the competencies that will help us anticipate new ways of perceiving and thinking about complex problems should be our foundation as we strive for relevance, usefulness and quality in our educational programs. Competencies is not necessarily a new term in Extension. Some states have identified core competencies for particular jobs. In 1993, the Extension Committee on Organization and Policy Personnel and Organizational Development Sub-committee developed a list of relevant core competencies used as part of a national needs assessment to determine existing competencies levels and staff development needs. CES should consider expanding the use of competencies as a foundation for organizational change and improved performance for three reasons. First, the link between individual performance and organization performance drives competency-based learning systems. Competency development focuses on areas in which an individual, team or work group demonstrates outstanding performance and links them to strategic directions. Competency modeling can be a vehicle for moving the Extension organization forward. Second, competency development is a highly participatory process. Extension professionals have the opportunity to identify the knowledge, skills and behaviors they will need to get the best results as well as skills and functions that are no longer effective. Third, the most important reason for Extension organizations to consider competencies is that competency models are powerful decision-making tools. Professional development specialists and administrative staff in businesses and agencies worldwide are finding competencies help make forward-looking human resource decisions by clarifying the knowledge, skills and behaviors needed in the future and by serving as a foundation upon which to build employee selection, training, professional development, performance appraisal, and succession planning. Competency modeling can yield long term benefits through employee selection. First, core competencies for a given position that will meet changing demands are agreed upon. For example, competencies leading to superior performance in a rural community may differ significantly from those associated with outstanding success in an urban setting. A job announcement is posted and applicants are interviewed based on the competencies. Behavior- based interviewing techniques determine if the individual demonstrates the ability to combine and leverage the skills in such a way that they have the potential to become outstanding Extension educators. Competency models are also commonly used in training and development. What learning needs to occur to be effective Extension educators in the future? Since the competencies of individuals and teams are closely linked with strategic opportunities, organizational training priorities can be identified that strengthen our skills and increase our ability to be responsive to the needs of our clientele. Competency-based training encourages Extension educators to assess their level of competence in a given area and participate in training that is relevant, useful and often customized to their learning style. Linking strategic vision and competency-based human resources decisions benefits both individuals and the Extension organization. Because the language of competencies is forwarding thinking and participatory, Extension educators learn to anticipate and recommend appropriate competencies to ensure that education program design and delivery are relevant and useful to their audiences. By identifying strategic competencies and determining training priorities to address high profile issues, we can deliver new and meaningful benefits to our clientele more quickly while demonstrating the value of the Cooperative Extension System to a broader public. It has been suggested that the single most important human resources issue for meeting the challenges of the future is developing our on-going ability to learn (Hamel and Prahalad, 1995). By mastering competency-based tools and applications, we can establish the link between where the Cooperative Extension System is going and what we will need to learn to get there.
Hamel, G. and Prahalad, C.K. (1995) Competing for the Future. Harvard Business School Press, Boston, Massachusetts.
Developing an Extension Pest Management Program Using the Needs-Assessment Process Sue L. Blodgett An Integrated Pest Management (IPM) initiative of the U.S. Department of Agriculture has a goal to develop and implement IPM on 75% of the total US crop acreage by the year 2000. The initiative involves farmers and practitioners in the development and assessment of IPM programs with a goal of increasing the use of IPM systems that enable farmers to achieve both economic and environmental benefits. A multi-phased sequence of needs-assessment programs was used to develop a cereal grains IPM program in Montana, integrating prioritized research and Extension goals. A three step series of surveys and workshops, was conducted to: (a) identify the key insect, weed, and disease problems, (b) prioritize identified pest problems and, (c) develop an action plan with specific research and Extension goals. Additionally, this process promoted dialog between clientele, researchers, and Extension personnel, allowed identification of barriers to IPM implementation, and involved clientele in the program development process.
Key insect and disease pests were identified by Montana producers through a survey conducted with cooperation of Montana Agricultural Statistics Service (MASS), USDA Agricultural Research Service Rangeland Insect Laboratory and Montana State University Department of Entomology. Questionnaires were distributed by MASS with the September Agricultural Survey (SAS) and with the Acreage and Production Survey (A&P). The SAS sampled 1,800 farmers and ranchers by telephone, while the A&P survey sampled 12,000 farmers; 6,000 by telephone and 6,000 by mail. During the phone surveys producers were asked to complete a series of questions. The same questions were sent via a mail survey and followed three weeks later with a reminder notice. Respondents were asked to rate crop damage attributed to insect and disease pests provided in the following categories: none; light (10% or less acres affected); moderate (11-50% acres affected); and heavy (>50% of acres affected). Insect and disease pests were specified, although an 'other' category was provided. A total of 2,500 questionnaires were returned representing 27, 28, and 25% of the total state's acreages of winter wheat, spring wheat (including durum), and barley, respectively. Results were summarized and projected on state acreages of the appropriate crop by adjusting the total acreage of each crop by the percent acreage reported in each damage category (Table 1). Wheat stem sawfly, grasshoppers, wheat streak mosaic virus, and cutworms were ranked by producers surveyed, one through four, respectively as the most damaging insect and disease pests.
Implementation teams were formed in Montana, Idaho, Nebraska, Wyoming, and North and South Dakota to identify pest complexes and coordinate the development and implementation of IPM efforts. A series of strategic planning workshops were funded by USDA, CSREES, IPM Initiative phase I through a multi-state project and MSU IPM-ES program. Leaders from each state's implementation team joined with local producers, county Extension faculty, Agricultural Experiment Station researchers, and agricultural consultants at three regional meetings to develop a user-driven strategic plan for pest management in the northern Great Plains and Intermountain dryland wheat cropping system. The goal of each workshop was to identify local research, technology, and educational needs for IPM implementation. A summary of the three workshops was used to develop a region-wide program. Consensus-driven strategic planning was applied as an iterative process that identified and prioritized pest management issues, and developed plans of action to address critical issues resulting in identification of research and Extension goals and objectives. As a result of this process key insect, disease, and weed pests were identified (Table 2) and six objectives were identified for IPM research and education efforts:
As a result of the survey conducted (Step 1) the wheat stem sawfly was determined to be the most damaging insect pest of Montana small grain crop production. The Strategic Planning Process (Step 2) determined research and Extension approaches, recommended for IPM systems development and implementation in the region. A Strategic Planning Workshop was used to focus specifically on wheat stem sawfly. The goal was to identify research and Extension needs for wheat stem sawfly management in Montana and develop a specific action plan for their implementation. Participants filled out a sensing survey prior to the meeting. At the meeting, results were tabulated, discussed, and issues related to wheat stem sawfly were identified. Once issues were identified, they were clarified, prioritized, and an action plan developed. The three most important issues, identified by participants were (a) development of management strategies (b) research and demonstration programs directed at practical management of sawfly, (c) identify a coordinator and advisory committee for the Montana wheat stemsawfly research and Extension program. The multi-phased strategic planning process has resulted in a coordinated cereal grains research and extension program in Montana. Producers have actively participated in the identification pest management issues, their prioritization, and development of an action plan with desired outputs described. Additionally, this process was integrated with the USDA, CSREES, IPM phase I Initiative and USDA, CSREES, IPM program to provide regional and local coordination. There were other, less tangible benefits derived from the process. Improved understanding of institutional and private constraints, value placed on a variety of research and Extension programs, and an atmosphere of mutual respect and understanding was built between clientele and federal and state agencies charged with delivering to this audience.
Balancing Work and Family in Cooperative Extension: History, Effective Programs, and Future DirectionsRobert J. FetschExtension Specialist, Human Development & Family Studies Colorado State University Cooperative Extension Fort Collins, Colorado Internet address: fetsch@cahs.colostate.edu Mary S. Kennington Extension Director & Extension Agent III Family and Consumer Sciences Institute of Food and Agricultural Sciences University of Florida Citrus County Inverness, Florida Balancing work and family effectively is a continuous struggle for many Cooperative Extension faculty. Extension work often demands long working hours, including nights and weekends. Extension faculty often find themselves with conflicting demands on their time and energy by clientele and administrators' expectations, family expectations, and personal priorities. In these times of uncertain funding, Extension faculty face increased expectations to do more with less. Many Extension faculty also face the complications of governmental partnerships between county and state governments where different superiors sometimes hold incompatible expectations and personnel policies.
Various physical illnesses and psychological problems have been associated with human stress. Stress symptoms include increased use of alcohol and other drugs, insomnia, intestinal distress, decreased concentration and memory, headaches, ulcers, nervous tics, infectious diseases, suicide, and many more (Matteson & Ivancevich, 1982; Selye, 1976). Individuals may exhibit different symptoms of stress based on personal factors such as heredity, nutrition, current health, personality, attitudes, and beliefs. Besides taking a toll on the well-being of individuals, stress is causing problems for corporations. Some experts put the overall cost to the economy as high as $150 billion a year due to absenteeism, diminished productivity, and spiraling medical costs. In addition, Americans are filing increased numbers of stress-related workers' compensation claims, citing problems ranging from surly supervisors to unsafe offices. According to recent research, 14 percent of occupational disease workers' compensation claims were related to stress and about 15 states now make disability payments in cases where anxiety, depression, and other problems have been caused by the work place or the task at hand (Kennington, 1988; Matteson & Ivancevich, 1982).
There are two aspects to balancing work and family in Cooperative Extension. There are steps individual faculty can take and there are steps the Cooperative Extension System can take. In 1981 an Extension Committee on Organization and Policy (ECOP) task force was established to formulate a position statement on Cooperative Extension's role in strengthening American families. ECOP's resultant position paper recommended adapting Extension employee policies to address the needs of Extension faculty: "Extension organizational managers need to critically examine policies and practices in relation to their effects upon the family life of Extension employees" (ECOP Task Force, 1981, p. 3). While a fair amount of research has been conducted on stress and burnout with many employees, research with Cooperative Extension faculty has been spotty at best. Research has been conducted with Extension agents as it relates to stress, stressors, burnout, and balancing work and family in Ohio, Minnesota, Kentucky, Illinois, Pennsylvania, and Colorado. Research was conducted on the stress levels of 241 Extension agents in Ohio during spring, 1985 by Igodan and Newcomb (1986). Burnout has been defined as "a syndrome of emotional exhaustion, depersonalization, and reduced personal accomplishment that can occur among individuals who do 'people work' of some kind (Maslach, 1982, p. 3). Burnout has been identified as an important occupational hazard in at least 25 different people- related fields (Maslach, 1982). Burnout symptoms include: "(1) low job performance/low job satisfaction, (2) physical exhaustion/fatigue, (3) rigidity to change/loss of flexibility, (4) decreased communication/withdrawal, (5) physical symptoms, (6) apathy/loss of concern, (7) cynicism, and (8) emotional exhaustion" (Igodan & Newcomb, 1986, p. 4). Burnout, therefore, relates directly to organizational policies and to administrative styles. At the same time it relates to individual coping behaviors. In Ohio, Igodan and Newcomb (1986) determined that: "As a group, 4-H agents experienced the most burnout, followed by young agents and single agents. Agents who were satisfied with their jobs didn't have much of a problem with burnout, but as job satisfaction decreased, burnout increased.... Typically, burned out agents are more likely to be young (between 20-30 years of age). They are more likely to be single than married, but could be male or female. They tend to be more involved in job responsibilities that relate to 4-H/youth work as opposed to agriculture or home economics, although agriculture agents and 4- H agents have a similar self-reported workload...." Agents with 4-H responsibility employed by Minnesota Cooperative Extension also experienced higher stress levels than agents working in other program areas. A study of Minnesota Extension employees by Patterson and McCubbin (1984) reported that agents working in the areas of Home Economics-Family Living and Home Economics-4-H experienced "the heaviest and broadest set of stressors and demands." Agents working in the joint program areas of Agriculture/4-H and 4-H also experienced numerous stressors and demands. As a group, agriculture Extension agents experienced the narrowest set of stressors and demands. Almost 56 percent of agents reported that job stress had a negative effect on their marital relationship. This was due to some spouses not understanding the nature of Extension work and agents not having enough time to spend with their spouses. Of the agents with children, over half felt that job stress had a negative effect on their parent-child relationship. As far as job stress and its impact on the agents' ability to do their job, more married than single agents reported that stressors had a negative impact on their ability to do their job (37.5 percent to 16.7 percent). (Patterson & McCubbin, 1984). In Kentucky, a study (Fetsch, Flashman, & Jeffiers, 1984) determined that Kentucky 4-H Extension professionals had a higher mean stress level score (M = 73.058; SD = 14.232; N = 154) than the normal adult (M = 60; SD = 9). The authors concluded that Kentucky 4-H agents who experienced prolonged exposure to such high stress levels were at risk for physiological and emotional stress-related problems. In Illinois, 54% of Extension agent respondents, and in Minnesota 56% said that Extension work created strains on their family and/or social relationships. Clark (1983) reported that unmarried Extension agents were more likely to say that Extension work strained family and/or social relationships than did their married counterparts, which is different than was found in the Minnesota study. St. Pierre (1984) found that Pennsylvania agents perceived their jobs to affect their family life more negatively than positively. Agricultural agents and youth agents reported that their jobs affected their family life more negatively than did home economists. In Colorado, the major differences in stress, depression, and life satisfaction levels for Extension agents were found associated not with occupation, gender, or job site, but rather with number of years of service and marital status (Fetsch & Kennington, 1988; Kennington, 1988). The shorter the number of years of employment with Cooperative Extension in Colorado, the lower the overall life satisfaction levels with personal life, relationship with spouse, relationship with children, work life, and family life (N = 120; r = .197; p = .03). Fetsch and Kennington were not surprised to find that the relationship between number of years of service and life satisfaction levels was a statistically significant one--a finding supported by Igodan and Newcomb's findings (1986). They were surprised, however, that the relationship's practical significance was so small--only about 4 percent of the variance was explained by this relationship. Fetsch and Kennington (1988) also found that the lower the number of years of service, the higher the agents' depression levels during the past year. Divorced (M = 4.67) and married (M = 4.92) Extension agents tended to report significantly lower personal life stress levels (p =.034) than did agents who were never married (M = 6.50), remarried (M = 6.44), and widowed (M = 6.33). Based on a Likert scale (1 = very low, 9 = very high), divorced (M = 4.17) and separated (M = 4.00) Extension agents reported significantly lower work-related stress levels (p = .050) than did Extension agents who were married (M = 5.63), never married (M = 6.50), remarried (M = 6.56), and widowed (M = 7.00).
Research indicates that Extension agents and others can improve their lives by practicing stress and time management strategies. Programs that contain time management strategies may reduce coronary risk factors (Bhalla, 1980; Gentry, 1978; Suinn, 1978, 1980). Two-day stress and time management workshops in Kentucky with 154 4-H agents resulted in a significant drop in stress levels (p = .001) a month following the program (Fetsch, Flashman, & Jeffiers, 1984). Tests of probability (e.g. p = .001 which is read as "one one thousandth") raise the question that if one repeated this study, what are the chances that one would find that there are no differences between pre-test and post-test group mean scores. A p = .001 suggests that if one repeated this study in the same fashion as the authors did, that only one time out of 1,000 would the significant results be due to chance. Significant decreases in stress levels and increases in positive coping behavior levels were found in participants of full-day stress and time management workshops in Kentucky (Fetsch & Botkin, 1984). They used Time Management for Busy People (Fetsch, 1991) to teach participants to establish life goals, prioritize daily tasks, develop problem-solving skills, use time and stress management strategies, and create a personal stress management plan. A Colorado study with 122 Cooperative Extension agents who participated in full-day Balancing Work and Family Workshops (Fetsch & Kennington, 1988; Kennington, 1988) found a number of significant increases three months following Balancing Work and Family workshop participation in personal life satisfaction levels, satisfaction levels with spouse or significant other, satisfaction levels with children, and overall life satisfaction levels. These changes were found even though participants' stress levels increased--probably due to seasonal increases in job demands. Participants wrote measurable, realistic goals, used "To Do Lists" daily, took personal time regularly to relax, and practiced saying "no" tactfully to low-priority expectations (Fetsch, 1991). The program impacts of full-day Balancing Work and Family workshops with 70 Florida Cooperative Extension agents were assessed via tests of significance comparing group mean pre-test and post-test (five months later) scores (Fetsch & Pergola, 1990, 1991). Burnout levels decreased (p = .00). Depression levels decreased (p = .03). Personal life satisfaction levels increased (p = .02). All of the above educational programs were tailored to address the identified needs of the participants. The workshops included both time and stress management research findings and practical applications.
There are at least two principal ways to reduce Balancing Work and Family stress and strain: (a) modify organizational policies and practices that contribute to high stress levels (see ECOP, 1981) and (b) implement effective Balancing Work and Family programs that increase Extension faculty's coping skills and productivity. There are steps Extension systems can take and steps individuals can take. First, without systemic changes, individual changes may be short-lived. The following recommendations are made for changes at least at the county, district, and state level, and preferably at the national level:
Second, once a system has significant administrative commitment to solve Balancing Work and Family problems at upper- and middle-management levels, then the changes individuals make to reduce stress and strain levels may be more long lasting. Recommendations for individuals include:
Bhalla, V. (1980). Neuroendocrine, cardiovascular, and musculoskeletal analyses of a holistic approach to stress reduction. Unpublished doctoral dissertation. University of Maryland, College Park. Clark, C. D. (1983). Stress, your job, and you. (Abstract available from author, Illinois Cooperative Extension, Urbana, IL.) ECOP Task Force. (1981). Extension's Role: Strengthening American Families. Lincoln: University of Nebraska. Fetsch, R. J. (1986). Juggling families and careers: Stress management programs that work. In S. V. Zandt (Ed.), Family Strengths 7: Vital Connections (pp. 331-342). Lincoln: University of Nebraska. Fetsch, R. J. (1989). Balancing work and family program impacts in Colorado: An example of family and economic well-being initiative programming. Impact 2000 Programs - 1989. Washington, DC: U.S. Department of Agriculture. Fetsch, R. J. (1991). Time management for busy people (Bulletin 549A). Fort Collins: Colorado State University Cooperative Extension. Fetsch, R. J., & Botkin, D. (1984). [Impacts of a stress and time management workshop]. Unpublished raw data. Fetsch, R. J., Flashman, R., & Jeffiers, D. (1984). Up tight ain't right: Easing the pressure on county agents. Journal of Extension, 22, 23-28. Fetsch, R. J., & Kennington, M. S. (1988, October). Research update on stress, depression, and satisfaction levels of 122 Colorado Extension professionals. Paper presented at the state meeting of Home Economists, Silverthorne, CO. Fetsch, R. J., & Pergola, J. J. (1990). Burnout levels drop significantly for balancing personal, work and family life program participants. Impact 2000 Programs-1990. Washington, DC: U.S. Department of Agriculture. Fetsch, R. J., & Pergola, J. (1991). Effective burnout prevention program. Journal of Extension, 29, 33. Gentry, W. D. (1978). Behavior modification of the coronary- prone behavior pattern. In T. Dembroski et al., (Eds.), Coronary -prone behavior (pp. 225-229). New York: Springer-Verlag. Igodan, O. C., & Newcomb, L. H. (1986). Are you experiencing burnout? Symptoms and coping strategies for extension professionals. Journal of Extension, 24, 4-7. Kennington, M. S. (1988). Research report: Perceived life and job satisfaction levels of Colorado Cooperative Extension agents. Unpublished masters thesis. Colorado State University, Fort Collins. Maslach, C. (1982). Burnout: The cost of caring. Englewood Cliffs: NJ: Prentice-Hall. Matteson, M. T., & Ivancevich, J. M. (1982). Managing job stress and health. New York: The Free Press, Collier Macmillan Publishers. Patterson, J. M., & McCubbin, H. I. (1984). Minnesota county extension agents: Stress, coping, and adaptation. St. Paul, MN: Family Stress and Coping Project. Selye, H. (1976). The stress of life (rev. ed.). New York: McGraw-Hill. St. Pierre, T. L. (1984, May). The relationship between work and family life of county extension agents in Pennsylvania (Abstract). Pennsylvania State University, University Park. Suinn, R. (1978). The coronary-prone behavior pattern: A behavioral approach to intervention. In T. Dembroski et al. (Eds.), Coronary-prone behavior (pp. 231-236). New York: Springer -Verlag. Suinn, R. (1980). Pattern A behaviors and heart disease: Intervention approaches. In J. Ferguson & C. B. Taylor (Eds.), The comprehensive handbook of behavioral medicine: Systems intervention Vol I (pp. 5-28). New York: Spectrum.
Reaching A New AudienceJohn CobournWater Resource Specialist University of Nevada Cooperative Extension Incline Village, Nevada Internet address: jcobourn@fs.scs.unr.edu Sue Donaldson Water Quality Specialist University of Nevada Cooperative Extension Reno, Nevada A major challenge for Extension personnel in the 90s is to respond to changing community needs. In western Nevada, the need to protect limited water resources has spurred faculty to seek out new target audiences for water quality education. One such audience who is responding well is the growing number of people who own and operate small non-commercial ranches in the suburbs. It can be difficult to get people to participate in new, voluntary educational programs without immediate personal or financial benefit. The job is to create an array of approaches to induce "buy in." By creating a flexible program with a many- pronged approach, including a variety of teaching techniques, it is possible to accommodate the diversity of most new audiences. Creative application of needs-assessment, curriculum development, program delivery, and evaluation principles can lead to rewarding outcomes, including broader ties to the community. In 1992, a regional needs-assessment was conducted to help Nevada's water specialists determine major water problems. Geographic areas of concern and various land uses thought to be potential sources of water pollution were identified by knowledgable community leaders and agency representatives in 54 individual interviews. Steamboat Creek in particular was singled out as a water body in need of better nonpoint source (NPS) pollution control. Published data confirmed this sub-watershed to be the most polluted tributary of the area's most important surface water resource, the Truckee River. In 1993, the Washoe County Department of Comprehensive Planning requested that Extension begin a public education program in non-point source pollution in the Dry Creek watershed, a tributary of Steamboat Creek. The county master plan had zoned over 3 square miles on the outskirts of Reno for 2-to-10-acre ranches in sub-divisions on old, large-scale agricultural ranches. New owners were often inexperienced in small ranch management. The planning department recommended a voluntary educational program to encourage implementation of best management practices (BMPs) on small ranches to help reduce NPS water pollution. The University of Nevada Cooperative Extension joined the Washoe-Storey Conservation District, the Natural Resources Conservation Service, and U.S. Geological Survey in forming a coalition to create a voluntary education program. A basic rule of writing, and of education, is "know your audience." What are obstacles to participation in the program? For this particular audience, obstacles could include a fear of regulatory requirements or of the "environmentalist label". At a more pragmatic level, how much time does the audience have to devote to educational activities? Are most employed during weekday, daylight hours, and does the programming thus need to be flexible? For example, Ritter & Welch (1988) found that it was necessary to design a home child care course to fit into the student's free time in an unstructured, self-study manner. Research about the audience began with interviews with the Extension agronomist, Extension horse specialist, members of the Conservation District Board, and the local executive director of the Nevada Farm Bureau, who were acquainted with members of the new audience. They advised on topics that might be of interest, including flood irrigation, animal waste management, and pasture management. It was also learned that many residents were either urban professionals or retired urbanites who had moved to a rural setting to enjoy living on the land, raising a garden, and keeping a horse or two in their small pasture. The county planning department provided a list of all property owners owning more than one acre in the Dry Creek watershed. A letter was sent to all 450 addresses, inviting residents to attend a public meeting where they could sign up for and learn more about a new volunteer/education program. It was emphasized that this was a free program sponsored by the university. Free well water tests were offered to the first ten people to sign-up as an added incentive. A room in a public school in the target neighborhood was reserved for the initial meeting. The first meeting was attended by 50 interested property owners. Slides of local creeks and ponds were shown with an explanation of how best management practices or BMPs were designed as methods to reduce water pollution, and questions answered. The intention to offer free workshops within 6 weeks was announced and their suggestions for topics were sought. One way to create audience buy-in is to involve the public in the development of the program and its materials (Coffman and Watkins, 1991). About half of the attendees at this meeting became regular program participants. A literature search failed to locate any published curriculum designed to reach this particular audience. While there was literature on best management practices for large commercial farms, it was either too general in scope or too technical in terminology to be useful to a group of suburban ranchers with little training or experience in ranch management. In a study of program delivery methods for farmers, Obahayujie and Hillison (1988) had found that part time farmers (similar to our "small ranchers") preferred on-farm demonstrations, newsletters and publications, workshops, and home visits as a method of disseminating Extension information. These elements were included in the program. Coalition members helped rough out a series of 6 topics that could be covered in a series of free Saturday morning pasture workshops during the spring and summer. While agriculture experts would teach each class, the water specialists would weave in principles of pollution prevention as secondary or indirect messages. Using lecture notes and information from other written sources, a curriculum was started that grew into a 96-page book, "The Small Ranch Manual." Before the book was completed, key elements were incorporated into a monthly newsletter sent to neighborhood addresses. Each issue contained a "BMP of the Month," along with background information about how to solve common ranch problems. A basic theme soon developed that became a central tenet of the curriculum: Use of best management practices is not only good for water quality, but also for the health of animals and the appearance and value of ranch property. Written materials normally emphasized practical, "how-to" ranch management information. In the newsletters, in the book, and in presentations, technical language was translated into easy-to-read, interesting, information. The goal was to use clear examples and to explain each concept, practice and technique. Extensive use was made of photographs, drawings, and diagrams in all written material, and most presentations occurred on a ranch, so people could see for themselves. On occasion, students participated in a "work party" where all learned "by doing" as they completed a project for a rancher such as building a compost bin or planting trees to shade a creek. Evaluation of the program was two-pronged. To measure knowledge gain, a 25-question water quality pre-test was administered to each participant when he or she first attended a workshop or presentation. It was explained that the test would allow evaluation of teaching effectiveness and would be a guide for future educational efforts. The same test was given to participants at the end of the second year of the program to document the increase in knowledge. A second evaluation method consisted of documenting actual changes in behavior. An overall goal of the program was to teach and motivate people to implement best management practices for reducing nonpoint source water pollution. After two summers of the program, there was photographic documentation of 22 demonstration projects on sixteen properties, and verbal descriptions of projects on at least 60 other properties in the neighborhood. Such results indicate that a significant portion of the target audience has been reached. The key to success in this program has been the friendly, engaging way people were treated. From the start, it was realized that to attract and hold the audience's attention, the events and social interactions had to be fun. Pasture "workshops" were casual question-and-answer sessions conducted during a leisurely stroll around a neighborhood ranch, with refreshments provided. The intention was to create, if possible, a sense of a community effort and even of neighborhood pride. It was felt that this would increase audience participation and build a program that would virtually run itself, allowing Extension educators to move on to new audiences in other neighborhoods. To cement a sense that volunteers were working together for the good of their shared water resource, the creek, a free barbecue was held each summer at the ranch of a participant. This informal, enjoyable afternoon emphasized that people were important, not just the creek. At each of these social events, the group and certain outstanding individuals were recognized for their good work and for their commitment to "doing the right thing." Certificates of Appreciation were given to these individuals. In addition to the popular group sessions, special attention was given to individuals requesting a visit to their ranch. These site visits allowed participants to ask questions about specific problems and allowed Extension educators to evaluate the property and suggest potential changes in practices. Sometimes this would be as simple as suggesting that pastures be mowed just before weed seeds matured to prevent further infestation. At other times the rancher might be receptive to a more technical fix such as upgrading an irrigation system. In such cases, the education coordinator would offer to schedule another visit and to bring a specialist who could offer specific suggestions or sketches for design of the new system. By visiting ranches and helping participants design actual management improvements, the program again went beyond mere teaching of facts to serve people's needs. While it was up to each property owner to finance his improvement and to either hire a contractor or do it himself, Extension agents would call or stop by every week or two to see how the work was going and offer tips and encouragement. In this program there was no sharp distinction drawn between volunteers and participants. To a degree, all participants were volunteers, because the whole program was indeed voluntary. On the other hand, all participants were encouraged to help Extension agents and to help each other as much as they cared to. They were asked to spread news of the program by word-of-mouth and to bring their neighbors to an event or to a barbecue. Neighbors were encouraged to try an informal buddy system, so they could ask for a helping hand or a bit of advice on occasion. During the first summer, it became obvious that some of participants were eager to do more to help. As fall progressed and outdoor activities became limited, a series of nine indoor classes were offered for the winter as a way to train a core of volunteers. The students had two agreements to fulfill. One was to design at least one BMP demonstration project for their ranch by the end of the course. The other was to perform 10 hours of volunteer service during the following spring and summer. Volunteers were given a list of possible jobs from which they choose activities they would enjoy. While expectations were made clear, class members also knew that they were trusted to fulfill their obligation in ways that suited their individual schedules and educational interests. The number of small farms and ranches surrounding America's cities is, by all accounts, increasing. Since many owners of small tracts are not experienced in agricultural land management, there is a need to reach this new audience and educate them about soil conservation and water pollution prevention. This program has addressed that need in western Nevada. It may serve as a model for similar programs throughout the west. The success of the program stems from a number of crucial factors:
Most important, the target audience was approached with respect, and given sound reasons for joining the program. They were not accused of polluting the water (after all, nonpoint source pollution comes from everyone). It was assumed that the polls were right when they said that 75% of Americans would help clean up our environment if they knew how. The volunteer participants enjoyed meeting their neighbors, they appreciated the technical assistance of our experts, and they took pride in doing the right thing for their community. By recognizing their individual differences, listening to their interests and needs, and responding with a variety of educational approaches, it has been possible to educate a significant portion of this new audience.
Coffman, C.W., & Watkins, S. M., (1991) "Getting the right stuff into the right hands", Journal of Extension, 29 (Spring),.23-25 Obahayujie, J., & and Hillison, J. (1988) "Now hear this!", Journal of Extension, 26 (Spring), 21-22. Ritter, E.M., & Welch, D. T., (1988) Reaching and teaching: A study in audience targeting", Journal of Extension, 26 (Fall), 5 - 7.
Instrument Development for Low Literacy Audiences: Assessing Extension Program Personnel Teaching EffectivenessLaryssa LackmanGraduate Student Department of Agricultural Education Ohio State University Columbus, Ohio Internet: lackman.1@osu.edu Ruben D. Nieto Assistant Professor Ohio State Univesity Extension Columbus, Ohio Internet: nieto.1@osu.edu Rosemary Gliem Graduate Research Associate Department of Agricultural Education Ohio State University Columbus, Ohio 43210 Internet: gliem.1@osu.edu Personnel evaluations serve two main purposes: to provide information for the improvement of performance and to assist administrators in making decisions regarding promotion/tenure and annual performance appraisals (Dick, 1981). The issue of who should provide information regarding personnel performance needs to be addressed. No one source of information for performance evaluation has been found to be the most effective (Fisher, Schoenfeldt, & Shaw, 1990). To achieve objectivity and fairness in an evaluation system for Extension program personnel, administrators must examine data from multiple sources. Sources of valid data could include: clientele, peers, experts, supervisors, as well as the individual instructor. In 1989, Ohio State University Extension (OSU Extension) established an evaluation system for program personnel to use for assessing teaching effectiveness (Spiegel, 1992). The OSU Extension's evaluation system, known as Evaluation of Effective Extension Teaching (EEET), gathers information from clientele, peers, experts, and supervisors. EEET data can be used for self improvement, performance appraisal, and/or promotion and tenure considerations (Nieto & Berry, 1996). Four distinct instruments were developed for the EEET: (a) Group Form (Form I and Form II) -- to be used by Extension clientele in group teaching situations; Form I measures teaching only, while Form II measures educational materials and content in addition to instruction; (b) Individual Form -- to be used by Extension clientele in evaluating one-on-one teaching/consulting settings; (c) Expert Form -- to be used by subject matter experts in evaluating lesson plans and/or educational materials; and (d) Peers Form -- to be used by Extension colleagues and/or supervisors for observation of group teaching situations (Nieto & Berry, 1996). Group Forms, particularly Form I, have been used extensively by Extension program personnel during the last four years. A database was created to provide comparison data to Extension personnel submitting Group Forms (Form I) for analysis. Scores are compared by type of appointment (i.e., district specialist, Extension agent, Extension associate, EFNEP nutrition educator, program assistant, and state specialist), program area (i.e., agriculture and natural resources, community development, family and consumer sciences, and 4-H youth development), and length of employment (i.e., less than two years, two - six years, and more than six years). Appropriate procedures were followed to address validity and reliability concerns for Group Form I. A reliability coefficient of .93 (Cronbach's alpha) was determined for the summated, Likert -scale instrument (a nine-item instrument); in addition, the readability index for the instrument was at the 7th grade level (Spiegel, 1992). Extension program personnel across the state indicated the need for an additional group form designed for clienteles with low literacy levels (e.g., youth, low-income individuals, senior citizens, immigrants). The Expanded Food and Nutrition Education Program (EFNEP) is one example of a program that deals with this type of adience. During the 1995-96 program year, EFNEP educators taught 9,157 eligible families across Ohio. In addition, the EFNEP youth program reached 28,636 youth last year with 75% in the 8-12 year age range (Coplin, 1997). A lack of evaluation instruments to be used by low literacy audiences in assessing teaching effectiveness was determined after conducting an extensive literature review. The 1992 National Adult Literacy Survey revealed that approximately 90 million U.S. adults exhibit low levels of literacy (Kirsch, Jungeblut, Jenkins, & Kolstad, 1993).
The main purpose of this research study was to develop an appropriate evaluation instrument to be used by low literacy audiences for the assessment of Extension program personnel teaching effectiveness. Specific objectives of the study were to: (a) describe the evaluation instruments and constructs used to write the low literacy instrument, (b) outline the procedures followed for establishing content and face validity of the low literacy instrument, and (c) report the reliability coefficients calculated for the instrument.
Four instruments were used to create an item pool for the development of the low literacy instrument: (a) the Ohio State University Extension EEET- Group Form I, (b) Ohio State University Student Evaluation of Instruction (SEI) (Gunter, 1996, 1986), (c) Students' Evaluation of Educational Quality (SEEQ) (Marsh, 1987, 1982), and (d) Arizona Western College Student Appraisal of Instruction (SAI) (Olp, Watson & Valek, 1991). Table 1 displays the number of items each source contributed to the six teaching effectiveness dimensions in the low literacy instrument. After removing duplicate statements, the first draft of the instrument consisted of 50 items.
The items in the pool were revised into more simple language for use with low literacy audiences. The revision process was completed with the help of an Ohio State University literacy program instructor. Fourteen Extension program personnel (county, district, and state levels) working with low literacy clienteles rated the readability of the revised statements. These individuals also commented on the ability of low literacy audiences to use a Likert-type scale. Revisions to statements and to the format were based on the input of the Extension program personnel. Several agents suggested a format using a pictorial scale (smiling and frowning faces) to aid participants in completing the instrument. The instrument was reduced to 21 statements, with seven statements on learning, three statements on enthusiasm, three statements on organization, two statements on group interaction, four statements on individual rapport, and two statements on overall teaching. The Flesch-Kincaid readability index rated the 21 statements at a fourth grade level ("Grammatik 6.0a," 1994).
A panel of experts was asked to indicate the combination of 21 statements that would best measure teaching effectiveness. They also commented on the format, scale, and wording of the statements and instructions. The panel included an Extension associate director, two agriculture education professors with emphasis in teaching, an agriculture education professor with emphasis in evaluation, two Extension district directors, an Extension agent, and a literacy program instructor. Based on the panel's input, nine statements were selected for the instrument, with an optional open-ended statement.
To establish face validity, four Extension agents conducted a field test of the nine-item instrument, with an open-ended statement. The agents were asked to use the instrument with a low literacy audience (n = 50) and obtain comments from the clientele regarding the clarity of the instructions, usefulness of the example provided, user friendliness of the scale, wording and clarity of the statements, and the clienteles' willingness to respond to the open-ended question. Field test responses to the instrument were very positive. Some agents commented that the open-ended question was intimidating and many clients did not feel comfortable writing. One group told the agent that while many program participants did not want to write comments, the question should remain as part of the instrument for those people who want to write comments. Minor changes to the format and title of the instrument were made based on agent and clientele suggestions. Group Form III was suggested to be the name for the new evaluation form to continue the sequence for group forms in the EEET packet. The statements reflected the six dimensions of teaching effectiveness: (a) learning (i.e., I learned a lot from this teacher, I learned something I can use, and the teacher made learning fun), (b) enthusiasm (i.e., the teacher held my interest), (c) organization(i.e., the teacher clearly answered questions and the teacher was easy to understand), (d) group interaction (i.e., I was asked to share my ideas), (e) individual rapport (i.e., the teacher made me feel welcome), and f) overall (i.e., I would take another class from this teacher). The Flesh- Kincaid readability index rated the nine-item instrument at a fifth grade level ("Grammatik 6.0a," 1994). Other readability statistics included: sentence complexity: 5 (100 = very complex) and vocabulary complexity: 15 (100 = very complex).
Test-retest and internal consistency coefficients were calculated to determine the reliability of Group Form III. A pilot test was conducted with low literacy Extension clients. A group of five Extension program personnel across the state, working with a low litercay clientele, was asked to administer Group Form III after completing an educational program. Four of these agents had already participated in the field test of the instrument. Clients were asked to provide the last four digits of their social security number for test/retest (two administrations of the same instrument) purposes. One week later, the same group of clients was asked to complete Group Form III again; paired data were needed to calculate reliability coefficients (percents of agreement). Paired data were collected from a total of 21 individuals. Percents of agreement between the first and second administration of Group Form III ranged from 81% to 95%, with an overall average of 91%. Table 2 displays percents of agreement for the nine statements of the instrument.
Since Group Form III is a summated scale, that is, the nine statements added together represent teaching effectiveness, then a measure of internal consistency as an additional reliability check was required. Cronbach's alpha is a reliability test used to determine the internal consistency of a non-dichotomous summated scale. Data from the first administration of Group Form III in the pilot test were used to run internal consistency. Twenty-seven cases were used for the internal consistency test; a Cronbach's alpha of .74 was calculated for Group Form III. As suggested by Nunnally (1967), reliability coefficients between .5 and .6 are adequate in the early stages of research.
Data have been collected from mid-February to mid-July 1996 from OSU Extension program personnel using Group Form III. Based upon 1,106 evaluations completed, the majority of those using Group Form III are program assistants (45%), followed by administrative and professional agents (43%). The program are submitting the highest number of evaluations was family and consumer sciences (58%), followed by 4-H youth development (38%). Performance evaluation plays an important role in providing feedback for Extension program personnel improvement and in assisting administrators with personnel decisions. The EEET packet lacked an evaluation instrument that could be used with low literacy audiences. Due to the lack of an appropriate evaluation instrument, Extension program personnel either were not collecting evaluation data or gathering inaccurate information in group teaching situations with low literacy audiences. Teaching performance of Extension program personnel needed to be accurately assessed. The availability of an evaluation instrument to be used by low literacy clientele for the assessment of teaching effectiveness provides OSU Extension system the opportunity to gather valid and reliable evaluation data. Extension program personnel are constantly reminded and encouraged to reach unrepresented groups in their programming efforts. The development of Group Form III confirmed Extension administration's views about the importance of serving a wide variety of audiences with its educational programs. However, in addition to an evaluation instrument for low literacy audiences, Extension needs to address the needs of other unrepresented groups such as the Amish, non-English speaking clientele, and various ethnic groups. Extension administration needs to design into the reward system the recognition of individuals documenting their teaching effectiveness with targeted groups.
Coplin, S. (1997). Ohio 1996 Annual Report: Expanded Food and Nutrition Education Program. Columbus: Ohio State University Extension. Dick, R.C. (1981). Chairperson's perspective on faculty evaluation. Indianapolis, IN: Meeting of the Indiana State Speech Association (ERIC Document Reproduction Service No. ED 210 714). Fisher, C., Schoenfeldt, L., and Shaw, J. (1990). Human resource management. Boston, MA: Houghton Mifflin Company. Grammatik 6.0a [Computer software]. (1994). WordPerfect Grammar Checker. Novell, Inc. Gunther, R. (1996). Report to the Council on Academic Affairs on the student evaluation of instruction. Columbus: The Ohio State University. Gunther, R. (1986). Final Report of the Student Evaluation of Instruction Committee. Columbus: The Ohio State University. Kirsch, I., Jungeblut, A., Jenkins, L., and Kolstad, A. (1993). Adult literacy in America: A first look at the results of the National Adult Literacy Survey. National Center for Education Statistics. Department of Education. Washington, D.C. Marsh, H. W. (1987). Student evaluations of teaching. In M. J. Dunkin (Ed.), The International Encyclopedia of Teaching and Teacher Education. New York: Pergamon Press. Marsh, H. W. (1982). SEEQ: a reliable, valid, and useful instrument for collecting students' evaluations of university teaching. British Journal of Educational Psychology, 52, 77-95. Nieto, R.D. and Berry, A. (1996). Evaluation of effective extension teaching (EEET). Evaluation Packet. Columbus: Ohio State University Extension. Nunnally, J.C. (1967). Psychometric theory. New York: McGraw-Hill. Olp, M., Watson, K., and Valek M. (1991). Appraisal of faculty: encouragement and improvement in the classroom. Yuma, AZ: Arizona Western College. (ERIC Document Reproduction Service No. ED 336 159). Spiegel, M. (1992). Synthesizing evaluation: Perspectives, practices and evidence. Proceedings of the American Evaluation Association 92: Extension Evaluation Education Topical Interest Group, Seattle, WA, 27-37.
Evaluating Evaluation -- What We've Learned
K. Chapman-Novakofski
L.S. Boeckner
R. Canton
C.D. Clark
K. Keim
P. Britten
J. McClelland Impact evaluation is becoming more important as accountability becomes more crucial and financial support becomes less available. Indicators of impact are expected in experimental situations, but are also becoming prominent in selected community and health education programs (American Dietetic Association, 1995; American Public Health Association, 1991; St. Pierre, 1982). Although many people intuitively recognize the value of documenting program impact, the literature is scarce outside the realm of large intervention education programs. Extension nutrition educators have long worked to develop national nutrition education impact indicators that would facilitate collection of national Cooperative Extension data (Brink, 1986; Voichick, 1991). In 1993, evaluation questionnaires, reflecting knowledge and behavior change in a pre -, post- and post-post method were distributed nationally by the authors at the meeting of the Society for Nutrition Education and later by a national program leader to a general list of specialists in each state and territory for voluntary field testing. The results of these pilot studies are reported elsewhere (Chapman, Clark, Boeckner, McClelland, Britten, and Keim, 1995; McClelland, Keim, Britten, et al., 1995). However, the evaluation of the Impact Indicators Project (IIP) generated an additional question: What do a) Extension field staff and b) Extension nutrition specialists think about the importance and use of evaluation measures? The Impact Indicators Project included the development of the indicators for nutrition education programs and accompanying instruments to assess knowledge and behavior change. Field testing of the evaluation instruments generated the additional question. This paper reports the results of telephone surveys with Extension nutrition specialists and field staff who participated in IIP. Although two separate surveys and analyses were used, both examined the use of nationally developed evaluation instruments focused on nutrition impact indicators and the perceived importance of the program evaluation process in general.
The basic questions asked during interviews evolved as a consensus of the authors in which key issues pertaining to the evaluation process relative to field staff were identified. The questions were refined, pilot-tested with students for clarity, and formatted using telephone survey techniques (Dillman, 1978). Open-ended questions followed by probes were used to off-set the tendency to provide socially desirable answers about the utility of program evaluation. Since the actual number of field staff who participated in the pilot study was small (n=26) compared to the potential participation across the country (n>3000), analysis of responses consisted primarily of identifying response trends. The strength of trends was described in relative terms rather than absolute terms: Most = more than half; Many = less than half but more than one-third; Some = between one-fourth and one-third; Few = less than one-fourth.
The telephone survey questions for the nutrition specialists were developed and reviewed by the authors. Initial contact was made by telephone or electronic mail to each state to determine the most appropriate specialist to contact, their willingness to participate, and establish a time and date for the interview call. Confidentiality of telephone interviews was maintained. Interview data and notes were sent to one researcher for compilation. Analysis of survey responses from the specialists differed from the method used for the field staff responses since all but three states were represented by specialists. Quantitative results, where appropriate, as well as qualitative responses are provided by the specialist survey responses. Review of qualitative data from the specialist interviews was completed by two research team members. Institutional Review Board approval for both surveys was obtained.
Interviews were completed with 26 of the 28 field staff members from the 12 states that had participated in the pilot study, and with 53 Nutrition Specialists from 47 states and two territories.
The 26 field staff who participated in IIP were a small fraction of the more than 3,000 field staff working in the Cooperative Extension System. Most field staff volunteered to participate in IIP after learning of the program from their state specialists. Many responded that their participation was the result of "requests" or direct instruction from their state specialist. A few felt the tools complemented their previously planned educational activities. All but one of the responding field staff perceived many benefits in using the nutrition impact tools. Most field staff felt that feedback on their programs and the opportunity to use the evaluation methodology contained in the pilot were important benefits. Some felt this information was valuable for future program planning. A few field staff felt a major benefit was the access to a standard methodology with which to measure program performance that would increase the internal validity of their evaluation results. A few field staff also felt that pre-testing provided bench marking that was previously unavailable because post-test only evaluations were typically used. Field staff respondents thought these tools helped verify the value of programming efforts by demonstrating their impact. This demonstration of impact was perceived as instrumental in securing new or continued program funding. A few felt the pre- testing procedure created program awareness and increased the overall quality of the educational activity. The pre-test was seen as promoting higher quality questions and discussion since participants were compelled to think briefly about the topic material in advance of the program. Possible barriers to using evaluation tools included perceptions that: (a) clientele were generally resistive to completing evaluation measures; (b) demographic information was difficult to obtain and offensive to their clientele; (c) written or "pen and paper" types of evaluations were difficult because of literacy and time constraints. Most field staff wanted a short, general, effective questionnaire specific for their program and flexible enough for them to change according to their programming changes and their diverse audiences. Field staff felt they were personally responsible for communicating results to their local funders. They apparently do not routinely use evaluation tools to document impact and thus provide a basis for communication. Less than one-fourth of this small sample reported using a written evaluation measure almost all the time. More than half responded that the last time they could have used a written evaluation tool they did not.
Forty-seven of the 53 specialists who were interviewed recalled seeing the IIP questionnaires. Seventeen specialists from 16 states (36%) participated in the field testing; 27 did not participate, and three were not sure about their state's participation. This participation rate differs from that reported by field staff. It is likely that specialists who considered themselves field study participants had sent IIP questionnaires to field staff who subsequently did not use the tools and failed to report back to the state specialist. Willingness to field test IIP questionnaires was associated with years employed in Extension. In this survey, 52% of the specialists who had worked for Extension ten years or less field tested the instruments compared to only 18% of specialists who had worked in Extension for more than ten years. Responses to questions related to general program evaluation showed that nearly half (48%) estimate they spend from 11-to-25% of their time on program evaluation. Program evaluation processes were described to include the determination of program objectives, development of evaluation instruments, collection and analysis of data and writing evaluation reports. Twenty-eight of the 53 respondents (53%) had evaluation assistance available to them from their universities. Of these 28 respondents, 23 received their assistance from an Extension evaluation specialist. Conversely, 25 of 53 respondents did not have evaluation assistance available to them. If impact evaluation continues to be important and needed, subject matter specialists who also have evaluation expertise may become more critical in the future. Specialists who participated in the IIP field testing (n=17) perceived the following benefits: (a) personal/professional responsibility (9 respondents), and (b) need (7 respondents). Specialists felt the instruments offered a step toward reporting needed national impact data. Comments indicative of a sense of personal or professional responsibility were similar to: "... could see a great benefit if we can report national data." Others indicated their participation in IIP field testing was based on increasing requests by state and federal officials for documentation of impact. Questions about the general nature of program evaluation asked of all specialists who were interviewed revealed the following major categories for the role of evaluation: (a) to show accountability to stakeholders and administrators (24 responses), (b) for program management and development (23 responses), and (c) to show effect on consumers and clientele (12 responses). When questioned about their uses of evaluation methods, specialists often mentioned using end-of-meeting evaluations. Assembling program management and development information has traditionally been met through end-of-meeting evaluations that determine suitability of the learning environment, acceptability of the teacher, delivery of appropriate topics, ideas for improvement of current programs, and suggestions for future programs. Although these are process types of evaluation, they fail to address whether program participants have made changes as a result of the program. When impact evaluations to show program effectiveness were mentioned by specialists, they were program specific, i.e., Expanded Food and Nutrition Education Program or multi-unit in-depth courses. Specialists who did not participate in pilot testing the questionnaires (n=27) were asked their reasons for non- participation. Nearly half (12) indicated they experienced time or staffing constraints that prevented them from undertaking the project. Nearly one-fourth of the respondents (7) indicated handling evaluations through the mail was too cumbersome. Some specialists (6) also perceived a reluctance on the part of Extension field staff to facilitate the process. All of the interviewed extension specialists revealed barriers to general program evaluation. Twenty-three of 53 specialists (43%) indicated experiencing time and dollar shortages. Acceptance of the evaluation process and cooperation by the county faculty (21) and lack of administrative support (8) were additional issues. Nearly one-third (17) reported they lacked expertise for conducting effective impact evaluations.
Although many field staff could relate evaluation results to improving their teaching and programming, others felt it was a waste of time and effort. The finding that staff found few rewards for conducting evaluations suggests that Extension's rhetoric about the need and desirability of measuring program outcomes is not matched by rewards. Field staff also seemed concerned that system-wide evaluation measures need to be sensitive to differences in both clientele and programs. Extension nutrition specialists considered program evaluation to be needed and important, especially for accountability purposes. Perceived lack of time and financial resources were common concerns. As the Extension system moves increasingly toward establishing accountability of programs, effective program evaluations become critical. The following are points to consider for the successful establishment of effective program evaluation within the Extension system:
The implication of this study can be carried over to any discipline or programmatic unit that is attempting to show impact of its local programming. Since capacity building at local and state levels is important if Extension is to show impact, other disciplines or program units may need to assess their personnel for some of these same concerns and barriers.
American Dietetic Association. (1995). Nutrition intervention and patient outcomes: A self-Study manual. Columbus, OH: Ross Laboratories. American Public Health Association. (1991). Healthy communities 2000 model standards: Guidelines for community attainment of the Year 2000 National Health Objectives (3rd ed.). Washington, DC: Author. Brink, M.S. (1986). FY88-91 Extension's food, nutrition and health program emphases and outcome indicators. Washington, DC: Home Economics and Human Nutrition Extension Service-USDA. Chapman, K., Clark, C., Boeckner, L., McClelland, J., Britten, P., & Keim, K. (1995). Multistate impact indicators project. Proceedings from the Society for Nutrition Education Annual Meeting, 20, 45. Dillman, D.A. (1978). Mail and telephone surveys: the total design method. New York: John Wiley and Sons, Inc. McClelland, J., Keim, K., Britten, P., Boeckner, L., Chapman, K., Clark, C., & Mustian, R. (1995). Measuring dietary fat knowledge and behavior using impact indicators. Proceedings from the Society for Nutrition Education Annual Meeting, 20, 46. St. Pierre, R.G. (1982). Specifying outcomes in nutrition education evaluation. Journal of Nutrition Education, 14(2), 49- 51. Summers, J.C., Miller, R.W., Young, R.E., & Carter, C.E. (1981, July). Program evaluation in extension: A comprehensive study of methods, practices and procedures (Executive Summary). Morganton, WV: West Virginia University Cooperative Extension Service, Office of Research and Development. Voichick, J. (1991). Impact indicators project report. Madison WI: Extension Service-USDA.
Forestry Demonstrations: What Good is a Walk in the Woods?
Alison H. Harmon
Stephen B. Jones The outdoor field demonstration is a popular educational tool for Extension programs. People young and old seem to prefer to learn in this manner (Riesenberg & Gor, 1989; Rznewnicki, 1991). But, considering the widespread use of field demonstrations, evaluations of educational effectiveness are surprisingly rare. Educational programs need to be evaluated to allow educators to choose among alternative teaching tools, activities, or delivery styles. Because monetary resources are limited in Extension education, they should be spent providing programs that have been demonstrated to be effective (Pigg 1980; Andrews, 1983). This study assessed the educational effectiveness of a Forest Stewardship Demonstration in Pennsylvania. More than 70% of Pennsylvania's extensive forested lands are non-industrial private forests (NIPFs) owned by more than a half million individuals and families (Birch & Stelter, 1993). These landowners make decisions that affect the long-term viability of Pennsylvania's hardwood industry (a $5 billion industry that employs 100,000), in addition to the sustainability of the broader set of values that society expects from even private forests. While there is a growing need for landowners to be knowledgeable of forest ecology and silviculture principles, most landowners in Pennsylvania say that they need more information to responsibly manage their forests (Luloff, Wilkinson, Schwartz, Finley, Jones, & Humphrey, 1993). Timber harvesting, a primary tool of forest management, is used by foresters not only to yield commercial products, but also to maintain healthy and productive forest ecosystems. Unfortunately, misconceptions about timber harvesting can fuel conflicts among landowners, forest resource professionals, timber harvesters and the public (Jones & Finley, 1995). Because trees and forests enrich our lives in so many ways, timber harvesting has become an emotional and controversial issue. Understanding natural forest processes, as well as some of the basic principles of forest management, may allow the citizens of the Commonwealth to re-evaluate their views and more realistically balance consumptive needs and wants with what the forest can sustainably provide. Timber harvesting must remain a viable management tool. Educational programming may play a significant role in reducing natural resource conflicts by providing an opportunity for and facilitating dialogue among the various interest groups (Jones & Finley, 1993). Demonstrating the various timber harvesting practices that are employed in Pennsylvania is the educational method that this research explores. An Extension education project coordinated by Pennsylvania State University is intended to introduce landowners, the general public, foresters and loggers to the role that timber harvesting plays in sustaining forests. The project, entitled "Integrating Sustainable Forestry into Total Farm Management," is funded by the USDA's Sustainable Agriculture Research and Education Program. Cooperators include the USDA Forest Service, the Pennsylvania Bureau of Forestry, the Pennsylvania Game Commission, the Pennsylvania Bureau of State Parks, the Timber Harvesting Council of Pennsylvania, and Tree Farmer George Freeman. There are seven project sites distributed across Pennsylvania and each demonstrates six different timber harvesting treatments, including no cutting. The five, two-acre cutting treatments include a diameter-based thinning removing the smallest trees, a diameter-limit cut removing only the largest trees ("high-grading"), a thinning of trees evenly distributed above and below the mean tree diameter (replaced by a shelterwood on three sites), an improvement thinning (using SILVAH prescriptions (Marquis & Ernst, 1992)), and a clearcut.
Participants in five Forest Stewardship workshops and meetings during the summer and fall of 1994 served as the sample for this study. Two of the demonstration sites, located in Huntingdon and Clarion Counties, were used in this evaluation. Foresters, landowners, timber harvesters, tree farmers, timber buyers and others constituted each of the audiences. The program had two parts: a scripted slide show and a tour of the field demonstration. The slide show presented objective information about forest history, ecology, silvicultural methods, and threats to the sustainability of Pennsylvania's forests. Participants viewed slides of each of the cutting alternatives. A walking tour of the field demonstrations followed, during which the purpose of each cutting alternative was discussed as were the benefits and consequences of each. Each workshop audience was randomly divided into three experimental groups. Group assignment merely dictated when participants would answer a questionnaire. One group completed the questionnaire upon arrival (control group), the second following the scripted slide show (slides only group), and a third at the end of the demonstration tour (slides/field tour group). All groups viewed the slide show and participated in the walking tour at the same time. The questionnaire, which was standard for all groups, contained a series of objective questions about forests and forest management, and a subjective section asking the degree to which respondents agreed with statements concerning forest management practices. Additionally, we asked respondents for demographic and land ownership information. The attitude/belief subjective questions came from a previous survey and had already been shown to be both reliable and valid measures (Luloff et al., 1993). Objective multiple choice questions came from an established instrument that is used in logger educational programs (L.E.A.P.). We expected that objective test scores would increase once respondents had seen the slide show, but we wanted to know if illustrating concepts using a field demonstration would improve them further. We were also interested in what impact the slides and/or field tour would have on respondents' beliefs about timber harvesting and clearcutting. We focused on clearcutting because this treatment is generally met with more public opposition than are the intermediate treatments in the demonstration.
The total number of people in the study sample was 197. Only about one-fourth of the respondents were women (28%). Respondents were aged 9 to 80 years, and were distributed across a range of levels of education, annual household income, and occupation categories (white collar, blue collar, student, retired and other). Sixteen percent of respondents' occupations were classified as "forest resource dependent" (i.e. foresters, timber harvesters, woodworkers etc.). More than half (57%) of the respondents answered that they had previously attended at least one workshop on forest management. Seventy-two percent were landowners, and about one- third (36%) of all the landowners, claimed to have a written forest management plan. These participants were interested enough in forest management to willingly attend forest stewardship workshops. Therefore, results can only be generalized to individuals with this degree of motivation. From the questionnaire, we derived for each respondent an objective test score (the number of multiple choice questions answered correctly) and two subjective scores: a score for clearcut acceptance, and a score for timber harvesting acceptance. An analysis of variance indicated that objective test score was influenced by experimental group. The slides only group scored significantly higher than the control group and the slides/field tour group scored significantly higher than the slides only group (Table 1). Respondents' education level and attendance at previous workshops also influenced objective test score. Experimental group, however, had the most impact.
The significant increases in mean objective test score following the slide show and following the field tour indicate that both are valuable and effective teaching tools. However, participants appear to learn most when a traditional classroom experience (using slides) is followed by an experience in the field, where visitors witness concepts first-hand. Experimental group did not have an impact on respondents' acceptance of timber harvesting. However, the group that completed the questionnaire after touring the demonstration area was more accepting of clearcutting than the other two groups (p<0.05). Workshop participants who had the opportunity to see and walk through a clearcut first-hand, were less likely to feel that clearcutting should be banned, less likely to think replanting was necessary, and more likely to see clearcutting as an opportunity for new seedlings to grow. With respect to timber harvesting, being interested in forest management, participants were more accepting of timber harvesting when they came to the workshop, leaving less room for change. Workshop participants who are more knowledgeable of basic forest ecology and silviculture concepts are more accepting of clearcutting and of using timber harvesting as a forest management tool. There are a few examples in the literature of studies that link forestry knowledge with attitudes toward timber harvesting and clearcutting (Willhite, Bowles & Talbet, 1973; Becker 1983). Findings from this study are consistent with these and another previous study (Brunson and Reiter, 1996) in which attitudes about timber harvesting were modified when information was provided.
Knowledge gain is often the goal of environmental, natural resource, agricultural and extension education programs. This study is not unique in finding significant gains in knowledge as the result of an educational experience. On the other hand, attitude change is not typically detected as the result of an educational program. However, in this study, one specific attitude change was detected, that is respondents became more accepting of the practice of clearcutting. Many who had been opposed to clearcutting also had misconceptions about clearcutting, for example, "a clearcut removes all vegetation from the site," and confusion about forest regeneration and growth. We believe that landowners in particular will benefit from exposure to timber harvesting alternatives, as they have the opportunity to make these kinds of decisions about their own holdings. However, there is something for foresters and timber harvesters to gain as well, and the general public should be included not only because they are potential landowners, but because the public is becoming increasingly involved directly in decisions concerning the management of public lands, and in influencing decisions on private lands. Natural resource conflicts among resource managers, timber harvesters, landowners and the general public are difficult to resolve because of the drastically varied perspectives involved. We are suggesting that education via demonstration area tours may play a significant role in reducing natural resource conflicts, not simply because forest management knowledge is positively related to acceptance of forest management practices, but because making visitors more familiar with forest ecology and management via demonstration provides an opportunity for discussion among groups with conflicting views. Although not part of the experimental design, audiences represented a full range of stakeholders in sustaining Pennsylvania's forests. Audiences included foresters, landowners, timber harvesters, loggers, and of course citizen consumers. Questions came from the audience throughout the demonstration tour, and often they were in turn answered by other participants. In this way, the leader of the workshop served not as the "expert," but simply as a facilitator of constructive exchange among program participants. The demonstration areas will not solve public conflict by themselves, but they provide illustrations of concepts and issues to discuss. The relaxed nature of the outdoor tour provides an atmosphere conducive to finding common ground, and therefore an opportunity to engage participants in a truly unique learning experience.
Andrews, M. (1983). Evaluation: An essential process. Journal of Extension (September-October) pp. 8-13. Becker, R.H. (1983). Opinions about clear-cutting and recognition of clear-cuts by forest recreation visitors. Journal of Environmental Management XVII. 171-177. Brunson, M.W. & Reiter, D.K. (1996). Effects of ecological information on judgements about scenic impacts of timber harvest. Journal of Environmental Management XLVI. 31-41. Birch, T.W., & Stelter, C.M. (1993). Trends in owner attitudes. in Penns Woods--change and challenge. Finley J.C. and Jones, S.B (Eds.). State College: The Pennsylvania State University, University Park, PA.. Jones, S.B., & Finley, J.C. (1993, Fall). Public forest stewardship ethic. Journal of Extension . 8-10. Jones, S.B., Luloff, A.E., & Finley, J.C. 1995. Another look at NIPFs, facing our 'myths.' Journal of Forestry XCIII. 41-44. Luloff, A.E., Wilkinson, K.P., Schwartz, M.R., Finley, J.C., Jones, S.B, & Humphrey, C.R. (1993). Pennsylvania's Forest Stewardship Program's media campaign: Forest landowners' and the general public's opinions and attitudes: Final report. The Pennsylvania State University. University Park, PA. 20p. Marquis, D.A. & R.L. Ernst. (1992). User's Guide to SILVAH. Stand Analysis, Prescription, and Management Simulator Program for Hardwood Stands of the Alleghenies. Northeastern Forest Experiment Station, Radnor PA. Pigg, K.E. (1980) Program evaluation: Extension needs to get serious about it, Journal of Extension September/October).pp. 7- 13. Riesenberg L.E., & Gor, C.O.(1989). Farmers' preferences for methods of receiving information on new or innovative farming practices. Journal of Agricultural Education XXX. 7-13. Rznewnicki, P. (1991). Farmers' perceptions of Experiment Station research, demonstrations, and on-farm research in agronomy. Journal of Agronomic Education XX 31-36. Willhite, R.G., Bowlus, D.R., & Tarbet, D. (1973). An approach for resolution of attitude differences over forest management. Environment and Behavior V. 351-365.
Profitability Plus Environmental Sustainability Equals Modified Relay Intercropping
Steven C. Prochaska Crop enterprises constitute about 76 percent of farm commodity sales in the seven county area around Crawford County, Ohio. However, grain farmers have had a relatively low rate of return (3.6 percent) on assets according to Forster and Stout (1993). In two studies, one commissioned by Sandoz Agro (1993) and another by Bruening (1991), the most serious problem facing agriculture is contamination of soil and water by fertilizer and/or pesticide use. In Ohio, herbicides are used on over 98 percent of soybean acres (Ohio Agriculture Statistics Service, 1995). To address the major issues of farm profitability and environmental sustainability, a double cropping system, Modified Relay Intercropping (MRI), has been studied. MRI is the planting of soybeans into headed wheat from 20 to 35 days prior to wheat harvest. Goals of the MRI system are to produce wheat at a yield level of 90 percent of monoculture wheat and to produce soybeans at yield level of 60 percent of monoculture soybeans. In the area of environmental protection, MRI offers two advantages. First, the only herbicide used to control weeds in the two crops in one year has been 2,4-D (MCPA in 1995) applied at the labeled rate. The cost for this herbicide has been about $4.00 per acre, compared to an average cost of $29.00/acre for conventional soybean production in Ohio, according to Schnitkey (1995). Thus, not only are weed control costs diminished, but the environment is protected. Since land in the MRI system is in a wheat sod for about 1-1/2 years, soil is protected from erosion. Wheat and soybean production in 10-inch row spacing has been a favorable combination as long as planting dates are in mid-to- late-June. Tram lines are an essential component of the MRI system. One of the questions associated with this production system is the effect of wheat nitrogen rate on soybean growth and maturation. Johnson (1987) states that when nitrogen fertilizer is applied to soybeans, usually smaller amounts of symbiotic nitrogen are fixed and nodulation is delayed. Since soybeans in the MRI system are planted approximately 30 days after the optimum date, it is important for soybeans to grow rapidly to insure crop maturity. To that end, the impact on soybean yield on variable rates of nitrogen applied to wheat has been examined with a replicated study in Crawford County. The objectives of the study were: (a) To measure wheat yield under two nitrogen rates in a MRI system; and (b) To measure soybean yield under two wheat nitrogen rates in an MRI system. Pioneer 2510 wheat was planted in the fall using a Great Plains 15-foot drill with 10-inch row spacing. A tram line (a blank row to allow tractor and implement wheels to travel without damage to the crop) of 18 inches in 1994 or 20 inches in 1995 was established to facilitate soybean planting in June. The two wheat top dress nitrogen treatments were applied in separate applications about six weeks apart. Soybeans were planted with the same Great Plains drill used to plant wheat in June. Wheat was harvested with a John Deere 6620 combine.
Pioneer 2510 wheat was planted in the fall using a Great Plains 15-foot drill with 10-inch row spacing. A tram line (a blank row to allow tractor and implement wheels to travel without damage to the crop) of 18 inches (1994) or 20 inches (1995) was established to facilitate soybean planting in June. The two wheat top dress nitrogen treatments were applied in separate applications about six weeks apart. Soybeans were planted with the same Great Plains drill used to plant wheat in June. Wheat was harvested with a John Deere 6620 combine.
There was no stastically significant differences between nitrogen rates on wheat yield (see table below). There was a significant difference between wheat nitrogen rates on soybean yield in 1995 only. Spring 1995 growing conditions were very favorable for wheat and the canopy cover of the crop was very dense. The extra nitrogen fertilizer applied on wheat in Treatment 2 further enhanced wheat growth. Thus, the soybean growing environment between the wheat rows was more shaded. The soybean yield effect observed in 1995 can be attributed more to a vigorous wheat plant that shaded the 10-inch soybean row than to nitrogen fertilizer reducing soybean nodulation.
In observing the MRI system in various farm trials since 1991, and in replicated research in 1994 and 1995, it was noted that light, or the lack of it, has a profound effect on soybean growth. Soybeans planted too early in the MRI system become etiolated and grow as a vine because the wheat plant shades the row. Wheat row spacing, weather conditions, and wheat variety have all been observed to influence soybean growth, development, and concomitant yield in a MRI environment. The average production of wheat at 68 bushels per acre and soybeans at 34 bushels per acre in 1994 and 1995 would support MRI as a profitable cropping alternative that also protects the environment.
Sandoz, A., (1993). The 1993 Sandoz national agricultural poll. Washington, D.C.: Author Bruening, T. (1991). Communicating with farmers about environmental issues. Journal of Applied Communications, 75(1), 34-41. Forster, L. & Stout, T. (1993). Rates of return to farm assets and other assets, (ESO 2072). Columbus: The Ohio State University, Department of Agricultural Economics and Rural Sociology. Johnson, J. (1987). Soil fertility and crop nutrition. Beuerlein, J. (Ed.). The Soybean in Ohio (pp. 34-41). Columbus: Ohio State University Extension. Ohio Agricultural Statistics Service (1995). Pest management practices used on soybeans. Columbus, OH: Author. Schnitkey, G. (Ed.) (1995). Ohio Crop Enterprise Budgets, Grains-Forages. Columbus: Ohio State University Extension.
An Evaluation of Two Modes of Self-Paced Agent Inservice Training
Jacki N. Fitzpatrick
Stephen F. Duncan
Sally A. Williamson
Thomas A. Smith Extension agents are called upon to provide accurate information to their clientele on a variety of topics. To this end, county agents receive in-depth training through statewide in -service meetings conducted in person by state specialists. Specialists also use a variety of "distance education" strategies, sending a substantial amount of material for programs through mail packets, electronic mail, etc. This material is commonly designed for self-paced learning in the agents' natural work environments. Written and audiotape materials are two commonly used modes of delivering learning materials to county agents. Each mode can be a concise and effective means to promote self-paced learning. One unique advantage of audiotape is that it can be used during the agents' extensive driving time. Extension strongly emphasizes the importance of evaluating the quality and effectiveness of educational programs offered to the community (Astroth, 1991; Butler, 1991; Johnson & Verma, 1990; Ro | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
