December 2002
Volume 40 Number 6

Features

Using Research Methods to Evaluate Your Extension Program

Keith G. Diem
Program Leader in Educational Design
Rutgers, The State University of New Jersey
Internet Address: kdiem@aesop.rutgers.edu

For the Extension practitioner, research is typically considered an ominous practice reserved for ivory tower academics, and evaluation is nearly as mysterious. Therefore, Extension agents often shy away from using scientific methods to evaluate educational programs. However, research is simply a methodical way of finding answers to questions, to be used to discover new information, or to prove scientific theories. And research methods can also be useful to effectively evaluate an educational program or its participants in the most objective way.

For evaluation purposes, the questions to be answered might be "Did this program meet its objectives?" or "How effective was the program in achieving desired results?" The challenge for the evaluator is to choose the most appropriate methods to systematically answer such questions. This article presents an overview of the types of research that might be used in program evaluation, organized by the prospective purposes of your study.

Start by Learning the "Lingo"

Research methods don't seem so intimidating when you're familiar with the terminology. This is important whether you're conducting evaluation or merely reading articles about other studies to incorporate in your program. To help with understanding, here are some basic definitions used.

• Variable: Characteristics by which people or things can be described. Must have more than one level; in other words, to be able to change over time for the same person/object, or from person to person, or object to object. Some variables, called attributes, cannot be manipulated by the researcher (e.g., socioeconomic status, IQ score, race, gender, etc.). Some variables can be manipulated but are not in a particular study. This occurs when subjects self-select the level of the independent variable, or the level is naturally occurring (as with ex post facto research).
  
  
• Manipulation: Random assignment of subjects to levels of the independent variable (treatment groups).
  
  
• Independent variable: The treatment, factor, or presumed cause that will produce a change in the dependent variable. This is what the experimenter tries to manipulate. It is denoted as "X" on the horizontal axis of a graph.
  
  
• Dependent variable: The presumed effect or consequence resulting from changes in the independent variable. This is the observation made and is denoted by "Y" on the vertical axis of a graph. The score of "Y" depends on the score of "X."
  
  
• Population: The complete set of subjects that can be studied: people, objects, animals, plants, etc.
  
  
• Sample: A subset of subjects that can be studied to make the research project more manageable. There are a variety of ways samples can be taken. If a large enough random sample is taken, the results can be statistically similar to taking a census of an entire population--with reduced effort and cost.

One of your first decisions to make is whether to use qualitative or quantitative research methods. Quantitative research focuses on measurement and counting, attempting to categorize and summarize using numbers and labels. Qualitative research aims more at thoroughly describing a situation or explaining reasons for a problem or circumstance. It is typically thorough and provides in-depth understanding of a situation or group of people but does not attempt to quantify results. Often, both quantitative and qualitative approaches are used in a research study or evaluation effort because they provide complementary information. This article deals primarily with quantitative methods.

To select the most appropriate methods to achieve the results you are trying to achieve, it is important to match the purpose of your study with the type of research to use.

Purpose: Explore or Describe
Type of Research to Use: Descriptive Study

If you are looking to gain insight into a problem or issue to better focus additional study or develop a clear research hypothesis, then the end sought is exploration. If you want to illustrate accurately and clearly the characteristics of a group or situation, then the purpose is description.

It is common for any type of research to include descriptive methods. Also, a descriptive method (such as a survey) is often used as the data collection technique for all kinds of research. Start with a research question or research objective.

Examples of research questions:

• How many farmers use no-till methods in the county?
• What is the household income of EFNEP participants?
• How many projects does a typical 4-H member complete each year?

Examples of research objectives:

• To determine the average number of acres of corn planted by dairy farmers.
• To determine the average number of calories consumed per person per day in the household.
• To determine the median family income of children enrolled in after-school child care programs.

Types of descriptive studies include survey research and developmental and case studies.

Survey Research

The researcher gathers data from a large group of subjects, usually via mail, telephone, or in-person interviews. Because information is gathered at one point in time, survey research is sometimes referred to as a "status" or "normative" study. Relationships between variables are not explored. Examples include public opinion surveys, needs assessments, follow-up studies, etc.

Developmental Study

A developmental study is survey research where surveys are taken at different points in time and compared. For instance, take longitudinal studies.

• Trend study: General populations are sampled at each collection point. (Example: a sample of participants from a specific Extension course is taken and studied every year. The sample differs each year.)
  
  
• Cohort study: A specific population is followed over a period of time and sampled at each data collection point. (Example: a sample of this year's participants in an Extension course is taken and studied, and a different sample taken from this year's participants is taken and studied next year and successive years.) Although the same population is studied each year, the sample from that population is different each year.
  
  
• Panel study: An identical sample taken from the initial population at the initial data collection point is used at each data collection point. (Example: a sample of this year's participants in an Extension course is taken and studied for successive years.) Although it is difficult to keep in contact with an identical group over a long period of time, this allows changes in both the group and the individuals in the group to be studied over time.

Case Study

A case study is conducted for similar purpose as the above but is usually done with a smaller sample size for more in-depth study. A case study often involves direct observation or interviews with single subjects or single small social units such as a family, club, school classroom, etc. This is typically considered qualitative research.

Purpose: Explain or Predict
Type of Research to Use: Relational Study

In a relational study you start with a research hypothesis, that is, is what you're trying to "prove."

Examples of research hypotheses for a relational study:

• The older the person, the more health problems he or she encounters.
• 4-H members attending 4-H summer camp stay enrolled in 4-H longer.
• The greater the number of money management classes attended, the greater the amount of annual savings achieved.

Types of relational studies include correlational studies and ex post facto studies.

Correlational Study

A correlational study compares two or more different characteristics from the same group of people and explains how two characteristics vary together and how well one can be predicted from knowledge of the other.

A concurrent correlational study draws a relationship between characteristics at the same point in time. For example, a student's grade point average is related to his or her class rank.

A predictive correlational study could predict a later set of data from an earlier set. For example, a student's grade point average might predict the same student's grade point average during senior year. A predictive correlational study could also use one characteristic to predict what another characteristic will be at another time. For example, a student's SAT score is designed to predict college freshman grade point average.

Ex Post Facto (After the Fact) Study

An ex post facto study is used when experimental research is not possible, such as when people have self-selected levels of an independent variable or when a treatment is naturally occurring and the researcher could not "control" the degree of its use. The researcher starts by specifying a dependent variable and then tries to identify possible reasons for its occurrence as well as alternative (rival) explanations. Such confounding (intervening, contaminating, or extraneous) variables are "controlled" using statistics.

This type of study is very common and useful when using human subjects in real-world situations and the investigator comes in "after the fact." For example, it might be observed that students from one town have higher grades than students from a different town attending the same high school. Would just "being from a certain town" explain the differences? In an ex post facto study, specific reasons for the differences would be explored, such as differences in income, ethnicity, parent support, etc.

It is important to recognize that, in a relational study, "cause and effect" cannot be claimed. All that can be claimed is that that there is a relationship between the variables.

For that matter, variables that are completely unrelated could, in fact, vary together due to nothing more than coincidence. That is why the researcher needs to establish a plausible reason (research hypothesis) for why there might be a relationship between two variables before conducting a study. For instance, it might be found that all football teams with blue uniforms won last week. There is no likely reason why the uniform color had any relationship to the games' outcomes, and it certainly was not the cause for victory. Similarly, you must be careful about claiming that your Extension program was the "cause" of possible results.

Purpose: Determine Cause and Effect
Type of Research to Use: Experimental or Quasi-Experimental Study

An experimental study start with development of a research hypothesis, that is, what you're trying to "prove." Such a research hypothesis is likely based on professional experience or review of prior studies.

Examples of research hypotheses for an experimental study:

• Youth who complete the school enrichment program will have higher math scores.
• Flossing teeth daily prevents gum disease.
• High blood pressure causes heart attacks.
• "Pesticide B" eliminates "Disease A" in soybean crops.
• Participants who complete the course will have increased household incomes.

Experimental research is a methodical way of comparing two or more groups to determine differences in the effect of different treatments received by each group. In experimental research, the researcher purposely manipulates a treatment (independent variable) to see if it causes a change in the dependent variable (effect). A treatment can be an educational program, new drug, herbicide, or procedure that is being tested for its "effect" on the dependent variable.

An example would be giving a new reading program to one group of students and using the old way of teaching reading to a different group of students to see if the new way yields higher reading scores. Extraneous variables are also controlled by the researcher so they can be ruled out as other possible" causes." Experimental research is the only type of study where true "cause and effect" can be claimed.

A true experiment requires the random assignment of subjects (such as people, animals, or plants) to a treatment group. Random assignment is the only way that groups can be considered statistically equivalent.

In a quasi-experiment, groups of subjects are constructed using a method other than random assignment. When using human subjects, it is often impossible to do random assignment. They are often part of intact groups such as school classrooms, community organizations, neighborhoods, 4-H clubs, or nursing homes. Although groups might be reasonably similar in a practical sense, using data from intact groups limits the conclusions that can be drawn regarding program effects. Still, quasi-experiments are useful in providing valuable evidence of program impacts. This is a highly under-utilized evaluation method that has great potential for determining the impact and value of educational programs.

A pre-experimental design has little control over environmental factors that could affect the outcome of a study. For example, a one-group, pretest/posttest design doesn't even use another group for comparison. But such a design does provide some evidence of program impact (with major limitations in the conclusions that can be drawn) and is commonly used when more elaborate designs are not possible. One-group designs can be strengthened as an evaluation method by simply adding a comparison group.

Conclusion: There Is No "Holy Grail" of Program Evaluation

Each research method has benefits, but no method alone is likely to solve all your problems or answer all your research questions. That is why methods are often combined. It just may not be possible to conduct a single study to give a complete and definitive result. Studies are often repeated over time. The most important recommendation is to choose methods that meet your needs and to conduct the study in a careful, thorough, and objective way. Then, you can be confident that your findings can be believed. Therefore, pay attention to the purpose of your study and match up the methods that help achieve that purpose.

The following references will provide help as you select the research methods to use.

Ary, D., Jacobs, L. C., & Razavieh, A. (1985). Introduction to research in education. New York, NY. Holt, Rinehart and Winston.

Brethower, D.M., Brinkerhoff, R. O., Hluchyj, T., & Nowakowski, J. R. (1983). Program evaluation: A practitioner's guide for trainers and educators. Boston, Massachusetts. Kluwer-Nijhoff Publishing.

Campbell, D., & Stanley, J. C. (1963). Experimental and quasi experimental designs for research. Chicago, Illinois: Rand McNally Co.

Diem, K. (1999). Choosing appropriate research methods to evaluate educational programs. Rutgers Cooperative Extension Fact Sheet #FS943. New Brunswick, NJ.

Diem, K. (1997). Measuring impact of educational programs. Rutgers Cooperative Extension Fact Sheet #869. New Brunswick, NJ

Gay, L. R. (1981). Educational research: Competencies for analysis & application. Columbus, Ohio. Bell & Howell Company. 1981.

Hagen, E. P., & Thorndike, R. L. (1977). Measurement and evaluation in psychology and education. New York, New York. John Wiley & Sons.

Wentling, T. L. (1980). Evaluating occupational education and training programs. Boston, Massachusetts. Allyn and Bacon, Inc.

Extension's Role in Responding to Community Crisis: Lessons from Klamath Falls, Oregon

Sharon Cartwright
Assistant Professor
School of Education
University of Alaska
Fairbanks, Alaska
Internet Address: ffskc@uaf.edu

Patricia Case
Assistant Professor
Klamath County Extension Service
Klamath Falls, Oregon
Internet Address: patty.case@orst.edu

Tom Gallagher
Leadership Development Specialist
Office of Professional and Organizational Development
Oregon State University Extension Service
Corvallis, Oregon
Internet Address: tom.gallagher@orst.edu

Ron Hathaway
Director, Klamath County Extension Service
Oregon State University Extension Service
Klamath Falls, Oregon
Internet Address: ron.hathaway@orst.edu

Introduction

The history of Extension is about helping people by providing objective information. To provide this information, the Extension Service in each state develops programs. In Oregon, the programs are agriculture, forestry, family and community development, 4-H/youth, and marine/Sea Grant. There is an informal, overarching program in community development. The challenge for Extension is to provide, through its programs, information necessary to help individuals and communities succeed. Changes in economics, demographics, technology, and the environment challenge Extension faculty, as we begin the 21st century, to provide information that is useful to a changing audience, with changing technology, in a changing world.

These changes are, from an historical perspective, very rapid. Sometimes they are extremely so, and a crisis ensues, as Klamath Falls, Oregon recently experienced. The changes there were abrupt and severe, involving several agricultural communities. This article is about how the Klamath Extension Service responded to a crisis in their community in the summer of 2001 and about what lessons were learned that may be of value to other offices and communities.

Background

In April 2001 the Bureau of Reclamation determined that it could not release the normal allocation of water from Klamath Lake to farmers in the Klamath Irrigation Project. In a period of drought, the Bureau found that the water was needed to protect two species of endangered fish in Klamath Lake. The effect was that over 1,200 families, farming over 220,000 acres, were without their normal irrigation water for the summer of 2001.

The irrigated lands, to the southeast of the city of Klamath Falls in Oregon and California, had been settled in the early 1900s under the federal Homestead Act. A supply of water was assured by federal agreement, and water flowed each summer for over 90 years, until the summer of 2001. With irrigation, the lands of the region produced a variety of forage crops as well as barley, potatoes, and onions. The people and economies of three small towns in the region--Merrill and Malin in Oregon and Tulelake in California--are highly dependent upon agricultural production.

The decision to not allocate water had an immediate and very dramatic impact on farmers who could not water crops. Most chose not to plant crops and tried to find alternative sources of water, mainly from wells, to preserve their perennial crops, such as alfalfa, through the season. As the farmers withheld investments to plant their summer crops, the impacts spread to the local businesses and communities, and then to the community of Klamath Falls and eventually to the county and region.

As the crisis in the communities increased, the faculty of the Klamath County Extension Office recognized that their traditional "technology transfer" information was of little value in this situation. The office faculty are specialists in such fields as livestock, forage, crops, horticulture, natural resources, 4-H/youth, and nutrition education. The specialists soon realized the crisis put their present informational programs "on hold" until the larger crisis for individuals and communities was addressed.

Extension and Crisis

Historically, the Cooperative Extension Service has responded to the problems and crises of communities. This includes responsiveness to small-scale problems such as local economic depressions and regional drought conditions to more nation-wide crises, especially the Great Depression and national efforts during both world wars (Rasmussen, 1989; Vines & Anderson, 1976).

1980's Farm Crisis

During the farm crisis of the 1980s, Extension played a significant role in helping communities recover. Extension's efforts included stress management workshops, family farm communication seminars, intergenerational farm transfer sessions, couples retreats, farm family support groups (Williams, 1998) as well as programs on financial management and marketing skills (South Dakota, 1996). The farm crisis caused severe stress among farmers and farm families (Williams, 1998), and Extension responded by organizing more intensive intervention for farm families. The University of Missouri trained mental health counselors to provide stress management and social service coordination (Meeker, 1992). Iowa State University Extension was able to secure funding to provide long-term (6-9 weeks), one-on-one outreach education to farm and rural families facing severe stress (Viegas, 1998). In other circumstances, Extension formed partnerships to offer valuable services such as a 24-hour Stress Management Hotline (South Dakota).

Extension has also responded to short-term crises during flooding (North Dakota, 1997), drought (Chenoweth, 1991), and even child abduction (Stark, 1990). In most of these incidences, Extension's primary role was to provide reliable information. This information was delivered by radio (addresses and question/answer sessions); television (interviews and informational segments); Web site links; and fact sheets, information packets, and other publications.

Research has supported the use of these techniques in dealing with community crisis. University of Wisconsin-Madison conducted a survey to assess the impact of the farm financial crisis (Williams, 1996). The survey indicated that churches and Extension were perceived as being more responsive to farm families than the helping agencies--social services, community action, health care agencies, mental health agencies--in the community. Williams explains this result in part by farmers' lack of awareness of helping organizations and by farmers' pride, which created a barrier to accessing services. In addition to the techniques listed earlier, Williams recommends Extension connect farmers with assistance by networking with agencies, publicizing these agencies, and by training these agencies on how best to respond to farm families in distress. Other reports have surmised that Extension's success in aiding farmers in crisis is due to the trust developed over many years of collaborating on rural issues (Meeker, 1992).

Helping People Help Themselves

Extension faculty have also developed tools to enable community members to help themselves during times of crisis. Conway, Corcoran, Duncan, and Ketchum (1996) developed Towns in Transition: Managing Change in Natural-Resource Dependent Communities as a video and study guide for community leaders for periods of crisis and transition. The guide highlights actions a leader can take, especially:

• Finding ways of helping community members feel more in control of their future, feel more competent,
• Communicating frequently and consistently,
• Making use of task forces and short range goals for community members to aim for, reach and celebrate, and
• Questioning the "usual", including one's own role.

This type of education suggests an emerging role for Extension professionals. Patton and Blaine (2001), in their article "Public Issues Education: Exploring Extension's Role," state that Extension professionals may be uncomfortable dealing with value-based conflicts (similar to the one in Klamath County) in which research-based (technology transfer) information addresses only a small part of the issue. Rather than providing technical information, the Extension professional may need to assume a facilitator or "process expert" role to help a community recover from crisis.

In his critical review of the modern Extension Service, Land-Grant Universities and Extension: Into the 21st Century, author George McDowell (2000) argues that Extension is in the process of renegotiating its "social contract" as it searches for the best way to serve in the 21st century. Nationally, Extension must broaden its program portfolio to better engage the society it serves. "Engagement means staying attuned to the issues faced by people" (p.196). The Klamath situation provides an example of just such engagement and serves as the foil for proposing some general guidelines for responding to crisis in a way that is both timely and effective.

From a decision-making perspective, the role of the facilitator in a time of crisis is to help the community go through the steps of problem resolution as a group. This challenge, then, is to help the group understand values at risk and to set goals, to gather information to better understand the situation, to generate opportunities for resolution of the crisis, and to make a decision (Gallagher, 1987; 2002).

Klamath Response

The initial response of the director of the Klamath County Extension Office was to contact specialists in the Office of Personnel and Organizational Development (OPOD) in the statewide Oregon State University Extension Office. OPOD is the professional development branch of the organization that provides training in leadership, evaluation, adult education, diversity, and volunteer program development. Several OPOD specialists also design and facilitate community meetings for county offices, and, in 2000, two OPOD specialists helped conduct an assessment of needs in all 36 Oregon counties <http://osu.orst.edu/extension/opod/needsassessment/needsindex.html>.

This assessment provided Klamath faculty with the experience of engaging the broader community in a general--"outside the box" of a particular program--discussion.

As the water crisis in the Klamath developed, Klamath faculty and OPOD specialists designed a series of three needs assessment and resolution meetings, called "Klamath Community Assessments."

Meeting 1

The office director sent invitations to about 50 community leaders, representing businesses, agencies, non-profits, and interested citizens. The meeting, held June 13, attracted 35 participants. The points in the discussion, borrowing the 2000 assessment of needs process previously conducted in the county, included:

1. Introductions (10 minutes)
2. List of Accomplishments (15 minutes)
3. List of Trends (15 minutes)
4. List of Barriers (15 minutes)
5. Question: "What does your community need, in the next 2 to 3 months, to move towards a viable future?" (60 minutes)
6. Alternatives (30 minutes)
7. Multi-Voting (10 minutes)
8. Discussion of "X by Y" (15 minutes)
9. Evaluation and Closure (10 minutes)

Using these steps the group identified four broad needs:

1. Improve information and coordination,
2. Pursue justifiable compensation (for water not delivered),
3. Increase scientific accountability, and
4. Increase public/national awareness.

The group concluded the meeting with specific action items (X by Y) for each issue.

Meeting 2

The second meeting was held on August 2. The process in this gathering, which attracted 25 participants, began with introductions, followed by an update of "needs and actions" identified in the earlier meeting. The group reported several major successes. For example, the county government created the Klamath Disaster Resource Center to coordinate information and respond to the crisis. Also, Oregon State University initiated, with the University of California, a process to provide a scientific assessment of the environmental, social, and economic impacts of the lack of water. And major progress was made on getting the issue before the national media.

After review of the progress to date, the facilitator moved the group to step 5 with the question: "What is needed in the Klamath Community to help sustain itself in the next year?" This question moved the focus of the assessment beyond the 2- to 3-month time frame from the first meeting. The participants, working initially in small groups and then as a whole, identified seven columns of needs:

1. Community spirit,
2. Youth support,
3. Water certainty,
4. Financial counseling,
5. Public policy,
6. Basic needs, and
7. Compensation.

Working in self-selected groups, participants then completed the remaining steps and identified alternative ways to meet the needs. A set of 20 action (X by Y) items were identified, some as simple as holding a community potluck to build community spirit and others more involved, such as engaging the county commissioners to appoint an advisory group to promote public policy. One action item was for the OPOD office to provide policy education, and a training session was held the next month.

Meeting 3

A third meeting was held September 26. After introductions, the dozen participants reviewed successes related to the action items. The successes were numerous, including:

• A community potluck that attracted 60 people,
• Bringing the "Ag in the Classroom" program to local schools,
• An agreement by the local medical service provider to withhold billing for farmers,
• A grant for over $100,000 from two foundations to provide winter clothing for children,
• A $.5 million road improvement program that provided work for 40, and
• Expansion of the food bank and food stamp program.

Despite the smaller group, it was clear that critical participants were present, that earlier participants had worked together creatively, and that a good deal of work had already been accomplished.

The facilitator then led the group through a review of recent events (this meeting followed the September 11 tragedy) and the current state of the community. Participants noted how vulnerable they felt to world problems but how strong they felt as a community. They felt that the confrontation that had been building to forcefully open the head gates that control the irrigation water was now over and that there was renewed opportunity for working together.

There were new aspects to the crisis, such as the lack of recharge from irrigation water causing shallow wells in the region to go dry, thus requiring families to haul all of their water. This new aspect of the crisis led to a request for more information about ground water, and economic impact information being developed by Oregon State University was needed to help with measuring the value of the water not delivered. This meeting closed with a strong sense that the mission of the meetings was accomplished; the series of meetings had met their objective to help the community identify needs and means to resolve them.

Lessons Learned

In review of the outcomes of the meetings, it is not clear that the successes in the community would have occurred without Extension facilitation. Certainly, many parties encouraged cooperation and action. It is clear, however, that several key contacts were made and several key opportunities were discovered in the Extension-facilitated meetings. Further, it was clear that the energy developed at the meetings was helpful in identifying tasks and people willing to do them. In review of the sessions, participating faculty identified the following five lessons learned.

First Lesson: Start Early

Start early! When a crisis develops rapidly, as it did in the Klamath Basin, it is desirable to be active quickly. The first meeting of the community leaders was held about 2 months after the decision to withhold water was made. This was sufficient time for the initial dust to settle and at the time when people were starting to care about "what to do now" It could be argued that Klamath Extension might have anticipated the crisis and been a voice to warn people. Indeed, Extension offices across Oregon are now beginning to recognize that they may wish to begin engaging people before the issue takes on crisis proportions.

Second Lesson: Involve Community Leaders

Get a broad group of community leaders involved. Participants in the meetings were people recognized for their leadership qualities in a great variety of areas, and some were associated with Extension programs. The breadth of participation enabled the group to identify the full range of values, to gather a great variety of information, and to generate a much broader array of alternatives. Interestingly, although the Klamath Basin is known as a small community, many participants knew each other by name and sight but had never worked together. The series of meetings encouraged new acquaintance and developed a host of new leaders familiar with Extension.

Third Lesson: Keep Meetings Focused

Keep the schedule and the meetings focused and productive. We chose to hold community meetings every 6 weeks to have early progress reports on the "X by Y" items and to keep the group energy high. Also, the meetings were managed with a moderately tight agenda so that we did not "just sit around and gripe" about the issue, but moved toward "what are we going to do about it." By concluding each meeting with the X by Y session, we hoped people would see that the time spent was worth their while.

Fourth Lesson: Start Meetings on a Positive Note

To give the meeting a positive, forward-looking feel, we started each meeting by looking at past successes. During the initial meeting, the community members quickly listed a range of "accomplishments" over the past few years--such as completing a new ice rink for youth. This strategy, which is part of a problem-solving method called "appreciative inquiry" (Hammond, 1998), promotes a positive atmosphere that reduces the tendency of people in crisis to focus on the negative. At the second and third meetings, we began with successes to continue this sense of accomplishment and positive outlook.

Fifth Lesson: Use Trained Facilitators

It was desirable to have trained facilitators to help design and guide the initial meetings. The facilitators modeled the process in the first meeting, and, by the third meeting, the Klamath faculty were prepared to run the meeting alone. One of the key tasks of the facilitators was to remain neutral in a situation where it was necessary to honor different points of view. The facilitators anticipated and allowed a measure of "venting," but then guided it into positive energy and action. In many respects, the meetings served as a forum for an "attitude adjustment" for citizens and a training session on meeting facilitation for faculty and staff.

Conclusion

As a Klamath office faculty member quipped at the start of this project, "Helping a farmer with an irrigation system design isn't very useful when the water is turned off." Increasingly, due to the rapid changes in the world around us, there will be a need for Extension to help people respond to crises outside program areas. The day that Extension faculty can remain specialists, dedicated solely to their specific crop or program, is probably over. There is a growing challenge to make sure that the existing Extension programs are relevant. In situations that require synthesis, the existing programs may function too much as silos, and it is between the silos where the problems and solutions reside. The three meetings facilitated by Klamath Extension provided a way for faculty to see the broader concerns of the community and to discover some methods to help address those concerns.

Where We Are Today

Community response to the water allocation was varied and in some cases led to public protest, resulting in national attention on the issues. Secretary of Interior Gail Norton responded by releasing 75,000 acre-feet of water to the irrigation project. The federal government provided $20 million in payments to affected farms. Farmers and state agencies in California and Oregon drilled new wells that provided additional irrigation water.

The farm community continues to live with the uncertainty of secure water delivery for irrigation. Federal agencies, led by the Bureau of Reclamation, and community organizations are continuing attempts to resolve issues related to endangered species and irrigation. The National Academy of Science in Science and the Endangered Species Act (1995) summarizes that "approaches must be developed that rely on cooperation and innovative procedures" to resolve endangered species and natural resource conflicts. Extension can provide relevant programs that support these needed cooperative and innovative approaches.

References

Chenoweth, K. (1996). Responding to crisis: Drought directives. Journal of Extension [On-line]. 29(4). Available at: http://www.joe.org/joe/1991winter/iwl.html

Conway, F., Corcoran, P., Duncan, A., & Ketchum, L. (1996) Towns in transition: Managing change in natural resource dependent communities. Oregon State University Extension Service. EM8648.

Extension assists a state in crisis. (1997, July). Impact EXTRA, North Dakota State University Extension Service. Retrieved November 12, 2001 from http://www.ext.nodak.edu/extpubs/ansci/

Gallagher, T. J. (1987). Problem solving with people: The cycle process. University of Alaska Extension Service: University Press.

Gallagher, T. J. (2002). Decision making: An architect's model for Extension use. Journal of Extension [On-line], 40(2). Available at: http://www.joe.org/joe/2002april/tt1.html

Hammond, S. A. 1998. The thin book of appreciative inquiry. Plano TX: Thin Book Publishing.

McDowell, G. R. (2001). Land-Grant Universities and Extension into the 21st Century: Renegotiating or abandoning a social contract. Ames, Iowa: University Press.

Meeker, D. (1992). From farm crisis to workforce readiness. Economic Development Review, 10(1)79-80.

National Research Council, Committee on Scientific Issues in the Endangered Species Act. (1995). Science and the Endangered Species Act. Washington D.C.: National Academy Press.

Patton, D., & Blaine, T. (2001). Public issues education: Exploring Extension's role. Journal of Extension [On-line], 39(4). Available at: http://www.joe.org/joe/2001august/a2.html

Rasmussen, W. (1989). Taking the university to the people: Seventy-five years of Cooperative Extension. Ames: Iowa State University Press.

South Dakota State University farm crisis response. (1996). College of Agriculture & Biological Sciences Extension Service. Retrieved November 11, 2001 from http://www.agbionews.sdstate.edu/articles/lostincome.html

Stark, S. (1990). Responding to Crisis. Journal of Extension [On-line] 28(1). Available at: http://www.joe.org/joe/1990spring/iw1.html

Viegas, S., & Meek, J. (1998). The Rural Families Program makes a difference. Journal of Extension [On-line] 36(4). Available at: http://www.joe.org/joe/1998august/rb3.html

Vines, C., & Anderson, M. (Eds.). (1976). Heritage horizons, Extension's commitment to people. Madison, WI: Journal of Extension.

Williams, R. (1996). The on-going farm crisis: Extension leadership in rural communities. Journal of Extension [On-line]. 34(1). Available at: http://www.joe.org/joe/1996february/a3.html

Williams, Roger. (1998). The farm crisis: Responding to the stress. University of Wisconsin Press. Retrieved November 11, 2001 at http://narmh.org/impactwo.html

Training 4-H Teen Facilitators in Inquiry-Based Science Methods: The Evaluation of a "Step-Up" Incremental Training Model

Martin H. Smith
Assistant Cooperative Extension Specialist
Veterinary Medicine Extension
University of California-Davis
Davis, California
Internet Address: mhsmith@ucdavis.edu

Richard P. Enfield
Youth Development Advisor
University of California Cooperative Extension, San Luis Obispo County
San Luis Obispo, California
Internet Address: rpenfield@ucdavis.edu

Introduction

Animal Ambassadors is a youth science education outreach program of the School of Veterinary Medicine, Veterinary Medicine Extension, at the University of California, Davis (UCD). Animal Ambassadors uses the world of animals, both domesticated and wild, as a "bridge" to help youth develop an interest in science while emphasizing important critical thinking and life skills. The program's concept-based curriculum stresses awareness and understanding while fostering animal care and responsibility.

Funding was received from the American Honda Foundation in 1999/2000 for a project titled "Animal Ambassadors--A Science Education Outreach Model," a collaborative effort between the UCD School of Veterinary Medicine and the 4-H Youth Development Program in San Luis Obispo (SLO) County, California. One goal of this project was to develop a state and national training model for the dissemination of the Animal Ambassadors curriculum through county-based 4-H Youth Development Programs using cross-age teaching strategies. This article describes a "Step-Up" Incremental Training Model for teen curriculum facilitators, which was designed and evaluated by the authors.

Background

Engaging children in science activities at an early age is crucial to developing a scientifically literate population. In order to accomplish this, we must train and support science educators (Dana, Campbell, & Lunetta, 1997). However, according to the National Education Association (NEA), current science training programs for educators fall short of what is needed to keep pace in an increasingly more scientific and technological world (NEA Today, 1993).

Most educators who work with elementary age children feel unprepared to teach science (Cannon & Sandler, 2000), and science education at the elementary age level has been viewed as weak (Dana, Campbell, & Lunetta, 1997). Hurd (1999) reports that findings from the Third International Mathematics and Science Study (TIMMS) demonstrate that professional development of science educators in the United States is not as well supported as it is in Germany and Japan. Hurd also states that the study's results showed a lack of common vision for science education in the U.S.

Surveys show the value of hands-on training programs in which teachers experience the curriculum components in a manner that models how the children will receive them (Konen & Horton, 2000). Teachers' anxiety level decreases, their confidence level increases, and their interest and curiosity about the subject matter increases. Overall, the more familiar they are with the subject matter and the materials, the greater their comfort level is. Lopez and Toumi (1995) report that in order to deliver a successful hands-on, inquiry-based curriculum, educators require:

1. Research-based, age-appropriate activities;
2. A science kit with relevant materials; and
3. Ongoing professional development in inquiry methods.

Methods

Curriculum

The Animal Ambassadors curriculum activities are age-appropriate and use a hands-on, interdisciplinary, inquiry-based approach that follows the Learning Cycle (Exploration, Concept Invention/Introduction, Concept Application) and emphasizes the Scientific Thinking Processes (observing, communicating, comparing, organizing, relating, inferring, and applying) (Marek & Cavallo, 1997). The content and instructional methodology of the curriculum align with the National Science Education Standards (1996) and the Life Science and Investigation and Experimentation strands of the California Science Content Standards (2001).

The Animal Ambassadors curriculum is subdivided into five units based on major animal-related concepts:

• Animal Habitats and Geography;
• Structure and Function;
• Dietary Needs and Habits;
• Animal Communication; and
• Human/Animal Interactions.

The curriculum is designed around these animal concepts, but does not use animals in its activities. To make the curriculum interactive, hands-on materials (e.g., foot molds, tooth molds, imitation animals coats) are integral to the activities and are organized into learning kits.

Teens as Cross-Age Teachers

Teens as cross-age teachers of younger youth are used commonly within 4-H Youth Development Programs (Lee & Murdock, 2001). Twenty-four teens from three different geographic locations and three 4-H clubs in San Luis Obispo County were recruited for the project. Fifteen teens continued to work with the Animal Ambassadors program in its entirety, attending all training sessions and implementing the program with younger youth. The teens worked in two groups and divided themselves further into teams depending on the nature of the sessions and activities.

Training Model

The "Step-Up" Incremental Training Model designed in this study involves a sequence of three training workshops that alternate with curriculum implementations to ensure mastery of content and methodology while providing the opportunity for group reflection and feedback. Each subsequent training is a "step up" from the one preceding. 4-H teens from San Luis Obispo County were trained as cross-age teachers of the Animal Ambassadors curriculum. Their target audience was primary 4-H members (5-8 year-old children) and young children (5-8 years old) at the local YMCA.

Key elements of the "Step-Up" Incremental Training Model included the following.

Introductory Session

Held on the Friday evening before the first Saturday training workshop, this 3 1/2-hour session began with an ice-breaker activity, followed by an agenda that focused on introducing the fundamentals of inquiry-based learning, including the Learning Cycle, science process skills, and questioning strategies. This introduction was crucial to the success of all subsequent curriculum trainings, laying a foundation of instructional methodology upon which curriculum content was applied during three training workshops.

Teen Facilitator Training Workshop I

During the subsequent full day of training, facilitators concentrated on activities from curriculum Unit I. The limited amount of curriculum content covered was designed to help the teen facilitators develop confidence in their abilities to implement the activities using inquiry methods. This approach is supported by Lee and Murdock (2001), who caution against overwhelming teens during the initial training, recommending an incremental approach instead.

Workshop facilitators modeled the first activity for the teens, placing an emphasis on the instructional methodology from the previous evening's Introductory Session. Subsequently, each team of teen facilitators planned and presented at least one additional activity from Unit I.

After each presentation, a significant amount of time was dedicated to group reflection. Facilitators and trainees spent time reviewing and discussing each activity presented as it related to inquiry methods, curriculum content, and age-appropriateness. Fundamental logistical challenges (e.g., room size, group size) the teens might expect to face during implementation with their target audiences were also discussed. Upon completion of Teen Facilitator Training Workshop I, teams of teens were charged with implementing the activities with young children in 4-H clubs or at the local YMCA.

Teen Facilitator Training Workshops II and III

Teen Facilitator Training Workshops II and III were 1-day curriculum trainings that focused on reflections from field experiences, review of methodology, activity modeling by workshop facilitators, and practice facilitation of activities by trainees. Significant time for group reflection also remained as a major component of both agendas.

Training II, covering curriculum activities in Units II and III, was scheduled approximately 4 weeks after Training I in order to provide the teen teams sufficient time to schedule and implement all activities from Unit I with their target audiences. Curriculum Training III also emphasized methodology and included the activities from Units IV and V. This training was scheduled at an even longer interval (approximately seven weeks) to allow for the larger number of activities to be implemented by the teen facilitators with their target groups. More curriculum material was covered in Curriculum Trainings II and III because of the higher levels of abilities and confidence of the teens gained through their experiences in previous trainings and during curriculum implementation.

Evaluation

In order to strengthen the evaluation design, a method using multiple tools known as triangulation was adopted to assess the training model. This method reduces threats to validity and allows a fuller and richer explanation of a given construct (Denzin, 1970; Coehen & Manion, 1980). The efficacy of the training model as it related to the teens' familiarity with the curriculum materials, their understanding of inquiry-based teaching methods, and their understanding of questioning strategies was measured. Two main tools were used to evaluate the training model: post-training surveys and a post-project focus group interview. Direct observations of teen facilitators during curriculum implementation with the target audience were also used to measure the use of effective questioning strategies. This tool was used as a supplement to questions asked during post-training surveys and the focus group interview.

Post-Training Surveys

Participating teens were asked to respond to post-training surveys developed using a Likert scale of 1 to 5, with 5 being Strongly Agree (SA) and 1 being Strongly Disagree (SD). A category for Not Applicable (NA) was also included. Survey content focused on workshop organization and facilitation, as well as on skills acquisition and use. Data were collected, and results were calculated and compared as percentages.

Focus Group Interviews

Participating teens were asked to take part in a focus group interview after all trainings and the implementations of the curriculum were completed. Two focus group sessions were held, one for each of the two groups. There was a group facilitator and a recorder for each of the interview sessions. In addition, the focus group interviews were audiotape recorded, and the tapes were transcribed. The focus group interview followed a standard format (Krueger, 1994) with four sections:

1. Welcome and thanks,
2. Overview of the focus group topic,
3. The ground rules, and
4. Questions (12), including some very open-ended questions such as "Have we missed anything during this focus group that you want to add?"

The moderator also gave a brief 2- to 3-minute summary of questions and asked for any changes or additions to the summary.

Direct Observations

A Site Observation Data Sheet was developed that was used twice at each site during actual implementation of curriculum activities. Members of the research team counted and rated (low, medium, and high) the instances of scientific thinking processes during set time intervals. The number of closed and open-ended questions posed by teens was also tallied. The open-ended questions were rated as low, medium, or high. Additionally, the observers answered two summary questions relating to how well the teens implemented inductive teaching methods and how well the children responded to the activities.

Results and Discussion

The 4-H teen participants in this project were effective in their roles as cross-age facilitators of inquiry-based science activities with younger youth. Data from 20 pre-/post- matched sets (nine boys; 11 girls) of children between the ages of five and eight were analyzed for changes in critical thinking skills through the use of the Science Thinking Processes.

Critical thinking skills were measured using a performance-based object description assessment tool. Children were given two different objects (pre- and post-test) that were independent of the curriculum content and asked to describe them. Data were quantified using a scoring rubric that measured the types of Scientific Thinking Processes and the manner in which they were used. Positive changes were seen, particularly among girls. These results were paralleled by other data that showed the children used more senses to observe and describe the objects during the post-test and that the children were far more inquisitive about the objects after project intervention, asking 75% more questions in the post-test than in the pre-test.

Elements integral to the design of the "Step-Up" Incremental Training Model provided the teen facilitators with the theory, skills, and practice necessary to achieve success during curriculum implementation. The results of survey and focus group interview data with teens were compared, consolidated, and analyzed. Outcomes and discussion related to these data are presented below.

Time to Practice Animal Ambassadors Curriculum Activities

Becoming familiar with all aspects of any curriculum (content, materials, and methods) is critical to the successful implementation of activities (Loucks-Horsley, Hewson, Love, & Stiles, 1998). Therefore, sufficient time must be allotted during trainings for effective modeling, in-depth investigations, reflection, and guided practice (Pottle, 1993; Loucks-Horsley, Hewson, Love, & Stiles, 1998; Lee & Murdock, 2001).

Post-training surveys were used by participants to rate the amount of practice time they had for curriculum activities during workshops I, II, and III (Figure 1).

Figure 1.
Activity Practice Time

During the post-program focus group interview, several teens reported that they had sufficient time to practice curriculum activities during the trainings. After the completion of all three trainings, some teens stated that they had spent enough time practicing the curriculum activities during the trainings that they did not need their binders or notes during implementation. Some teens shared that practicing activities during trainings helped them learn and apply methods and techniques when working with children. Furthermore, although some participants from an outlying area who had to travel 90 minutes each way thought that the individual trainings were too long, they were also of the opinion that the length of time was necessary to cover the material adequately.

Understanding Inquiry-Based Teaching Methods

The Animal Ambassadors curriculum activities use a hands-on, inquiry-based approach. This method of science education is especially effective in arousing the curiosity of young children and holding their interest (Hinman, 1999). Hinman continues by noting that most science education is taught in a traditional manner that emphasizes the rote memorization and recitation of facts. Consequently, this is the method that is most familiar to facilitators.

After trainings II and III, participants were asked if they understood inquiry-based teaching methods. (This question was unintentionally excluded from the first training questionnaire.) Based on their responses, the teens' understanding of inquiry-based teaching methods improved from Workshop II to Workshop III (Figure 2).