Using Research Methods to Evaluate
Your Extension Program
Keith G. Diem
Program Leader in Educational Design
Rutgers, The State University of New Jersey
Internet Address: kdiem@aesop.rutgers.edu
For the Extension practitioner, research
is typically considered an ominous practice reserved
for ivory tower academics, and evaluation is nearly
as mysterious. Therefore, Extension agents often
shy away from using scientific methods to evaluate
educational programs. However, research is simply
a methodical way of finding answers to questions,
to be used to discover new information, or to prove
scientific theories. And research methods can also
be useful to effectively evaluate an educational
program or its participants in the most objective
way.
For evaluation purposes, the questions
to be answered might be "Did this program meet
its objectives?" or "How effective was
the program in achieving desired results?"
The challenge for the evaluator is to choose the
most appropriate methods to systematically answer
such questions. This article presents an overview
of the types of research that might be used in program
evaluation, organized by the prospective purposes
of your study.
Start by Learning the "Lingo"
Research methods don't seem so intimidating
when you're familiar with the terminology. This
is important whether you're conducting evaluation
or merely reading articles about other studies to
incorporate in your program. To help with
understanding, here are some basic definitions used.
- Variable: Characteristics by which people
or things can be described. Must have more than
one level; in other words, to be able to change
over time for the same person/object, or from person
to person, or object to object. Some variables,
called attributes, cannot be manipulated
by the researcher (e.g., socioeconomic status, IQ
score, race, gender, etc.). Some variables can be
manipulated but are not in a particular study. This
occurs when subjects self-select the level of the
independent variable, or the level is naturally
occurring (as with ex post facto research).
- Manipulation: Random assignment of subjects
to levels of the independent variable (treatment
groups).
- Independent variable: The treatment, factor,
or presumed cause that will produce a change in
the dependent variable. This is what the experimenter
tries to manipulate. It is denoted as "X"
on the horizontal axis of a graph.
- Dependent variable: The presumed effect
or consequence resulting from changes in the independent
variable. This is the observation made and is denoted
by "Y" on the vertical axis of a graph.
The score of "Y" depends on the
score of "X."
- Population: The complete set of subjects
that can be studied: people, objects, animals, plants,
etc.
- Sample: A subset of subjects that can
be studied to make the research project more manageable.
There are a variety of ways samples can be taken.
If a large enough random sample is taken, the results
can be statistically similar to taking a census
of an entire population--with reduced effort and
cost.
One of your first decisions to make
is whether to use qualitative or quantitative research methods. Quantitative research focuses
on measurement and counting, attempting to categorize
and summarize using numbers and labels. Qualitative
research aims more at thoroughly describing a situation
or explaining reasons for a problem or circumstance.
It is typically thorough and provides in-depth understanding
of a situation or group of people but does not attempt
to quantify results. Often, both quantitative and
qualitative approaches are used in a research study
or evaluation effort because they provide complementary
information. This article deals primarily with quantitative
methods.
To select the most appropriate methods
to achieve the results you are trying to achieve,
it is important to match the purpose of your study
with the type of research to use.
Purpose: Explore or Describe
Type of Research to Use: Descriptive Study
If you are looking to gain insight into
a problem or issue to better focus additional study
or develop a clear research hypothesis, then the
end sought is exploration. If you want to illustrate
accurately and clearly the characteristics of a
group or situation, then the purpose is description.
It is common for any type of research
to include descriptive methods. Also, a descriptive
method (such as a survey) is often used as the data
collection technique for all kinds of research.
Start with a research question or research objective.
Examples of research questions:
- How many farmers use no-till methods in the county?
- What is the household income of EFNEP participants?
- How many projects does a typical 4-H member complete
each year?
Examples of research objectives:
- To determine the average number of acres of corn
planted by dairy farmers.
- To determine the average number of calories consumed
per person per day in the household.
- To determine the median family income of children
enrolled in after-school child care programs.
Types of descriptive studies include
survey research and developmental and case studies.
Survey Research
The researcher gathers data from a large
group of subjects, usually via mail, telephone,
or in-person interviews. Because information is
gathered at one point in time, survey research is
sometimes referred to as a "status" or
"normative" study. Relationships between
variables are not explored. Examples include public
opinion surveys, needs assessments, follow-up studies,
etc.
Developmental Study
A developmental study is survey research
where surveys are taken at different points in time
and compared. For instance, take longitudinal studies.
- Trend study: General populations are sampled
at each collection point. (Example: a sample of
participants from a specific Extension course is
taken and studied every year. The sample differs
each year.)
- Cohort study: A specific population is followed
over a period of time and sampled at each data collection
point. (Example: a sample of this year's participants
in an Extension course is taken and studied, and
a different sample taken from this year's participants
is taken and studied next year and successive years.)
Although the same population is studied each year,
the sample from that population is different each
year.
- Panel study: An identical sample taken from the
initial population at the initial data collection
point is used at each data collection point. (Example:
a sample of this year's participants in an Extension
course is taken and studied for successive years.)
Although it is difficult to keep in contact with
an identical group over a long period of time, this
allows changes in both the group and the individuals
in the group to be studied over time.
Case Study
A case study is conducted for similar
purpose as the above but is usually done with a
smaller sample size for more in-depth study. A case
study often involves direct observation or interviews
with single subjects or single small social units
such as a family, club, school classroom, etc. This
is typically considered qualitative
research.
Purpose: Explain or Predict
Type of Research to Use: Relational Study
In a relational study you start with
a research hypothesis, that is, is what you're trying
to "prove."
Examples of research hypotheses for
a relational study:
- The older the person, the more health problems
he or she encounters.
- 4-H members attending 4-H summer camp stay enrolled
in 4-H longer.
- The greater the number of money management classes
attended, the greater the amount of annual savings
achieved.
Types of relational studies include
correlational studies and ex post facto
studies.
Correlational Study
A correlational study compares two or
more different characteristics from the same group
of people and explains how two characteristics vary
together and how well one can be predicted from
knowledge of the other.
A concurrent
correlational study draws a relationship between
characteristics at the same point in time. For example,
a student's grade point average is related to his
or her class rank.
A predictive
correlational study could predict a later set of
data from an earlier set. For example, a student's
grade point average might predict the same student's
grade point average during senior year. A predictive
correlational study could also use one characteristic
to predict what another characteristic will be at
another time. For example, a student's SAT score
is designed to predict college freshman grade point
average.
Ex Post Facto
(After the Fact) Study
An ex post facto
study is used when experimental research is not
possible, such as when people have self-selected
levels of an independent variable or when a treatment
is naturally occurring and the researcher could
not "control" the degree of its use. The
researcher starts by specifying a dependent variable
and then tries to identify possible reasons for
its occurrence as well as alternative (rival) explanations.
Such confounding (intervening, contaminating, or
extraneous) variables are "controlled"
using statistics.
This type of study is very common and
useful when using human subjects in real-world situations
and the investigator comes in "after the fact."
For example, it might be observed that students
from one town have higher grades than students from
a different town attending the same high school.
Would just "being from a certain town"
explain the differences? In an ex post facto
study, specific reasons for the differences would
be explored, such as differences in income, ethnicity,
parent support, etc.
It is important to recognize that, in
a relational study, "cause and effect"
cannot be claimed. All that can be claimed is that
that there is a relationship
between the variables.
For that matter, variables that are
completely unrelated could, in fact, vary together
due to nothing more than coincidence. That is why
the researcher needs to establish a plausible reason
(research hypothesis) for why there might be a relationship
between two variables before conducting a study.
For instance, it might be found that all football
teams with blue uniforms won last week. There is
no likely reason why the uniform color had any relationship
to the games' outcomes, and it certainly was not
the cause for victory. Similarly, you
must be careful about claiming that your Extension
program was the "cause" of possible results.
Purpose: Determine Cause and Effect
Type of Research to Use: Experimental or Quasi-Experimental
Study
An experimental study start with development
of a research hypothesis, that is, what you're trying
to "prove." Such a research hypothesis
is likely based on professional experience or review
of prior studies.
Examples of research hypotheses for
an experimental study:
- Youth who complete the school enrichment program
will have higher math scores.
- Flossing teeth daily prevents gum disease.
- High blood pressure causes heart attacks.
- "Pesticide B" eliminates "Disease
A" in soybean crops.
- Participants who complete the course will have
increased household incomes.
Experimental research is a methodical
way of comparing two or more groups to determine
differences in the effect of different treatments
received by each group. In experimental research,
the researcher purposely manipulates a treatment
(independent variable) to see if it causes a change
in the dependent variable (effect). A treatment
can be an educational program, new drug, herbicide,
or procedure that is being tested for its "effect"
on the dependent variable.
An example would be giving a new reading
program to one group of students and using the old
way of teaching reading to a different group of
students to see if the new way yields higher reading
scores. Extraneous variables are also controlled
by the researcher so they can be ruled out as other
possible" causes." Experimental research
is the only type of study where true "cause
and effect" can be claimed.
A true experiment
requires the random assignment of subjects (such
as people, animals, or plants) to a treatment group.
Random assignment is the only way that groups can
be considered statistically equivalent.
In a quasi-experiment,
groups of subjects are constructed using a method
other than random assignment. When using human subjects,
it is often impossible to do random assignment.
They are often part of intact groups such as school
classrooms, community organizations, neighborhoods,
4-H clubs, or nursing homes. Although groups might
be reasonably similar in a practical sense, using
data from intact groups limits the conclusions that
can be drawn regarding program effects. Still, quasi-experiments
are useful in providing valuable evidence of program
impacts. This is a highly under-utilized evaluation
method that has great potential for determining
the impact and value of educational programs.
A pre-experimental design
has little control over environmental factors that
could affect the outcome of a study. For example,
a one-group, pretest/posttest design doesn't even
use another group for comparison. But such a design
does provide some evidence of program impact (with
major limitations in the conclusions that can be
drawn) and is commonly used when more elaborate
designs are not possible. One-group designs can
be strengthened as an evaluation method by simply
adding a comparison group.
Conclusion: There Is No "Holy Grail"
of Program Evaluation
Each research method has benefits, but
no method alone is likely to solve all your problems
or answer all your research questions. That is why
methods are often combined. It just may not be possible
to conduct a single study to give a complete and
definitive result. Studies are often repeated over
time. The most important recommendation is to choose
methods that meet your needs and to conduct the
study in a careful, thorough, and objective way.
Then, you can be confident that your findings can
be believed. Therefore, pay attention to the purpose
of your study and match up the methods that help
achieve that purpose.
The following references will provide
help as you select the research methods to use.
Ary, D., Jacobs, L. C., & Razavieh,
A. (1985). Introduction to research in education.
New York, NY. Holt, Rinehart and Winston.
Brethower, D.M., Brinkerhoff, R. O.,
Hluchyj, T., & Nowakowski, J. R. (1983). Program
evaluation: A practitioner's guide for trainers
and educators.
Boston, Massachusetts. Kluwer-Nijhoff Publishing.
Campbell, D., & Stanley, J. C. (1963).
Experimental and quasi experimental designs for
research. Chicago,
Illinois: Rand McNally Co.
Diem, K. (1999). Choosing appropriate research methods
to evaluate educational programs. Rutgers Cooperative
Extension Fact Sheet #FS943. New Brunswick, NJ.
Diem, K. (1997). Measuring impact of
educational programs. Rutgers Cooperative Extension
Fact Sheet #869. New Brunswick, NJ
Gay, L. R. (1981). Educational research:
Competencies for analysis & application.
Columbus, Ohio. Bell & Howell Company. 1981.
Hagen, E. P., & Thorndike, R. L.
(1977). Measurement and evaluation in psychology
and education.
New York, New York. John Wiley & Sons.
Wentling, T. L. (1980). Evaluating
occupational education and training programs.
Boston, Massachusetts. Allyn and Bacon, Inc.
This article is online at http://www.joe.org/joe/2002december/a1.shtml.
Copyright © by Extension
Journal, Inc. ISSN 1077-5315. Articles appearing in the Journal become
the property of the Journal. Single copies of articles may be reproduced
in electronic or print form for use in educational or training activities.
Inclusion of articles in other publications, electronic sources, or systematic
large-scale distribution may be done only with prior electronic or written
permission of the Journal Editorial Office, joe-ed@joe.org.
|
