|
December 2005
|
       |
Contents
Editor's PageSome JOE Style Points: Article vs. ResearchAs I've explained before, I am starting to build a JOE style guide in "bits and pieces." The bit I'm tackling this month is the confusion between an article about research and the research, itself. Many articles submitted to JOE refer to "this research." But the articles are not research; they are reports or descriptions of research and research findings. The research was performed before the articles about it were written. We're talking two different things. On first reference, I generally change "this research" to "the research reported here" and, later in the articles, change "this" to "the." However, there are other ways to maintain the important distinction between the research you have done and your article describing it. By the way, articles aren't studies, either. They're about studies. This JOE style point will also be posted on the Help for JOE Authors page. December JOEThe first two Feature articles in this issue, "The Craft of Cross-Cultural Engagement" and "Is 10% Good Enough? Cooperative Extension Work in Indian Country," deal in quite different ways with the same topic: extending Extension across cultures. The second article, especially, reverberates with a Tools of the Trade article in the October issue, "More Tips: What If a Cooperative Extension Professional Must Work with Native American Institutional Review Boards?" I call your attention to both articles and suggest you revisit the third article, too. There are also a number of fine articles on 4-H, on evaluation, on volunteers, and on water quality. How we can use technology to better serve and understand our audiences comes in for its share of attention, too. A lot of good articles on a lot of important and interesting topics. What can I say? It's a good issue. Best wishes for the holiday season. Laura Hoelscher, Editor
The Craft of Cross-Cultural EngagementCraig A. Hassel IntroductionWhat Is Cross-Cultural Engagement?Cross-cultural engagement (CCE) is a distinct type of community-based engagement, partnering with citizens whose knowledge does not necessarily correspond to scientific models or Eurocentric worldviews (Hassel, 2004). Although "non-scientific" knowledge is often considered "off limits" at research universities, the practice of CCE embraces ancient or ancestral knowledge understandings that may be valuable resources for solving societal problems. CCE practice has the following characteristics:
Why Is CCE Needed?In most cases, Extension professionals have focused on how to get diverse stakeholders to understand the western/scientific view of the world prevailing in land-grant research universities. But little attention has focused on getting academic professionals to understand knowledge systems within communities that lie beyond a western/scientific orientation. Because their knowledge is usually excluded from serious academic consideration, community-based partners in CCE may feel little reason to trust working with land-grant research universities (Hassel, 2004). CCE is a practice in which scientists and Extension professionals can learn to recognize and better understand the knowledge that underserved stakeholders bring to the table. It focuses explicit attention on embedded ideas about how the world works--worldviews and "ways of knowing." A worldview can be defined as basic assumptions and beliefs that form the basis of a people's comprehension of the world (Cajete, 2000). "Ways of knowing" refers to epistemology--differences in the nature of knowledge and its construction, including what counts as knowledge and the degree to which knowledge is certain (Nisbett, Peng, Choi, & Norenzayan, 2001). These ideas are elaborated below. By directing attention toward understanding the world from within different worldview orientations, CCE represents a means of helping Extension professionals to better appreciate the existence and academic value of multiple worldviews. In bringing voices of diverse communities together with the work of the academy, both benefits and challenges arise for Extension professionals who wish to practice CCE (Hassel, 2004). What Are the Benefits and Challenges of CCE?The practice or "craft" of CCE offers a path for academic professionals to respectfully engage knowledge that lies beyond the boundaries of conventional scientific thought. By bringing diversity of perspective to academic work, CCE enhances opportunities for innovation. Greater innovation can generate solutions to societal problems and lead to wealth creation for communities (Barker, 2000). Conducted appropriately, the craft of CCE represents a means to:
The challenges of CCE include:
Legitimate opportunities for the "two way" dialogue essential to CCE can occur if academic professionals are open to learning about knowledge systems beyond those of a western/scientific orientation. CCE as a Two-Way ProcessIn engaging diverse communities, CCE practice emphasizes community-based partners as a valuable source of knowledge. In this regard, CCE builds upon the concepts of engagement put forward by NASULGC (1999a): " . . . two-way partnerships, reciprocal relationships between university and community defined by mutual respect for the strengths of each" where the "purpose of engagement is not to provide the university's superior expertise to the community, but to encourage joint academic-community definitions of problems, solutions and success." Holland (2001) explicitly stated the ideal that engagement with "external constituencies" can benefit communities while improving the university: An engaged institution is committed to direct interaction with external constituencies and communities through the mutually-beneficial exchange, exploration, and application of knowledge, expertise and information. These interactions enrich and expand the learning and discovery functions of the academic institution while also enhancing community capacity. In the case of CCE, participants often feel that they have been denied opportunities for the kind of two-way exchange described above. For example, indigenous education has helped American Indian people to understand the western/scientific view of the world, but little help is available to academic professionals wanting to understand indigenous worldviews (Semali & Kincheloe, 1999). Two-way exchange implies building communicative space perceived by all who participate as "a level playing field" for authentic exchange of perspectives (Hassel, 2004). Given our history of a more one-sided educator-delivery model, creating and maintaining what everyone perceives as a level playing field can be a challenge, but it is a prerequisite for further progress. Role Shift for Academic ProfessionalsAcademic and Extension professionals who create and maintain authentic communication are likely to perceive a shift in their professional role from content expert to one that feels more like an acolyte or co-learner attempting to get up to speed (Carr & Kemmis, 1986; Simpson & Driben, 2000). This is to be taken as a positive sign, even though it may feel awkward at first. It is a sign of engaging within another way of understanding the world that may be completely unfamiliar. Practicing skillfully the craft of CCE requires one to think within a knowledge system that lies beyond the boundaries of a western/scientific perspective. This skill can be facilitated by appreciation for the deeper worldview and epistemic aspects of cultural diversity. Understanding CCE Through the Concept of WorldviewThe concept of worldview is critical to practicing CCE. This is because scientific inquiry, like any other form of human thought and expression, is based upon culturally constructed worldviews about how the world works (Wallace, 2000). As mentioned earlier, a worldview represents a set of assumptions and beliefs that form the basis of a people's comprehension of the world (Cajete, 2000). A worldview is comprised of cultural "givens," basic assumptions and metaphysical ideas that tend to be taken for granted and as non-negotiable. It represents the least visible, yet most entrenched aspect of our thinking. Most of us are not conscious of our worldview because it is not consciously learned so much as implicitly absorbed from our surrounding culture. If a worldview goes unchallenged, members may find behavior based on any other premise almost inconceivable. For example, a materialistic worldview is so entrenched within scientific thinking that it has become difficult for scientists to accommodate knowledge that is grounded in subjective experience, or contextual knowledge of place (Wallace, 2000). Being able to recognize these ideas as assumptions, not givens, allows one the room to "let go"--at least temporarily--of entrenched assumptions. This awareness is vital to the practice of CCE. The work of Nisbett, Peng, Choi, and Norenzayan (2001) argues that social organization filters perception of experience, which influences one's worldview. Worldview, in turn, guides epistemology--what is important to know and how knowledge can be obtained. Because worldview represents the foundational level of a culture, it also represents the ultimate source of cultural diversity (Figure 1). It provides grounding for epistemologies used to construct knowledge (western/scientific methods, Chinese medical theory, Indigenous knowledge systems). Figure 1 illustrates progressively deeper yet more powerful aspects of culture. Figure 1.
Adapted from Schein, Organizational culture and leadership, Jossey-Bass, San Francisco, 1992, pp15-20. CCE Brings Multiple WorldviewsA key feature of CCE is that it brings multiple worldviews together around a problem or issue. CCE helps us as Extension professionals to acknowledge the diversity of metaphysical orientations and worldviews existing within communities. This diversity helps us to recognize as cultural constructions the worldviews foundational to scientific thinking as well. By bringing to the table alternative worldviews, CCE helps to expose materialistic worldviews that:
CCE brings contrasting East Asian worldviews that:
CCE also brings indigenous worldviews that:
Academic professionals may be unaware of the metaphysical ideology lying beneath western/scientific approaches. Scientists investigating other cultures frequently inspect tangible objects (artifacts) using scientific methods, but usually dismiss underlying epistemologies such as Yin/Yang theory, Qi, and Indigenous knowledge (Leslie & Young, 1992; Semali & Kincheloe, 1999). Preoccupation with a western/scientific perspective offers no place for epistemologies incongruent with a materialistic worldview, excluding potentially useful ideas (Hassel, Hafner, Soberg, Adelmann, & Haywood, 2002). One outcome of successful CCE practice is to recognize that diversity of worldview and epistemology is a valuable resource for innovation and should be preserved, not discarded. The Craft of CCECCE practice represents broader, more inclusive work for Extension professionals. Its currencies are relationship and trust with communities over time. Trust and relationships, in turn, are built upon a sincere and demonstrated willingness to:
The craft of CCE is more about working with partners on a journey of learning and change than following a preconceived research design or blueprint for action. Extension professionals need to be willing to adapt their academic agendas to the open-ended, unpredictable processes of CCE. Examine Ideas About ScienceThe craft of CCE first asks academic professionals to be aware of that which we tend to take for granted. CCE challenges us to critically examine the worldview assumptions and scientific ideology that we bring to our work (Hassel, 2004). CCE does not question the success of western science. But it does challenge some ideas about science, like the notion that "science provides the only credible view of the world," or "science provides our only source of genuine knowledge about the world." These notions do not represent scientific knowledge per se, because they are not the result of scientific experiments (Wallace, 2000). Rather, they are a priori assertions or beliefs about the superiority or universality of science. Challenge to tightly held assumptions disguised as reality can be destabilizing to one's cognitive world, creating anxiety and fear (Schein, 1992). CCE requires the intellectual humility needed to critically examine embedded assumptions and to accommodate other ways of understanding the world. To summarize, CCE first asks us to:
Assimilation or Pluralism?Second, CCE brings diverse worldviews to the table, offering a rich source of alternative knowledge, ideas, and practices for consideration. The craft of CCE asks academic professionals to consider alternative knowledge from within the worldview of its origin. This statement implies that merely learning about different worldviews from a familiar frame of reference is insufficient. CCE asks academic professionals to actively shift their own perspectives in order to experience learning and thinking from within the worldview being explored. This means "trying on" the assumptions and values of another worldview for the purpose of reasoning within a different knowledge system. The metaphor of "walking in another's shoes" is relevant to pluralistic thinking skills. Time, critical reflection, and substantive, ongoing relationships with non-academic communities are required to gain these skills. CCE discourages the reflexive instinct to automatically impose scientific evaluation criteria upon any knowledge that is accessed (Hassel, 2004). To do so presumes either superiority of western/scientific epistemologies or coherence between the materialistic worldviews of western science and other worldviews. It is crucial not to distort diverse ways of knowing by forcing them to conform to epistemologies of Western science (Aikenhead, 1997). From a community-based perspective, science often appropriates cultural artifacts (acupuncture, herbs, wild rice) that achieve scientific standards of "evidence" while discarding epistemologies (Qi, Yin/Yang theory, Indigenous knowledge) that are incoherent with scientific understandings (Hassel et al., 2002; Hassel, 2004). Accordingly, scientific advancement is often seen to leave a wake of destruction upon the philosophical underpinnings of other knowledge systems (Fruehauf, 1999; Tuhiwai-Smith, 1999; Scheid, 1999). An excellent example is institutional policies regarding intellectual property that do not recognize or give protection to knowledge of oral traditions. These consequences may be invisible to academic professionals unskilled in reasoning within another knowledge system. Destructive consequences may be avoided through action research in which decision-making lies with community-based partners (Hassel et al., 2002). Such consequences can be minimized if all participants are engaged in examining presumptions and come to consensus on whether scientific criteria are appropriate. The craft of CCE also asks us to:
Issues of Power and ControlThird, the craft of CCE works best when programmatic leadership and ownership lie with community-based partners. This point gets to issues of trust, power, and control, including prevailing cultural norms, values, and power dynamics of large land-grant universities. CCE participants may distrust these cultural norms, because they can be damaging to knowledge systems not corresponding to western/scientific perspectives (Fruehauf, 1999; Tuhiwai-Smith, 1999; Hafner, Hassel, Soberg, Adelmann, & Fetch, 2004; Hassel, 2004). Community-based leadership allows CCE participants to define problems and set agendas according to their perspectives and priorities. Community-based leadership can also facilitate for Extension faculty the role shift from content expert to that of co-learner (Carr & Kemmis, 1986; Simpson & Driben, 2000). Extension professionals can then engage in the work of listening and empathetically experiencing other worldviews and ways of knowing. If undertaken sincerely, these actions facilitate building trust and developing long-term relationships. CCE asks that:
DiscussionEngaging across cultures is not new to Extension. The late Erl Bates, once Advisor on Indian Extension, Cornell University, reported the following (1949): The fundamental idea behind the Indian extension program is to build on the best of their ancient civilization and to add those things in ours which will enable them to live better. The craft of CCE as described here, asks us to think carefully about:
Pursued from different worldviews and epistemologies, judgments about what constitutes "the best of ancient civilization" or "living better" will clearly yield differences. CCE advocates a shift in thinking from a one-way process of assimilation into a dominant western/scientific perspective toward a two-way process of engaging multiple worldviews, each producing its own truths. In considering the questions above, CCE asks academic professionals to loosen ties to academic fundamentalism--the refusal of the academy to value any truth that does not conform to its own professional standards (Gerber, 1997). The rush to impose "solutions" derived exclusively through a western/scientific perspective can reflect a process of "colonizing" as eloquently discussed by Tuhiwai-Smith (1999). The first priority of academic work according to the craft of CCE is that it increases the potential or capacity of a community to live well as defined by the community itself. Academic concerns of creating "valid" knowledge from a scientific perspective are subordinated to the first priority of creating benefit for the community (Maxwell, 1984). In addition to understanding the dispositions and applying the principles described above, the craft of CCE is facilitated through:
Extension professionals skilled in CCE can bring cultural diversity, breadth of perspective, and innovation to the core of the academic enterprise at land-grant research universities. In so doing, we can help the land-grant university serve as a resource through which to access, network, exchange, and navigate the many different forms of expertise within academic and non-academic communities. AcknowledgementsThe author wishes to acknowledge the Woodlands Wisdom Confederation of Tribal Colleges, Medicinal Herb Network, Dream of Wild Health Network, Visions For Change, and Leadership For Institutional Change for the insights that contributed to this manuscript. This work resulted from funding support from the following sources: USDA Agricultural Experiment Station Project MIN-54-059, University of Minnesota Extension Service, USDA CSREES Projects #99-47002-0746 and #2001-52102-1177, Minnesota Food Stamp Nutrition Education Program, Minnesota Institute for Sustainable Agriculture (MISA), Agriculture Utilization Research Institute (AURI) and Organic Farming Research Foundation (OFRF). ReferencesAikenhead, G. (1997). Toward a first nations cross-cultural science and technology curriculum. Science Education 81:217-238. Barker, J. (2000) Wealth, innovation and diversity: The spiral of hope. Presentation to College of Human Ecology Centennial Celebration, Oct 31, St. Paul, MN. Bates, E. (1949). Extension Service with American Indians, Ch XIX. In: The peoples colleges. Cornell Univ. Press, Ithaca, NY, p 201. Cajete, G. (2000). Native science. Natural laws of interdependence. Clear Light, Santa Fe, MN. Carr, W., & Kemmis, S. (1986). Becoming critical. Education, knowledge and action research. The Falmer Press, Philadelphia, PA. Fruehauf, H. (1999). Chinese medicine in crisis. Science, politics and the making of "TCM." Journal of Chinese Medicine 61: 6-14. Gerber, J. M. (1997). Rediscovering the public mission of the Land-Grant University through Cooperative Extension. In W. Lockeretz, (Ed.) Visions of American agriculture. Iowa State Univ. Press, Ames IA, pp175-186. Greenwood, D. J., & Levin, M. (1998). Introduction to action research. Sage Publications, Thousand Oaks, CA. Hafner, C., Hassel, C., Soberg, R., Adelmann, J., & Fetch, C. (2004). Toward bridging perspectives with integrity. Reg. Chinese Herb. Med. J. 3(3):4-12. Hassel, C., Hafner, C., Soberg, R., Adelmann J., & Haywood, R. (2002). Using Chinese medicine to understand medicinal herb quality: an alternative to biomedical approaches? Ag. Human Val. 19:337-347. Hassel, C. (2004). Can diversity extend to ways of knowing? Engaging cross-cultural paradigms. Journal of Extension [On-line], 42(2). Available at: http://www.joe.org/joe/2004april/a7.shtml Holland, B. (2001). Exploring the challenge of documenting and measuring civic engagement endeavors of colleges and universities. Unpublished paper given at the Campus Compact Advanced Institute on Classifications for Civic Engagement, March 23, 2001. Available at: http://www.compact.org/advancedtoolkit/measuring.html Leslie, C., & Young, A. (1992). Paths to Asian medical knowledge. Berkeley, CA: University of California Press, pp. 1-18. Maxwell, N. (1984). From knowledge to wisdom: A revolution in the aims and methods of science, Blackwell Press, Oxford. NASULGC (1999a). Kellogg Commission on the Future of State and Land-Grant Universities. Returning to our roots. The engaged institution. Available at: http://www.nasulgc.org/publications/Kellogg/engage.pdf Nisbett, R. E., Peng, K., Choi, I. & Norenzayan, A. (2001). Culture and systems of thought: holistic versus analytic cognition. Psychological Review 108(2):291-310. Paul, R. (1993). Critical thinking. What every person needs to survive in a rapidly changing world. Foundation for Critical Thinking, Santa Rosa, CA. Scheid, V. (1999). The globalisation of Chinese medicine. Lancet 354 (9196 Suppl 1): 10. Schein, E. (1992). Organizational culture and leadership.Jossey-Bass, San Francisco, CA. Semali,, L. & Kincheloe, J. (1999). What is indigenous knowledge and why should we study it? In: What is indigenous knowledge? Voices from the academy. Falmer Press, NY, pp3-57. Simpson, L. R., & Driben, P. (2000) From expert to acolyte: Learning to understand the environment from an Anishinaabe point of view. Am. Indian. Culture. Res. J. 24(3):1-19. Tuhiwai-Smith, L. (1999). Decolonizing methodologies: Research and indigenous peoples. Dunedin, New Zealand. University of Otago Press. Wallace, B. A. (2000). The taboo of subjectivity. Toward a new science of consciousness. Oxford University Press, New York, pp17-39, 2000.
Is 10% Good Enough? Cooperative Extension Work in Indian CountryJoseph G. Hiller IntroductionExtension is not reaching Indian Country. This article presents a brief background of life in Indian Country, a historical snapshot of Extension work on Indian reservations as related to conventional non-Indian Extension work and reveals a number of institutional obstacles blocking progress towards equity. As a starting point, it is recommended that national dialogue with tribes and tribal organizations is needed. The extensive background section in this article is intended to familiarize the reader with key historic and contemporary features of American Indian life on reservations. Brief comments are made on frequently asked questions related to Indian Country. Finally, this article concludes with a sketch of who is doing what and suggests we reevaluate the local Extension model and refine it to operate more effectively in Indian Country. As is the case with most Extension field programs, work in Indian Country suffers from an inequitable distribution of scarce resources. The U.S. Department of Agriculture's (USDA) Cooperative State Research, Education, and Extension Service (CSREES) administers the national Extension Indian Reservation Program (EIRP) and the "1994" Tribal College Land-Grant programs. From a national perspective, however, these programs are only beginning to address needs of underserved reservation-bound American Indians. The programs are reaching surprisingly few Indian communities. Many tribes and reservations have no Extension services whatsoever--be it from EIRP, a 1994, the tribe, or conventional 1862 county-based Extension services. It should be noted that "Indian Country" is a legal term, not a derogatory one. It is used in Title 18, U.S. Code to generally describe land within the boundaries of a federally recognized reservation, but includes other lands as well. Included in the legal definition are "dependent Indian communities" whether inside or outside of a reservation, lands acquired by tribes and nations, and allotted lands holding Indian title. There are over 175 different indigenous languages spoken by American Indians in this country. BackgroundIndian Country Extension ironically exhibits a living reenactment of predecessor Extension work in rural America. Historically, county Extension agents worked predominantly in rural counties and had to establish personal relationships with farm families before beginning to deliver educational programs. Agents communicated regularly with boards of county commissions and special interest groups to develop support and suggestions for Extension work. This was pioneering work in rural America and began a new era for the 1862 land-grant and the U.S. Department of Agriculture. Today's contemporary county-based Extension programs benefit tremendously from over 90 years of visibility, particularly with respect to institutional and political relations at the community level. Extension's history and prominence in a community commonly leads to habitual and often profitable relationships with clientele. This arrangement provides important political support for the sustainment of Extension programs. Extension programs exist in 3,057 counties across the nation, staffed by 8,987 FTEs at the county level--including U.S. territories (CSREES, 2003; NACO, 2003). In a similar way, Extension work in Indian Country is moving along the same trajectory, albeit without the benefit of community familiarity. Clientele are typically rural dwellers. Many live in villages, settlements, and small towns, some are rural non-farm or farming/ranching/fishing/subsistence families. As with county Extension programs, productive and valued personal and community relationships are critical precedents to successful programs. These relationships must be built almost entirely from scratch in Indian Country--even today. There is virtually no historical relationship between a tribal government and an Indian reservation Extension office that compares to, or parallels, that between off-reservation county Extension offices and county government. There are 562 tribes and 314 federally recognized Indian reservations in America. By comparison, there are dedicated Extension programs on 27 reservations (28 FTEs) under the EIRP. There are a number of important differences between Indian Country and the rest of America that may help explain how Extension and the 1862 land-grant universities relate to America's Indian tribes, nations, and communities. The illustration may also help us understand why a few reservations have as many as three or four independently operating Extension services, while most have only minimal contact, at best, with Extension in any form. Indian Country 101--A PrimerWho Is Indian?An "Indian" or "Native American" is a person recognized as such by a federally recognized tribe. Tribal enrollment requirements are typically specified in a tribe's constitution or tribal law. There are no U.S. government standards of blood quantum. Many tribes require certain degrees of blood for enrollment; others require proof of lineal descendancy from an ancestor originally enrolled as a member of the tribe. Individual Native Americans are U.S. citizens. An "Indian Tribe" (or Nation, Community, Colony, or Alaskan Native Village) is an ethnological and a legal term. Over a dozen tribes are petitioning for federal status through the Bureau of Indian Affairs (BIA), litigation, or congressional action. Tribes having state recognition, with or without a reservation, have different legal standing from federally recognized tribes when dealing with other forms of government in the U.S. About TaxesGeneralizations about the tax environment in Indian Country are made only at considerable risk of being inaccurate. Property taxes cannot by law be paid on reservation trust lands--for the same reason that the U.S. Forest Service pays no property tax on the federal estate under its control. Members who live on-reservation but work off-reservation in non-tribal enterprises pay both state and federal income taxes as appropriate (NGISC, 2003). Some states and some tribes have negotiated tax exchange arrangements on fuel, tobacco sales, and other miscellaneous taxes, but resulting agreements are far from uniform across the nation, and many are under legal review. Land IssuesLands held in trust for the tribes by the U.S. government through its agent, the BIA, dominate most reservations. The issue of land tenure in Indian Country is exceedingly complex--it drives issues of home siting, agriculture, loans, mortgages, commercial enterprises, and virtually all other potential uses and incomes from land. There are two general categories of land on reservations, tribal and allotted. Tribal land can be either "tribal original" or "tribal purchase." Allotted land can be either "undivided interest heirship" or "single ownership" lands (Miller, 1989). Tribal original land has been in trust status since the formation of the reservation, never having been owned by the U.S. government. Tribal purchase land comes from transfer to the tribe by an allottee or heir. Allotted lands were taken from tribal ownership and granted to individual Indians under the series of federal allotment acts. Undivided heirship is the result of allotted land that has been passed to heirs in the form of shares, not as individually identified tracts. After only a few generations, a typical allotment may have several hundred shareholders. Single ownership land usually results from an allottee will or sole surviving heir. There are obvious implications for land management activities such as access, grazing, logging, fishing, hunting, gathering, mining, farming, and other uses that are often confounded by the status of land ownership. There is no known parallel or similarity of this type of land tenure situation in non-Indian Country anywhere in the world. Indian GamingThe Indian Gaming Regulatory Act (IGRA) of 1988 governs legal gaming on Indian reservations (P.L.100-497; 25 U.S.C. лл2701). Tribes are required to enter into gaming compacts with their respective state legislatures that are subject to approval by the Secretary of the Interior. Casinos may only be located on trust lands, although some tribes have successfully petitioned to convert fee simple titled lands to trust for this purpose. About 57% of all tribes are engaged in gaming. Nationally, Indian gaming amounts to about 10% of all legal gaming in America (NIGA, 2003). Per Capita ChecksOne does not automatically receive a government payment every month as a result of being Native American. Some tribes issue per capita payments to members in a fashion similar to corporate stockholder dividend payments. Revenues from natural resources or business enterprises provide a basis for payments when there are profits to be distributed. About 25% of gaming tribes issue per capita payments to individual enrolled members. These payments are not allowed, however, until the Secretary of Interior certifies that tribal gaming revenues first provide for government services, economic and community development, general tribal welfare, charitable donations, and requirements for aid to local governments. Higher EducationHaskell Indian Nations University, the Southwestern Indian Polytechnic Institute, and the Institute of American Indian Arts are operated by the BIA and offer exemption from or reduced tuition to members of federally recognized tribes. They join America's four military academies and the Uniformed Services University of Health Sciences as federal universities. One's tribe does not automatically pay expenses for members to attend college--even to a tribal college. They may, however, grant scholarships and financial aid to members (AIHEC, 2003). Some tribes have established scholarship endowments in support of non-Indian and Indian students in need. Indian Country Extension ModelWith the brief background provided, one can see the conventional American model for Extension work in Indian Country is challenged with political, financial, historical, and institutional situations much different from those in a conventional Extension relationship. There is no political counterpart to county-level government within most tribal governments, although most tribes have political subdivisions of central government--often called "agencies," "districts," "villages," or "chapters." There is generally no tax or revenue flow from the subordinate levels to the higher central government on a reservation unless commercial (i.e., gaming) or natural resources produce the income. The legal government-to-government relationship between tribes and states may differ between states, particularly with respect to taxes. Tribes enjoy sovereignty and special relationships with other governments in America that are typically described in treaties, legislation, or court decrees. They are not subordinate to counties or states. The propensity in CSREES for competitive funding for Extension work makes the matter worse. Tribes do not generally have resources to provide for in-kind or cash match on grant proposals. They do have, however, significant familiarity with short-term federal programs that tend to have marginal impact and are not sustained over time. Ninety years of formula funding made the U.S. Extension organization the model for worldwide adoption, yet Indian Country Extension/EIRP operates exclusively on year-to-year funds. In view of this, tribes pose tough questions about long-term commitment to EIRP personnel that are difficult to answer. Does the same dialogue occur in counties? What is Being Done--or Not--About Extension WorkThere is an eclectic mix of higher education and Extension providers in America. Some are operating in Indian Country--but only in a few places. Providers include county Extension, the Extension Indian Reservation Program, Tribal college Extension, and Tribal Extension. All these programs are constrained to varying degrees by political boundaries of state borders, Indian reservation boundaries, county lines, and community college districts, to name a few. County ExtensionAs early as 1930, Congress rejected a request by the Office of Indian Affairs to send funds to USDA for Extension work on Indian reservations (Kelsey & Hearne, 1949). One typically assumes the 1862 land-grant county Extension program is mandated to serve Indian Country in locations where Indian reservations coexist with counties. It is often a challenge to persuade county government to de facto allow county funds to be disbursed on an Indian reservation, in support of county-based Extension work. Certainly there are instances of cooperation, collaboration, and mutual benefit. But there are many cases where this is not true, and the resulting political and governmental obstacles are indeed daunting. The county-based Extension model may be ill built for this challenge of serving America's Indigenous people. Similar institutional paradigms can be found in state legislatures and land-grant universities. Extension Indian Reservation ProgramIn l988, the Intertribal Agriculture Council and the Southwest Indian Agricultural Association lobbied Congress and federal agencies in an effort to reinstate federal support for Extension programs on American Indian reservations (Smitman, 1989). Together, they represented nearly 70 tribes. Section l677 of P.L. 101-624, the Food, Agriculture, Conservation and Trade Act (the "Farm Bill") is the result of their work, authorizing the EIRP. Initially authorized at $8.0M, but funded at $1.0M in 1991 and supporting 14 projects, it has grown slowly to $2.0M and 28 projects on 27 reservations in 15 states nationwide--now reaching about 4% of all reservations and tribes. EIRP places Extension agents/educators and their educational programs on Indian reservations. Their principal audience is residents of the reservation. These offices are much like a conventional rural county Extension office. The agents/educators are employees of the 1862 land-grant Cooperative Extension organization. Many report directly to the state Extension director; others report through a neighboring county Extension director or administrative head. Some are faculty, but some are not. Nationally, only one is tenured, owing that status to a previous position and career in non-Indian Extension. Only one has ever been promoted. Program funding is nationally competitive on 5-year cycles, with non-competitive annual renewals. EIRP agents are not able to write proposals for outside federal short-term funding that show their salaries as partial match, because their salaries are paid with federal funds. Accordingly, they are not able to compete in this arena with their peers. 1994 Tribally Controlled Colleges Extension ProgramCongress enabled and funded the Equity in Educational Land Grant Status Act of 1994 (Section 354 of P.L. 103-382). The Act initiated a USDA revenue stream to existing tribally controlled colleges in 12 states and issued them land-grant status within the community of 55 state universities ("1862's") and 17 Historically Black Colleges and Universities ("1890's"). There are 33 tribal colleges; 31 are designated as "1994's." The 1994's receive four types of funding from the Act, including:
Tribal college Extension programs have been shaped by tribal college influences, by community needs, and by the competitive grants processes at CSREES. USDA is not, however, the sole source of tribal college funding. They commonly receive funds from the tribe, from congress, the executive branch agencies, and from various grants. Five states have both EIRP and tribal college Extension; these programs are co-located in several instances. The 1994's are operated by about 5% of all tribes. Several tribes are developing new colleges and will eventually apply for 1994 designation. Tribal ExtensionYet another form of Extension exists on a few reservations that are remnants of the first Extension work in Indian Country. Dating as early as 1892, the "boss farmer" worked for the "Indian Agent" (later called the "BIA agency superintendent") and managed tribal agricultural operations (Rooks, 1910; Bruguire, 1999). In the 1940's the boss farmer effort began to model after the USDA-Agricultural Extension Service organization, but still differed in a number of significant respects. Boss farmers were BIA employees. There was no funding relationship with county, state, or federal government (except through the Department of Interior) and no formal link to USDA or to the 1862 land-grant institution. Most of these early boss farmers had received agricultural training and degrees from 1862 land-grant institutions and regularly cooperated with neighboring county agents. Most of the boss farmer positions had disappeared by the mid-1950's, with a few still on the books in the early 1980's. For a short period of time in the latter years, the BIA contracted with several 1862's to conduct Extension work on reservations, but these arrangements disappeared as well. USDA Ag CensusThe USDA Census of Agriculture has historically not reported on Indian Agriculture until the last few years. The data are admittedly difficult to acquire, but for many generations, USDA apparently considered that Indian reservations had only one farm and only one farmer--the "tribal farm" and the tribal "boss farmer." Accordingly, livestock and agricultural production data apparently came from only one farm. This could be to blame for the error of delaying access to USDA farm programs for individual agricultural producers on many reservations for several decades. It may have also adversely affected the distribution of Smith-Lever Extension formula funds to some states. Those states with larger-than-thought numbers of farms due to under-reported Indian agricultural producers and farms may benefit from a recalculated formula. A full and accurate Indian agriculture census is urgently needed, along with a commensurate reevaluation of USDA formulas for funding that are driven by these data. So What?Extension is not reaching Indian Country. The clientele group is small in number and has not been able to attract the political attention within USDA, CSREES, and NASULGC necessary to establish a solid financial foundation and future. EIRP is administered as a national program, but is in fact a regional program with a small clientele group. This makes it difficult to endorse proposals for program enhancement and expansion on a cost effectiveness and national need basis. There are unique challenges for Indian tribes as they seek equity in access to Extension. Dialogue is urgently needed. Federal legislation funding EIRP and 1994's suggests that Congress believes Indian Country should have the same kind of access to Extension as non-reservation county residents across America. The reach of these programs is inadequate on a national scale--many tribes and reservations are simply not being served. EIRP and the 1994's, when combined, reach less than 10% of American Indians on reservations. Generally, tribes are interested in and are seeking Extension services, but long-term funding partners such as Smith-Lever are virtually nonexistent. Tribes currently served by EIRP regularly solicit additional resources for existing and lobby for additional programs for neighboring tribes. Non-EIRP tribes seek program support and expansion through their congressional delegations, the Intertribal Agriculture Council, the Southwest Indian Agriculture Association, the National Congress of American Indians (NCAI, 2004), and other supporters. Half of the tribes conduct gaming, but revenues are not accessible for Extension programs because of language in the National Indian Gaming Act that restricts how these revenues are used. There are few other sources of revenue to support Extension on reservations. State legislatures are reluctant to appropriate funds for Extension work on reservations. There is some well-developed confusion about the various Extension programs operating in Indian Country. This makes it difficult to enhance and expand programs. Only 36 states have Indian reservations. Fifteen of these states have managed to secure funding for EIRP. Twenty-one other states could be building relationships with Indian Country to establish and enhance EIRP. Perhaps they are doing so, but there have not been recent national discussions on the issue at CSREES, USDA, or the National Association of State Universities and Land-Grant Colleges (NASULGC). A high priority should be placed on developing formal programs to serve more reservation-bound American Indians with additional field-based operations. The EIRP model presents some hope that Extension may be "coming soon to a reservation near you" for American Indian residents on reservations so they have the same access to Extension programs as any other American citizen. Many tribes, nations, and communities have appealed for Extension programs, but without adequate political advocacy and lukewarm federal support, the program is unlikely to grow as needed. ReferencesAmerican Indian Higher Education Council (AIHEC). (2003). Retrieved March 22, 2004, from http://www.aihec.org/ Bruguier, D. (1999). Joseph Harper Cash Memorial Lecture. Institute of American Indian Studies, South Dakota State University. December 11, 1999. Cooperative State Research, Education, and Extension Service (CSREES). (2003). Salary Analysis of Cooperative Extension Service Positions. Retrieved May 12, 2004, from http://www.csrees.usda.gov/about/human_res/pdfs/report2002.pdf Kelsey, L. D., & Hearne, C. C. (1949). Cooperative Extension work. Ithaca, NY: Comstock Publishing, Cornell University Press. Miller, R. (1989). Prepared Statement of the Intertribal Agriculture Council Before the Oversight Hearings on Indian Agriculture. Senate Select Committee on Indian Affairs. November 2, 1989. National Association of Counties (NACO). (2003). Retrieved March 22, 2004, from http://www.naco.org/ National Congress of American Indians (NCAI). 2004. Resolution FTL 04-004: Support Enhanced Authority and Increased Budget for the Extension Indian Reservation Program (EIRP). National Gambling Impact Study Commission (NGISC). (2003). Retrieved March 22, 2004, from http://govinfo.library.unt.edu/ngisc/ National Indian Gaming Association (NIGA). (2003). Retrieved March 19, 2004, from http://www.indiangaming.org/ National Indian Gaming Commission (NIGC). (2003). Retrieved March 18, 2004, from http://www.nigc.gov/nigc/laws/igra/overview.jsp Rooks, C. (1910). Sworn Affidavit. Kadoka, Stanley County, South Dakota. January 24, 1910. General Federation of Women's Clubs of South Dakota. Smitman, G. (1989). Prepared Statement of the Intertribal Agriculture Council before the Oversight Hearings on Indian Agriculture held by the Senate Select Committee on Indian Affairs, Intertribal Agriculture Council. Billings, MT. November 2, 1989.
Limited Resources--Growing Needs: Lessons Learned in a Process to Facilitate Program EvaluationCristobal Aguilar Suzanne Thornsbury Department of Agricultural
Economics IntroductionThis article draws on a process used to evaluate member perceptions of program effectiveness for a commodity organization where the program beneficiaries are also the funding source. While vast literature covers evaluation procedures and theory, there is little information on practical evaluation examples linked with this theory, a gap this article addresses. In today's economic climate, public and private entities are critically evaluating expenditures as they search for more efficient and effective allocation of available resources. Program evaluation is often undertaken by the funding source who, as a part of the assessment, wants to see measures of effectiveness in addition to a descriptive report of activities (O'Neill, 1998). Program beneficiaries are also important evaluators and must ultimately register positive impacts in order to justify program continuation. Commodity and/or industry organizations have been traditional partners in Extension programming and frequently serve as both funding source and program beneficiary. It is not only Extension that must increase evaluation efforts in order to generate systematic and convincing evidence of programming value (Stupp, 2003) but also traditional users of that programming. Like most state Extension services, resources within commodity organizations themselves are becoming more limited, with members demanding increased services and greater accountability from their own boards. The role of Extension educators expands to include facilitating program evaluation and providing measures of accountability for their partners' activities (Decker & Yerka, 1990). Limited resources often require commodity groups to make difficult decisions regarding allocation of resources among programmatic areas of emphasis (e.g., production vs. marketing). We use a recent project with the Michigan Apple Committee (MAC) to illustrate the process and draw attention to critical steps for a successful evaluation. MAC is an organization of approximately 1,000 apple growers in Michigan, supported through a check-off on commercial apple sales. These grower resources are then used to fund six programmatic areas: Advertising, Merchandising, Export, Public Relations, Industry Services, and Apple Research. The MAC program evaluation provides insight about the process, linkages to group decision-making, and lessons learned through implementation. Evaluation ProcessThe evaluation process is straightforward, yet there are critical points that an Extension educator must address to maximize useful feedback for their partners. The general steps in the evaluation process are first outlined below and then discussed in greater detail in the following sections. In the discussion we compare the literature on program evaluation with our own experience to emphasize critical points of the process and highlight lessons learned in application of the concepts.
Leadership SupportExplanationDouglah (1998) highlights the critical importance of a positive attitude towards evaluation from people involved in the process. In our case, the Executive Director and the Chair of the MAC Board initiated the evaluation, motivated, at least in part, by membership demands for Board accountability. Despite the request for assistance and interest by the MAC Board in evaluating its programs, there remained some doubts about self-assessment. The proactive participation of the MAC Executive Director and the Chair of MAC was vital to convince board members of the usefulness of evaluating past performance. Evaluation does not aim to replace decision makers' experience and judgment but rather offers systematic evidence that can inform further experience and judgment (Alkin, 1990). It was necessary to spend considerable time with the board discussing methodology and the pros and cons of anticipated outputs. It is important to make sure individuals interested in the study results understand that findings will not be a panacea for all possible problems they are facing. Evaluation will serve to guide future decisions by helping identify issues of relevant and current significance. A clear definition of the evaluation objective is essential in order to identify information needed, as well as to define which instruments will be used (Taylor-Powell, Steele, & Douglah, 1996). For example, in our case the initial purpose of the evaluation process was to reveal member perceptions about past performance and solicit opinions about future directions for MAC programs in order to provide input for subsequent Board decisions. A second, but not necessarily less important, purpose of the MAC Board was articulated during early discussions. The leadership was very interested in demonstrating to their members that individual opinions and beliefs were a valued input to organizational decision-making. Thus, the act of undertaking evaluation was itself an integral part of achieving success. Reviewing goals is critical to ensure that everybody has a common understanding of each programmatic area and terminology used. For example, while reviewing goals of existing programs (Advertising, Merchandising, Export, Public Relations, Industry Services, and Research), some board members found that even they were unclear about differences between Advertising and Merchandising Programs. The names could be confused, while the programs themselves had very different goals. Thus a very early success of the evaluation process was identification of an immediate opportunity to clarify program area definition and goals among respondents. Consistency in understanding insures a greater degree of accuracy when assessing the programs and thus guarantees more reliable results. Having leadership that is highly motivated, committed, and influential acting as a catalyst is important to ensure success of an evaluation. Active participation of the MAC Board and leadership was key, both to ensure political support for the initiative and to obtain important survey design input to ensure results addressed the organization's concerns. Lessons Learned
Evaluation InstrumentExplanationThe literature distinguishes two primary purposes for evaluation: "summative" and "formative" (Scriven, 1967). A summative evaluation serves to document and/or quantify total effectiveness of a program. Such documentation most often takes place once activities of the program have been completed (Douglah, 1987). In comparison, formative evaluation is designed to assess an on-going program with the goal of improving current efforts. Scriven (1991) recommends that the design of any formative process and instrument be broad enough to serve as the basis for future summative evaluation at a project's end. Our assessment of MAC provided formative input, but was also designed for later summative use. Design of an evaluation instrument depends on assessing project scope subject to resource availability (money and time) (Israel, 1992). Taylor-Powell and collaborators (1996) elaborate on the importance of understanding the social, cultural, and political environment and provide additional considerations for instrument selection. Factors that may influence this decision can be classified into three groups: 1) technical adequacy (e.g., reliability, validity, freedom from bias) 2) practicality (e.g., cost, political consequences, duration, personnel needs) and 3) ethics (e.g., protection of human rights, privacy, legality) (Summerhill & Taylor, 1992). Scope of the project influences sample size. The MAC made its mailing list available, which included the entire population (1,123 growers). We learned that the list was out-of-date (e.g., some growers listed had already left the business and others had an incorrect address), which would undermine any attempt to select a probabilistic sample. Thus, guided more by limitations than by choice, the entire population was included in our survey. Such a sampling procedure has been referred to as a "convenience sample" in the literature (Patton, 1990). This non-probabilistic procedure where the individual is self-selected (e.g., respondents) requires careful inference of survey outcomes to the entire program (Cochran, 1963). Since the MAC evaluation involved the direct participation of growers, ethical considerations were important. Primary concerns were voluntary participation, provision of sufficient information about the reach of the study, and ensuring confidentiality of the respondents (Rosey, 1992). Available resources influence instrument selection. A mail survey offers the advantage of requiring minimum staff to prepare and mail, and low overall cost (Diem, 1999). A resource-constrained environment limits resources for the evaluation, thus factor 2 (practicality) drove our choice of instrument after technical adequacy and ethics were incorporated. In our case, information was ultimately gathered through a written mail survey that was divided into three parts. One section solicited demographic information related to the respondents and their operation, including trends in production and markets. A second part of the survey asked the respondents to evaluate six programs supported by MAC. An additional section was designed to explore apple grower beliefs about resource allocation. The survey was administered following Dillman's (2000) methodology, which is framed on social exchange theory (Homans, 1973). Such a process, also called Total Design Method (TDM), views response between respondent and evaluator as a social exchange, where voluntary participation is a function of the benefit-cost ratio of taking part in the study. To encourage participation, it is necessary to either reduce costs (e.g., succinct surveys, pre-paid envelops) or increase benefits (e.g., feedbacks, incentives). In our case, encouragement from the MAC Board was an additional way to increase benefits. Steps followed from Dillman's TDM in the implementation of our survey were: a personalized cover letter, a simple and straightforward survey, and a follow-up mailing. To improve the response rates, two reminder letters were sent; one of them included a new copy of the survey along with another postage-paid reply envelope. These techniques have proven very effective in improving rate of response in earlier applications (Brennan, 1992). Of the total number of surveys mailed, 282 were returned (25% response). Seventeen percent of the 282 respondents indicated that they were no longer growing apples, confirming an observed trend in this sector. This response rate provided enough data to analyze general trends and is considered adequate given the scope of the evaluation. Lessons Learned
Results Analysis and DeliveryExplanationFollowing needs expressed by the MAC Board, a first step in analysis of survey results was to define demographic variables that could be used to sort and compare responses. The objective was to put growers with common characteristics into the same group, so that they could be contrasted with other groups (e.g., is there any differences in the responses of growers that have less than 30 acres of apples compared to those who have more than 100 acres?). In the MAC survey, five variables were used to group growers: geographic region (defined by MAC), scale of production, target market (fresh or processed), grower age, and how members graded the MAC programs using an overall evaluation index. Members were asked to evaluate past performance in each of six MAC program areas on a scale from 1 (poor) to 5 (excellent). An overall evaluation index (OEI) was calculated as the simple average of assessment scores across areas. Two types of questions were included to evaluate grower perceptions of budget allocation. First, members were asked how they would allocate, in percentage terms, future economic resources among five of the six programmatic areas. (The research area was excluded from this question, as it has historically been administered through a separate budgetary process.) A second question was similar, but it asked how members believed MAC had allocated past resources. An important, and unexpected, result was the extremely low capacity of members to identify or even estimate past allocations. This outcome clearly demonstrated a vital gap in information and understanding among growers and identified an important educational opportunity for board members to consider in the future. Several significant differences were found that provided important information to industry leadership, as illustrated by the following example. Growers were asked how they would allocate economic resources for support of fresh versus processed markets. On average, growers who target the fresh market believe that for each dollar allocated to processed markets, $1.70 should be destined to fresh markets. Not surprisingly, growers who target processed markets would allocate the resources between the two categories in a ratio of almost 1 to 1. Further information was elicited through a series of comparisons among member categories. Growers who target fresh markets farm 80% more acres on average than those who target processed markets. Cross-tabs showed the West Central region is highly targeted to the processed market compared to the fresh market focus of Southwest Michigan. At first glance, numeric differences among all groups vary in absolute terms; however, given that these are average responses for certain variables (e.g., acreage), a closer comparison must be made. Armstrong and Overton (1997) present a series of alternatives to estimate non-response bias in mail surveys that may be considered if the evaluation makes inferences to the whole population. The number of growers in each group and the heterogeneity of the responses within the group will determine confidence intervals. A mean comparison test (Tukey, t-test or paired sample test) was used to test for statistical significance. Findings were reported as statistically significant if the result obtained would have occurred by chance no more than 5 times out of 100. After all responses were processed and analyzed, a comprehensive report was prepared that incorporated an extensive degree of detail and disaggregated information (e.g., by age, size of farm, target market, geographic region, etc.). This comprehensive report was distributed prior at a meeting with the MAC Board, where the main findings were then discussed, as well as a careful explanation of how to interpret the figures in the report. A summary of survey highlights was widely distributed to the general membership and other interested parties. Targeted reports were compiled and distributed for some particular groups such as the MAC research committee (Aguilar & Thornsbury, 2003a; 2003b). Numerous follow-up presentations were made with specific groups such as the research committee or to the board as additional questions arose. The MAC Board and membership have already begun to use the information collected in their decision-making. One example relates to the administration of research funding. In the spring 2004 grower newsletter, the Michigan Apple Committee Chair announced a slight shift in direction, moving from what had been a complete production-related research focus to one that includes value-added apple projects as well as consumer and market research. The newsletter cited the survey as evidence that the membership would support such a shift. Lessons Learned
ConclusionsExtension programs have always faced the challenge of delivering reliable information as a valued input to the decision-making process of clientele groups. This role continues to expand, a notable function being the need to help clientele groups conduct their own internal program assessments. This article draws on a process used successfully at Michigan State University to evaluate member perceptions of program effectiveness for a commodity organization where the program beneficiaries are also the funding source and to link a practical evaluation example with theory. Based on our experience, key factors that should be considered include the following.
ReferencesAguilar, C., & Thornsbury, S. (2003a). Michigan apple committee winter 2003 grower survey: Summary of results. Agricultural Economics Report 618, Michigan State University. Aguilar, C., & Thornsbury, S. (2003b). Michigan apple committee winter 2003 grower survey: Research programs. Agricultural Economics Report 619, Michigan State University. Alkin, M. C. (1990). Debates on evaluation. Thousand Oaks, CA: Sage. Armstrong, J. S., & Overton, T. S. (1977). Estimating nonresponse bias in mail surveys. Journal of Marketing Research, 14, 396-402. Brennan, M. (1992). Techniques for improving mail survey response rates. Marketing Bulletin, 3, 24-37, Article 4. Cochran, W. G. (1963). Sampling techniques. Second Edition. Wiley and Sons, New York. Decker, D. J., & Yerka, B. L. (1990). Organizational philosophy for program evaluation. Journal of Extension [On-line], 28(2). Available at: http://www.joe.org/joe/1990summer/f1.html Diem, K. (1999). Choosing appropriate research methods to evaluate educational programs. Rutgers Cooperative Extension Fact Sheet #FS943. New Brunswick, NJ Dillman, D. (2000). Mail and Internet surveys: The tailored design method. Second Edition. New York: John Wiley & Sons. Douglah, M. (1998). Developing a concept of Extension program evaluation. Program Development and Evaluation, Series G3658-7. University of Wisconsin-Extension. Homans, G. C. (1974). Social behavior. New York: Harcourt, Brace, Jovanovich. Israel, G. D. (1992). Sampling the evidence of Extension program impact. Fact Sheet PEOD-5, Institute of Food and Agricultural Sciences, University of Florida. O'Neill, B. (1998). Money talks: Documenting the economic impact of extension personal finance programs. Journal of Extension [On-line], 36(5). Available at: http://www.joe.org/joe/1998october/a2.html Patton, M. Q. (1990). Qualitative evaluation and research methods. Newbury Park, CA: Sage Publications. Rosey, D. (1992). Program evaluation. Chicago: Nelson-Hall Publisher. Scriven, M. (1967). The methodology of evaluation. In Stake R.E. (Ed.), Curriculum evaluation. American Educational Research Association Monograph Series on Evaluation, No. 1. Chicago: Rand McNally. Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage Publications. Stupp, R. (2003). Program evaluation: Use it to demonstrate value to potential clients. Journal of Extension [On-line], 41(4). Available at: http://www.joe.org/joe/2003august/comm1.shtml Summerhill, W. R., & Taylor, C. L. (1992). Selecting a data collection technique. Circular PE-21, Program Evaluation and Organizational Development, Florida Cooperative Extension Service. Taylor-Powell, E., Steele, S., & Douglah, M. (1996). Planning a program evaluation. Program Development and Evaluation, Series G3658-1. University of Wisconsin-Extension.
A Multipurpose Evaluation Strategy for Master Gardener Training ProgramsEmelie Swackhamer Nancy Ellen Kiernan Introduction and ProblemMaster Gardener volunteers greatly extend the educational reach of Extension staff. Interest in developing and maintaining effective Master Gardener programs continues across the country (Bobbitt, 1997; Finch, 1997; Mechling & Schumacher, 2001; Ruppert, Bradshaw, & Stewart, 1997). Training volunteers requires time and money (Meyer & Hanchek, 1997; Ruppert, Bradshaw, & Stewart, 1997). To get a good return on this investment, Extension should engage trained Master Gardeners in many volunteer activities for many years. Retention of Master Gardener volunteers seems to increase as their level of experience and familiarity with Cooperative Extension programs increases (Ruppert, Bradshaw, & Stewart, 1997). A Master Gardener who is comfortable and confident doing the volunteer work that is needed is more likely to remain active in the program and gain experience and familiarity. Thus, training for Master Gardeners should increase their knowledge on a variety of topics. The training must also impart the confidence needed to field home gardening questions from the public. State level reports of Master Gardener programs tend to include anecdotal impressions of the benefits of the volunteer experience, the number of hours of volunteer time contributed, and the contacts made (Boyer, Waliczek, & Sajick, 2002; Kirsch & VanDerZanden, 2002; Mechling & Schumacher, 2001; Schrock, Meyer, Ascher, & Snyder, 2002). These data are not useful for program improvement because they do not identify the source of strengths and weaknesses of the program. Some Master Gardener training programs have used opinion surveys to provide qualitative data on how the process is perceived and pre- and post-tests to measure knowledge change (Jeanette & Meyer, 2002; Ruppert, Bradshaw, & Stewart, 1997; Stack, 1997; VanDerZanden, 2001; VanDerZanden & Hilgert, 2002; VanDerZanden, Rost, & Eckel, 2002). These measures, however, do not provide quantitative data on the important issue of how confident the volunteers are in applying what they have learned through their training. Another difficulty in evaluating Master Gardener training occurs because in most states, Master Gardener programs are county-based, resulting in a great deal of variation in how the training is conducted and how the program is administered. This mosaic of programs often leads to local approaches to evaluation, where one or a few counties develop their own evaluation and gather data unique to their situation (Finch, 1997; Ruppert, Bradshaw, & Stewart, 1997; Warmund & Schrock, 1999). The information gathered from these evaluations is useful for program improvement and accountability on a local level, but due to variations in evaluations among counties, the information is not useful for showing success of the training across a wider region or at the state level. There has been little emphasis on trying to capture an expanded view of the impact that Extension has on Master Gardener trainees, but there is great potential to show success. Developing local evaluations also results in duplication of effort by county program coordinators across the state. Often, the staff who design these evaluations have limited experience in evaluation, which can result in low quality designs. Penn State requires each new Master Gardener to attend training on eight core topics. The overall success of the core training could be documented by compiling uniform evaluation data. Program DescriptionThe training classes for new Master Gardeners in the Southeast and Capital Regions of Pennsylvania are organized as a circuit. This includes 16 counties and more than half of the Master Gardeners in the state. Generally, one instructor teaches the same topic at many training sites. This circuit approach is efficient because it reduces duplication of effort on curriculum development and allows instructors to concentrate on improving the one topic they teach. It also allows for consistency of content of classes across counties and for evaluation data to be summarized from all sites. In 2002, there were eight sites on the training circuit. The 12 topics taught included: soils, botany, turf, plant propagation, plant disease, entomology, plant identification, vegetables, herbaceous plants, integrated pest management and pesticides (IPM), ornamentals, and diagnosing plant problems. Recognizing the need for quantitative impact data that could be used for multiple purposes, the authors developed a uniform evaluation strategy. To achieve this, they considered how to evaluate, what to evaluate, and how to summarize the data for different uses. This article describes how to use this evaluation to quantify increase in learning and confidence. The evaluation takes into consideration the fact that the volunteers come to the program with different levels of confidence. The article demonstrates how the same data can be summarized in different ways to use for program improvement or accountability reports. The evaluation for the training classes that is described in this article is the first component of a three-part evaluation plan for the Master Gardener Program in Pennsylvania (Swackhamer & Kiernan, 2002 <http://www.extension.psu.edu/evaluation/pdf-others/MGTrainingINTRO.pdf>). The second component of this plan is a post-training class evaluation, which measures intentions to use new gardening practices and knowledge about the Master Gardener program. The third component is an evaluation for Master Gardeners who have been active in the program for one or more years, and it measures the long-term impact of the training and their involvement in the program combined. Evaluation of the Training ClassesHOW to Evaluate the ProgramThe training evaluation survey has several important features. First, formatted as a tri-fold brochure, the layout is aesthetically pleasing and has a professional appearance. The professionalism of the evaluation survey is important to encourage participants to take it seriously and to faithfully fill it out after each class (Dillman, 2000). It also imparts a general image of professionalism and organization to Extension. Second, because the survey is handed out at the beginning of the 12-class training period, the evaluation can be completed over time. This allows participants to evaluate each class when they attend, whenever it is scheduled in their county. Participants respond immediately after each class takes place, which gives a more valid assessment than would be obtained if participants were asked to recall their impressions of individual classes at the end of the 12-week session. Third, the questions in this evaluation collect data that are useful for many purposes. The questions provide insight into the effectiveness of the instructor for each topic, allowing for program improvement. The questions also provide data to examine the success of the training in one or more counties, or even the whole state. WHAT to EvaluateIn the first question the trainees indicate how much they learned in each class, i.e., "nothing new," "some new knowledge," "a lot," or "a great deal" (Figure 1). The structure of this question allows instructors to see how effectively they taught their topic at each site, at all sites, and in comparison to other instructors. Figure 1.
In the second question the trainees indicate how confident they are in fielding questions on each topic on a five-point scale before attending the class, and again after attending the class. The answer categories are "not too confident," "somewhat confident," "moderately confident," "very confident," or "extremely confident" (Figure 2). The structure of this question allows the instructor to calculate a quantitative change in confidence imparted by their teaching. Demonstrating a change using before and after measures contributes to the validity of the data and the educators' documentation of their effectiveness (Rossi, 1982). Figure 2.
 The structure of this question allows a second benefit, comparison of trainees who come into the program with different levels of prior experience and confidence. In the third question (not shown) trainees provide comments about the class in each topic. The question provides a forum for qualitative feedback, useful for program improvement. HOW to Summarize the DataThe data from each question can be summarized from two perspectives. Each type of summary serves a different evaluation purpose for Extension. Summarizing topic-by-topic gives information useful for program improvement. Summarizing across all topics gives information more useful for accountability, such as reporting to stakeholders and funding sources. Once the data are entered, both summaries can be accomplished using Excel or other similar programs that are basic software on most computers. ResultsIn the fall of 2002, 178 new Master Gardeners from 12 counties participated in 12 classes of training and contributed to the evaluation of each class they attended. The counties included metropolitan, suburban, and rural areas. Question One: Measure of LearningTopic-by-Topic ResultsThe percent of Master Gardeners who learned "a lot" or "a great deal" about 10 of the 12 topics was substantial (Figure 3). Those topics include: soils, botany, turf, propagation, plant disease, entomology, plant identification, IPM, ornamentals, and diagnosing plant problems. The percent of Master Gardeners who learned "a lot" or "a great deal" in two topics, vegetables and herbaceous plants, was somewhat less. Figure 3.
Across Topics ResultsSummarizing the data not for each topic as above, but for individuals across the topics, the results indicate that the majority of the Master Gardeners (77%) learned a "lot" or "a great deal" about most topics (Figure 4). The remainder of the Master Gardeners (23%) had a similar gain of knowledge in up to six topics. Figure 4.
Question Two: Measure of Confidence in Ability to Field QuestionsTopic-by-Topic ResultsThe results indicate that the Master Gardeners increased their confidence in their ability to field questions to "moderately," "very," or "extremely" confident from before the training to after it in all twelve topics (Figure 5). A greater percent of Master Gardeners increased in confidence in fielding questions in soils, botany, turf, propagation, plant disease, entomology, plant identification, IPM, ornamentals, and diagnosing plant problems. A lesser percent of Master Gardeners increased their confidence in vegetables and herbaceous plants. Figure 5.
Tabulating the number of levels for which the trainees increased their confidence on the five-point confidence scale, the results indicate that in plant pathology, for example, 41% increased two or more levels and 52% increased by one level (Figure 6). A similar analysis is possible for each topic. Figure 6.
Across Topics ResultsSummarizing the data not for each topic as above, but for individuals across topics, the results indicate 89% of the Master Gardeners increased their confidence in their ability to field questions in seven or more topics (Figure 7). The rest increased their confidence in their ability to field questions in up to six topics. Figure 7.
Implications of WHAT Is AskedQuestion One: Measure of Learning
Question Two: Confidence in Ability to Field Questions
Implications of HOW Results Are SummarizedTopic-by Topic-Results
Across Topics Results
DiscussionEvaluation of training programs for Master Gardeners should take into account the county-based nature of these programs and the different experience levels of the volunteers. Evaluation efforts should also lend themselves to multiple uses because, while instructors need information to improve programs, stakeholders require information to see the value of their investment. Information obtained by this evaluation also gives coordinators a unique opportunity to consider the trainees' perception of their confidence, which can influence the coordinators' goal of long-term retention of volunteers. These results of this study suggest that the training in Pennsylvania was very successful. Trainees learned a substantial amount about most topics, and they gained confidence in applying what they learned in all topics by attending the classes. Program coordinators can use information from both questions to decide if the classes with less impact are worth the expense of resources and staff time to continue offering in the future. Therefore, the value of each of these pieces of information to a program |
