June 2017 // Volume 55 // Number 3 // Ideas at Work // 3IAW3
A Framework for the Evaluation of Large-Scale Regional Conservation and Management Strategies
The success of complex, regional strategies depends on the ability to evaluate the process of implementation as well as progress toward achieving intended outcomes. The Southeast Cohesive Wildland Fire Management Strategy and the Comprehensive Strategy for Prescribed Fire to Restore Longleaf Pine in the Southeast United States are examples of regional strategies that require extensive efforts across a large landscape. This article presents a framework developed by North Carolina State University Extension Forestry for evaluating these types of regional strategies so that informed recommendations for resource allocation, new program development, and strategic collaboration can be made.
Regional approaches to conservation and natural resources management are becoming commonplace throughout the United States (McKinney, Scarlett, & Kemis, 2010). Strategies are being developed and implemented across large landscapes, requiring action from multiple stakeholder groups. For example, widespread efforts are being made to increase prescribed fire use in the Southeast to support restoration and enhancement of fire-dependent ecosystems, reduce wildfire risk, and promote active land management (Burke, Steelman, & Gharis, 2012).
In this article, we describe a framework for evaluating such large-scale strategies. This framework was used to evaluate prescribed fire efforts across the Southeast in relation to the goals and priorities outlined in two strategic plans: the Southeast Cohesive Wildland Fire Management Strategy (Cohesive Strategy) (Southeast Regional Cohesive Strategy Committee, 2012) and the Comprehensive Strategy for Prescribed Fire to Restore Longleaf Pine in the Southeast United States: A Vision for 2025 (Comprehensive Prescribed Fire Strategy) (Burke et al., 2012). This approach is informing both policy and practice, ensuring that the goals and priorities of each strategy are achieved.
Step-by-Step Process for Evaluating Regional Strategies and Identifying Performance Gaps
North Carolina State University (NCSU) Extension Forestry developed and used the eight-step framework presented here for evaluating both aforementioned fire strategies. The framework can be applied during any phase of regional strategy implementation and may be adapted as needed.
1. Form an Advisory Committee and Establish a Collaborative Network
Select key stakeholders to participate on an advisory committee. This committee's role is to regulate bias, assist with data collection efforts, and provide guidance based on expertise (Berg, 2004). Include diverse interests and stakeholder groups to promote robust evaluation recommendations (Cole, 1980). Also, develop a network of collaborators to provide support in reviewing data quality and ensuring that recommendations are substantiated. Figure 1 lists the agencies and organizations included in the evaluation of the prescribed fire strategies.
|Advisory Committee Members||U.S. Department of Defense, National Park Service, North Carolina State University, Southern Group of State Foresters, Southeast Regional Extension Forestry, U.S. Forest Service, Natural Resources Conservation Service, U.S. Environmental Protection Agency, U.S. Fish and Wildlife Service, U.S. Geological Survey|
|Collaborators||Southeast Regional Cohesive Strategy Committee, Southeast Regional Partnership for Planning and Sustainability, Southern Fire Exchange, Southern State Fire Chiefs|
2. Select an Appropriate Frame for the Evaluation Effort
Determining an evaluation frame is paramount when planning assessments of complex regional strategies. In developing an evaluation frame, consider applicable components of the strategy for evaluation as well as complementary strategies that exhibit similar priorities and goals.
3. Develop an Evaluation Worksheet
An evaluation worksheet helps individuals organize an evaluation and connect results to the implementation of recommendations (Chhetri, 2016). The worksheet should be based on the function of the evaluation performed. Figure 2 shows the structure of such an evaluation worksheet.
|Desired State||Current State||Gap?||Description of Gap||Factors Responsible||Recommendation to Bridge Gap|
4. Identify Existing Efforts Through an Environmental Scan
An environmental scan allows for the identification of current efforts across a region. Data collection involves online research and input from the advisory committee and collaborative network to ensure increased data validity (Guion, 2010). To organize data, use a spreadsheet that includes a description of the effort, classification of functionality, designation of a lead agency or organization, identification of collaborators, and relevant contact information.
5. Tie the Environmental Scan to an Associated Strategy
Connect the data collected during the environmental scan to the associated strategy or strategies for evaluation by assigning each effort a code that identifies the corresponding strategy and action item(s) (Chhetri, 2016). This approach allows for quantitative analysis of the current state of efforts in relation to goals, priorities, and actions of the strategy of interest. Figure 3 provides an example of a coded item from the environmental scan for the prescribed fire strategies evaluation.
|Cohesive Strategy Code||Comprehensive Prescribed Fire Strategy Code||Program/Area||Description|
|7.A.2||R.1.5||Longleaf Stewardship||The project will support a Conservation Delivery Specialist to work directly with local Natural Resources Conservation Service staff and improve delivery of Farm Bill cost-share programs as well as a full-time coordinator to work with partners and private landowners and enhance participation in programs focused on longleaf restoration on private lands.|
6. Interview Key Informants to Qualify On-the-Ground Experiences
Follow the environmental scan with key informant interviews to gain insight into on-the-ground experiences from stakeholders in the field. Extension can leverage its connections and those of the advisory committee to create an initial sampling frame of informants working in priority areas. To achieve a rich data set, use snowball sampling (Berg, 2004) for identifying additional informants and, thereby, gaining new insights and perspectives.
7. Analyze Results and Integrate Findings into the Evaluation Worksheet
To analyze the collected key informant interview data, NCSU Extension Forestry used content analysis (Figure 4) identifying major themes from the data that were also triangulated with the results of the environmental scan. Using this approach increases the credibility of assertions and recommendations coming from an evaluation (Berg, 2004).
Adapted from "The Ten Steps of Content Analysis," by University of Surrey, 2016, Module 9: Introduction to Research. Retrieved from http://libweb.surrey.ac.uk/library/skills/Introduction%20to%20Research%20and%20Managing%20Information%20Leicester/page_74.htm
8. Facilitate Discussion to Develop Consensus on Gaps and Contributing Factors
Leverage the expertise of the advisory committee through facilitated discussion to develop consensus of the evaluation results. This endeavor allows for multiple interpretations of the data and more robust recommendations based on the breadth of experience and expertise leveraged by the consensus-driven process.
Challenges and Lessons Learned
One challenge to this approach is ensuring effective participation among advisory committee members. This issue is common in collaborative program evaluation and requires effort in familiarizing the committee with the process (Cole, 1980; Layman, Doll, & Peter, 2013). Committee members have limited time to devote to complex evaluation efforts, so it is important to quickly develop the knowledge and skills necessary to facilitate effective involvement. Focusing on their specific role(s) may prevent advisory committee members from feeling overwhelmed by the extent of the process (Cole, 1980).
The ability to achieve large-landscape conservation guided by regional strategies is closely linked to the capability of those in charge of oversight to stay in tune with its implementation and overall impact (Prokopy et al., 2009; Wondolleck & Yaffee, 2000). For promoting a strategy that is in tune with the on-the-ground reality, it is important to develop a mechanism that allows for adaptive management (Layman et al., 2013; Muñoz-Erickson, Aguilar-González, Loeser, & Sisk, 2010). Using a practical approach for evaluating complex strategies is critical in gaining a holistic view of a region's progress toward specific conservation goals.
Inclusive strategies involving multiple key stakeholders allow evaluations to result in robust recommendations that have the potential to create meaningful change (Layman et al., 2013). NCSU Extension Forestry demonstrated this by developing a framework for promoting a collaborative approach by multiple entities in identifying programmatic and resource gaps across a region to inform the potential development of a collaborative prescribed fire initiative. Others may be able to use the framework for similar endeavors.
Berg, B. L. (2004). Qualitative research methods (5th ed.). Boston, MA: Pearson Education, Inc.
Burke, C., Steelman, T., & Gharis, L. (2012). Comprehensive strategy for prescribed fire to restore longleaf pine in the southeast United States: A vision for 2025. Retrieved from https://serppas.org/media/1939/comprehensive-strategy-for-prescribed-fire.pdf
Chhetri, S. (2016). How to perform gap analysis and create gap documents. Retrieved from http://professional-businessanalysis.blogspot.com/2012/03/how-to-perform-gap-analysis-and-create.html
Cole, J. (1980). Developing effective advisory councils. Journal of Extension, 18(4). Available at: http://www.joe.org/joe/1980july/80-4-a1.pdf
Guion, L. (2010). A 10-step process for environmental scanning. Journal of Extension, 48(4), Article 4IAW2. Available at: http://www.joe.org/joe/2010august/iw2.php
Layman, C., Doll, J., & Peter, C. (2013). Using stakeholder needs assessments and deliberative dialogue to inform climate change outreach efforts. Journal of Extension, 51(3), Article 3FEA3. Available at: http://www.joe.org/joe/2013june/a3.php
McKinney, M., Scarlett, L., & Kemis, D. (2010). Large landscape conservation: A strategic framework for policy and action. Cambridge, MA: Lincoln Institute of Land Policy.
Muñoz-Erickson, T. A., Aguilar-González, B., Loeser, M. R. R., & Sisk, T. D. (2010). A framework to evaluate ecological and social outcomes of collaborative management: Lessons from implementation with a northern Arizona collaborative group. Environmental Management, 45, 132–144. doi:10.1007/s00267-009-9400-y
Prokopy, L. S., Genskow, K., Asher, J., Baumgart-Getz, A., Bonnell, J. E., Broussard, S., . . . Wood, D. (2009). Designing a regional system of social indicators to evaluate nonpoint source water projects. Journal of Extension, 47(2), Article 2FEA1. Available at: http://www.joe.org/joe/2009april/a1.php
Southeast Regional Cohesive Strategy Committee. (2012). The national cohesive wildland fire management strategy: Phase III. Retrieved from https://www.forestsandrangelands.gov/strategy/documents/rsc/southeast/SERAP_Final20130405.pdf
University of Surrey. (2016). Module 9 : Introduction to Research. Retrieved from http://libweb.surrey.ac.uk/library/skills/Introduction%20to%20Research%20and%20Managing%20Information%20Leicester/page_74.htm
Wondolleck, J. M., & Yaffee, S. L. (2000). Making collaboration work: Lessons from innovation in natural resource management. Washington, DC: Island Press.