The Journal of Extension - www.joe.org

June 2017 // Volume 55 // Number 3 // Ideas at Work // 3IAW3

A Framework for the Evaluation of Large-Scale Regional Conservation and Management Strategies

Abstract
The success of complex, regional strategies depends on the ability to evaluate the process of implementation as well as progress toward achieving intended outcomes. The Southeast Cohesive Wildland Fire Management Strategy and the Comprehensive Strategy for Prescribed Fire to Restore Longleaf Pine in the Southeast United States are examples of regional strategies that require extensive efforts across a large landscape. This article presents a framework developed by North Carolina State University Extension Forestry for evaluating these types of regional strategies so that informed recommendations for resource allocation, new program development, and strategic collaboration can be made.


John M. Diaz
Assistant Professor and Extension Specialist
Department of Agricultural Education and Communication
University of Florida
Gainesville, Florida
john.diaz@ufl.edu

Jennifer Evans Fawcett
Extension Associate
Department of Forestry and Environmental Resources
North Carolina State University
Raleigh, North Carolina
jlevans3@ncsu.edu

K. S. U. Jayaratne
State Leader for Program Evaluation and Associate Professor
Department of Agricultural and Extension Education
North Carolina State University
Raleigh, North Carolina
jay_jayaratne@ncsu.edu

Jackie Bruce
Associate Professor
Department of Agricultural and Extension Education
North Carolina State University
Raleigh, North Carolina
jackie_bruce@ncsu.edu

Regional approaches to conservation and natural resources management are becoming commonplace throughout the United States (McKinney, Scarlett, & Kemis, 2010). Strategies are being developed and implemented across large landscapes, requiring action from multiple stakeholder groups. For example, widespread efforts are being made to increase prescribed fire use in the Southeast to support restoration and enhancement of fire-dependent ecosystems, reduce wildfire risk, and promote active land management (Burke, Steelman, & Gharis, 2012).

In this article, we describe a framework for evaluating such large-scale strategies. This framework was used to evaluate prescribed fire efforts across the Southeast in relation to the goals and priorities outlined in two strategic plans: the Southeast Cohesive Wildland Fire Management Strategy (Cohesive Strategy) (Southeast Regional Cohesive Strategy Committee, 2012) and the Comprehensive Strategy for Prescribed Fire to Restore Longleaf Pine in the Southeast United States: A Vision for 2025 (Comprehensive Prescribed Fire Strategy) (Burke et al., 2012). This approach is informing both policy and practice, ensuring that the goals and priorities of each strategy are achieved.

Step-by-Step Process for Evaluating Regional Strategies and Identifying Performance Gaps

North Carolina State University (NCSU) Extension Forestry developed and used the eight-step framework presented here for evaluating both aforementioned fire strategies. The framework can be applied during any phase of regional strategy implementation and may be adapted as needed.

1. Form an Advisory Committee and Establish a Collaborative Network

Select key stakeholders to participate on an advisory committee. This committee's role is to regulate bias, assist with data collection efforts, and provide guidance based on expertise (Berg, 2004). Include diverse interests and stakeholder groups to promote robust evaluation recommendations (Cole, 1980). Also, develop a network of collaborators to provide support in reviewing data quality and ensuring that recommendations are substantiated. Figure 1 lists the agencies and organizations included in the evaluation of the prescribed fire strategies.

Figure 1.
Prescribed Fire Evaluation Advisory Committee and Collaborators

Advisory Committee Members Illustrazione vettoriale silhouette di testa delle persone royalty-free stock vector art U.S. Department of Defense, National Park Service, North Carolina State University, Southern Group of State Foresters, Southeast Regional Extension Forestry, U.S. Forest Service, Natural Resources Conservation Service, U.S. Environmental Protection Agency, U.S. Fish and Wildlife Service, U.S. Geological Survey
Collaborators Illustrazione vettoriale silhouette di testa delle persone royalty-free stock vector art Southeast Regional Cohesive Strategy Committee, Southeast Regional Partnership for Planning and Sustainability, Southern Fire Exchange, Southern State Fire Chiefs

2. Select an Appropriate Frame for the Evaluation Effort

Determining an evaluation frame is paramount when planning assessments of complex regional strategies. In developing an evaluation frame, consider applicable components of the strategy for evaluation as well as complementary strategies that exhibit similar priorities and goals.

3. Develop an Evaluation Worksheet

An evaluation worksheet helps individuals organize an evaluation and connect results to the implementation of recommendations (Chhetri, 2016). The worksheet should be based on the function of the evaluation performed. Figure 2 shows the structure of such an evaluation worksheet.

Figure 2.
Sample Evaluation Worksheet Structure

Desired State Current State Gap? Description of Gap Factors Responsible Recommendation to Bridge Gap
           
         
           

4. Identify Existing Efforts Through an Environmental Scan

An environmental scan allows for the identification of current efforts across a region. Data collection involves online research and input from the advisory committee and collaborative network to ensure increased data validity (Guion, 2010). To organize data, use a spreadsheet that includes a description of the effort, classification of functionality, designation of a lead agency or organization, identification of collaborators, and relevant contact information.

5. Tie the Environmental Scan to an Associated Strategy

Connect the data collected during the environmental scan to the associated strategy or strategies for evaluation by assigning each effort a code that identifies the corresponding strategy and action item(s) (Chhetri, 2016). This approach allows for quantitative analysis of the current state of efforts in relation to goals, priorities, and actions of the strategy of interest. Figure 3 provides an example of a coded item from the environmental scan for the prescribed fire strategies evaluation.

Figure 3.
Example Coding for Environmental Scan

Cohesive Strategy Code Comprehensive Prescribed Fire Strategy Code Program/Area Description
7.A.2 R.1.5 Longleaf Stewardship The project will support a Conservation Delivery Specialist to work directly with local Natural Resources Conservation Service staff and improve delivery of Farm Bill cost-share programs as well as a full-time coordinator to work with partners and private landowners and enhance participation in programs focused on longleaf restoration on private lands.

6. Interview Key Informants to Qualify On-the-Ground Experiences

Follow the environmental scan with key informant interviews to gain insight into on-the-ground experiences from stakeholders in the field. Extension can leverage its connections and those of the advisory committee to create an initial sampling frame of informants working in priority areas. To achieve a rich data set, use snowball sampling (Berg, 2004) for identifying additional informants and, thereby, gaining new insights and perspectives.

7. Analyze Results and Integrate Findings into the Evaluation Worksheet

To analyze the collected key informant interview data, NCSU Extension Forestry used content analysis (Figure 4) identifying major themes from the data that were also triangulated with the results of the environmental scan. Using this approach increases the credibility of assertions and recommendations coming from an evaluation (Berg, 2004).

Figure 4.
The Ten Steps of Content Analysis

  1. Copy and read through the transcript, making brief notes in the margin when you find interesting or relevant information.
  2. Reread the notes in the margins, and list the different types of information there.
  3. Read through the list, and categorize each item in a way that offers a description of what it is about.
  4. Identify whether the categories can be linked, and list them as major categories (or themes) and/or minor categories (or themes).
  5. Compare and contrast the various major and minor categories.
  6. If there is more than one transcript, repeat the first five steps for each transcript.
  7. When you have finished with all the transcripts, collect all the categories or themes and examine each to consider whether it fits and its relevance.
  8. Once all the transcript data are categorized into major and minor categories/themes, review to ensure that the information is categorized as it should be.
  9. Review all the categories to ascertain whether some categories can be merged or some need to be subcategorized.
  10. Return to the original transcripts, and ensure that all the information that needs to be categorized has been.

Adapted from "The Ten Steps of Content Analysis," by University of Surrey, 2016, Module 9: Introduction to Research. Retrieved from http://libweb.surrey.ac.uk/library/skills/Introduction%20to%20Research%20and%20Managing%20Information%20Leicester/page_74.htm

8. Facilitate Discussion to Develop Consensus on Gaps and Contributing Factors

Leverage the expertise of the advisory committee through facilitated discussion to develop consensus of the evaluation results. This endeavor allows for multiple interpretations of the data and more robust recommendations based on the breadth of experience and expertise leveraged by the consensus-driven process.

Challenges and Lessons Learned

One challenge to this approach is ensuring effective participation among advisory committee members. This issue is common in collaborative program evaluation and requires effort in familiarizing the committee with the process (Cole, 1980; Layman, Doll, & Peter, 2013). Committee members have limited time to devote to complex evaluation efforts, so it is important to quickly develop the knowledge and skills necessary to facilitate effective involvement. Focusing on their specific role(s) may prevent advisory committee members from feeling overwhelmed by the extent of the process (Cole, 1980).

Implications

The ability to achieve large-landscape conservation guided by regional strategies is closely linked to the capability of those in charge of oversight to stay in tune with its implementation and overall impact (Prokopy et al., 2009; Wondolleck & Yaffee, 2000). For promoting a strategy that is in tune with the on-the-ground reality, it is important to develop a mechanism that allows for adaptive management (Layman et al., 2013; Muñoz-Erickson, Aguilar-González, Loeser, & Sisk, 2010). Using a practical approach for evaluating complex strategies is critical in gaining a holistic view of a region's progress toward specific conservation goals.

Inclusive strategies involving multiple key stakeholders allow evaluations to result in robust recommendations that have the potential to create meaningful change (Layman et al., 2013). NCSU Extension Forestry demonstrated this by developing a framework for promoting a collaborative approach by multiple entities in identifying programmatic and resource gaps across a region to inform the potential development of a collaborative prescribed fire initiative. Others may be able to use the framework for similar endeavors.

References

Berg, B. L. (2004). Qualitative research methods (5th ed.). Boston, MA: Pearson Education, Inc.

Burke, C., Steelman, T., & Gharis, L. (2012). Comprehensive strategy for prescribed fire to restore longleaf pine in the southeast United States: A vision for 2025. Retrieved from https://serppas.org/media/1939/comprehensive-strategy-for-prescribed-fire.pdf

Chhetri, S. (2016). How to perform gap analysis and create gap documents. Retrieved from http://professional-businessanalysis.blogspot.com/2012/03/how-to-perform-gap-analysis-and-create.html

Cole, J. (1980). Developing effective advisory councils. Journal of Extension, 18(4). Available at: http://www.joe.org/joe/1980july/80-4-a1.pdf

Guion, L. (2010). A 10-step process for environmental scanning. Journal of Extension, 48(4), Article 4IAW2. Available at: http://www.joe.org/joe/2010august/iw2.php

Layman, C., Doll, J., & Peter, C. (2013). Using stakeholder needs assessments and deliberative dialogue to inform climate change outreach efforts. Journal of Extension, 51(3), Article 3FEA3. Available at: http://www.joe.org/joe/2013june/a3.php

McKinney, M., Scarlett, L., & Kemis, D. (2010). Large landscape conservation: A strategic framework for policy and action. Cambridge, MA: Lincoln Institute of Land Policy.

Muñoz-Erickson, T. A., Aguilar-González, B., Loeser, M. R. R., & Sisk, T. D. (2010). A framework to evaluate ecological and social outcomes of collaborative management: Lessons from implementation with a northern Arizona collaborative group. Environmental Management, 45, 132–144. doi:10.1007/s00267-009-9400-y

Prokopy, L. S., Genskow, K., Asher, J., Baumgart-Getz, A., Bonnell, J. E., Broussard, S., . . . Wood, D. (2009). Designing a regional system of social indicators to evaluate nonpoint source water projects. Journal of Extension, 47(2), Article 2FEA1. Available at: http://www.joe.org/joe/2009april/a1.php

Southeast Regional Cohesive Strategy Committee. (2012). The national cohesive wildland fire management strategy: Phase III. Retrieved from https://www.forestsandrangelands.gov/strategy/documents/rsc/southeast/SERAP_Final20130405.pdf

University of Surrey. (2016). Module 9 : Introduction to Research. Retrieved from http://libweb.surrey.ac.uk/library/skills/Introduction%20to%20Research%20and%20Managing%20Information%20Leicester/page_74.htm

Wondolleck, J. M., & Yaffee, S. L. (2000). Making collaboration work: Lessons from innovation in natural resource management. Washington, DC: Island Press.