December 2020 // Volume 58 // Number 6 // Tools of the Trade // v58-6tt5
Assessing Healthful Eating and Physical Activity Practices in Places Children Learn
Site-level assessment questionnaires (SLAQs) were developed to assess nutrition and physical activity practices and environments in schools and other places children spend time in order to facilitate program planning and evaluation. After expert panel review for content validity, questionnaires were feasibility tested by users in six schools, three early care and education programs, and two out-of-school programs. Findings indicate that the questionnaires are feasible and useful for planning interventions. Extension programs can use SLAQs to support policy, systems, and environment change efforts that promote healthful eating and physical activity in children and to measure intervention effectiveness.
Schools and other places children spend time are important targets for health promotion (Centers for Disease Control and Prevention, 2009; Institute of Medicine, 2012; Story et al., 2009; U.S. Department of Agriculture [USDA], 2016) and among the most common settings for Supplemental Nutrition Assistance Program-Education (SNAP-Ed) interventions. Cooperative Extension programs partner with schools, out-of-school programs, and early care and education (ECE) programs to improve child health through SNAP-Ed policy, systems, and environment (PSE) interventions (Bergman et al., 2018; Lee et al., 2019; Schroeder et al., 2018).
Sensitive, easy-to-use tools that measure institutional practices related to healthful eating and physical activity are critical for identifying areas for improvement and measuring progress (Cates et al., 2014; Naja-Riese et al., 2019). Herein we describe the development and content validity and feasibility testing of questionnaires for initial assessment, annual reassessment, and evaluation of health practices in these three settings.
Site-Level Assessment Questionnaires
We developed site-level assessment questionnaires (SLAQs) as program planning and evaluation instruments for SNAP-Ed programs in elementary and secondary schools plus out-of-school and ECE programs. SLAQs assess institutional healthful eating and physical activity practices for each setting and were designed to be
- concise enough that sites could complete them annually,
- comprehensive regarding healthful eating and physical activity practices,
- demonstrative of face validity,
- reliable between respondents and time points, and
- sensitive enough to detect meaningful differences between sites and change over time within a site.
Items include a practice statement followed by a Likert-scale series of response options indicating degree of implementation; some items use yes/no responses. Likert-scale items are scaled 0–4, and yes/no items are coded as "no" = 0 and "yes" = 4; 4 represents a best practice. Questionnaires are completed as fillable Adobe pdfs, programmed with skips to reduce error, with autocalculated section and overall scores that allow respondents to see results immediately upon completion. Figure 1 shows an example portion of an SLAQ.
Portion of Early Care and Education Site-Level Assessment Questionnaire
To develop the SLAQs, we conducted a scan of existing environmental, policy, and practice assessments from the SNAP-Ed Evaluation Framework's recommended evaluation tools (USDA, 2016), the National Collaborative on Childhood Obesity Research Measures Registry (McKinnon et al., 2012), and other known sources. We adapted items from existing instruments to a standardized statement-and-response format; constructed new items as needed; and aligned items with state and national regulations and recommendations and the SNAP-Ed Evaluation Framework.
We developed instruments in parallel to maintain consistency across settings while keeping instruments true to setting-specific needs. Table 1 shows domains assessed by each instrument. Each questionnaire is organized into sections that we anticipated could be answered by the same staff person at a site to facilitate coordinated data collection among multiple staff members.
|Wellness policies and committee||All||Students' families are active members of the school level wellness committee.|
|Meal/snack program participation||All||School participates in the Fresh Fruit & Vegetable Program.|
|Meals and snacks||All||Students receive at least one fruit or vegetable with each meal.|
|Food environment/feeding practices||All||Students have at least 30 minutes for eating lunch, including time to get food and sit at the table.|
|Nonmeal foods (e.g., competitive foods, fundraisers)||All||Items sold as part of fundraisers are limited to non-food items or foods that meet the Smart Snacks criteria.|
|Physical activity||All||Staff actively facilitate physical activity during recess.|
|Sedentary/screen time||Out-of-school, early care and education||When entertainment screen time is offered, active alternatives are provided.|
|Physical education (PE)||School||At least 50% or more of PE class time is spent in moderate to vigorous physical activity.|
|Nutrition education||All||Nutrition education is taught with sequential curricula consistent with state or national standards.|
|Gardens||All||Produce from garden is used in meals or snacks.|
|Student involvement||School||Students take part in taste tests to determine food preferences for menu items.|
|Parent/family involvement||All||Nutrition education (workshops, activities, and take-home materials) is offered to parents.|
|Breastfeeding support||Early care and education||A welcoming and private space is provided for mothers to breastfeed or express breast milk.|
|Staff training||All||PE teachers complete annual professional development on PE or physical activity.|
|Environment and policy surrounding the school||School||Mobile vendors sell unhealthy food items within visual range of school entrances and within 30 minutes of school start or end times. a|
|a This item is reverse-coded. Most Site-Level Assessment Questionnaire items are phrased as best practices; however, we wrote a small number of items in the opposite direction when doing so was necessary to improve item clarity.|
Topic experts reviewed the questionnaires to assess content validity. Experts included Extension and SNAP-Ed professionals, education professionals with nutrition expertise, and academic researchers. Reviewers considered the goals of the SLAQs while providing feedback regarding missing content, items that would not contribute substantially to assessment, and clarity and logic of the items and response options.
Questionnaires were tested at a convenience sample of sites: three elementary schools, three secondary schools, two out-of-school programs, and three ECE programs. Each site's administrator coordinated self-assessment, involving appropriate staff. Sites completed the SLAQ as a paper-and-pencil form and then answered a survey asking how many staff were needed to complete the SLAQ, how long it took to complete, the usefulness of the SLAQ for their program, and what challenges they faced when completing it. In addition, some testers provided substantial, question-by-question feedback.
Results and Discussion
Expert panel content review yielded feedback categorized as follows:
- language that may be confusing or difficult to interpret;
- need for inclusion of definitions and references, especially for state or federal guidelines;
- response options needing revision;
- items needing a "skip" or "N/A" option;
- unclear intent of items; and
- items that could be removed.
Revisions incorporating expert panel suggestions were made prior to feasibility testing.
Feasibility testing indicated that SLAQs are feasible and useful for program planning. All testers reported that the results would help them promote healthful eating and physical activity at their sites. However, testers expressed concern about questionnaire length and confusion around specific items. Responses and patterns of nonresponse revealed areas for improvement.
Feasibility testing revealed what resources were needed for the self-assessment. The school SLAQ was most resource intensive. Representatives of schools took 1–7 hr (M = 3.6 hr) to complete the questionnaire, with answering of questions divided among three to seven staff members (M = 4.5 staff); more staff were involved at secondary schools (four to seven staff) compared to elementary schools (three or four staff). Representatives of out-of-school programs completed the SLAQ in an average of 2 hr (answering of questions by three staff per site). Representatives of ECE programs completed the SLAQ in 1.5 hr on average (answering of questions by one or two staff per site). Issues encountered during feasibility testing are summarized in Table 2, along with remedies that addressed them. Questionnaire content did not change substantively with this revision.
|Items or associated response options on the school questionnaire were not relevant to specific grade level.||School questionnaire was divided into separate versions for elementary and secondary levels.|
|Testers indicated that wellness policy information is time-consuming to gather and report. Several testers left policy questions blank.||Item-by-item wellness policy question was replaced with one policy question per section.|
|Items or response options were unclear or confusing.||Items and/or response options were revised or deleted.|
|Terminology, such as program names, was unfamiliar or too technical.||Items were revised to simplify language. Where relevant, definitions were added to the questionnaires as endnotes, including web links for more information if available.|
|Items needed a not applicable (N/A) option.||N/A was added, or where relevant, questions were grouped, and a skip pattern was added.|
|Testers did not follow instructions on the questionnaire, including regarding skip patterns.||Questionnaire was reformatted where possible to reduce reliance on instructions. When moving from feasibility testing to the next project phase, respondents were encouraged to use the fillable pdf form, which automates skips.|
|Items contained within a single section were completed by multiple staff.||Items were reorganized into sections according to who completed them.|
|Testers desired a place to write in comments for each section, for additional information and for future reference.||Added a page at the end for comments by section.|
|Items asking about combined program participation were difficult to answer.||Where appropriate, items asking about program participation were separated.|
SLAQs are novel tools that support Extension professionals working with schools, out-of-school programs, and ECE sites to assess needs, create action plans, and evaluate progress, while also offering high-quality, aggregable data for larger scale analysis. Schools that self-assess using SLAQs benefit from collecting data aligned with the Final Rule of the Healthy, Hunger-Free Kids Act of 2010 (Food and Nutrition Service, 2016), which outlines school wellness requirements for local educational agencies and schools participating in national school meal programs. This alignment means that schools using an SLAQ can measure and report how they meet each school wellness requirement of the Final Rule and can identify areas of Final Rule implementation that may be lacking.
A strength of the SLAQs, when compared to existing assessments, is that they were designed with the sensitivity to detect change between sites and across time points. Using aggregated scores from multiple sites, Extension professionals can associate the strength of measured institutional practices with nutrition and physical activity outcome data. When SLAQs are repeated over time, change in scores can be correlated with changes in measured outcomes.
SLAQs are feasible, comprehensive assessments of school, out-of-school, and ECE practices related to healthful eating and physical activity with usefulness for planning and evaluating PSE efforts in these settings.
We thank the many stakeholders and expert reviewers who contributed to the development of the SLAQs. The California Department of Public Health Nutrition Education and Obesity Prevention Branch supported the effort. The project was funded by USDA's Supplemental Nutrition Assistance Program Education. Our institution is an equal opportunity employer.
Correspondence concerning this article should be addressed to Carolyn D. Rider. Email: firstname.lastname@example.org
Bergman, J. J., Linnell, J. D., Scherr, R. E., Ginsburg, D. C., Brian, K. M., Carter, R., Donohue, S., Klisch, S., Lawry-Hall, S., Pressman, J., Soule, K., & Zidenberg-Cherr, S. (2018). Feasibility of implementing a school nutrition intervention that addresses policies, systems, and environment. Journal of Extension, 56(1), Article v56-1a6. https://joe.org/joe/2018february/a6.php
Cates, S., Blitstein, J., Hersey, J., Kosa, K., Flicker, L., Morgan, K., & Bell, L. (2014). Addressing the challenges of conducting effective Supplemental Nutrition Assistance Program Education (SNAP-Ed) evaluations: A step-by-step guide. https://snaped.fns.usda.gov/library/materials/addressing-challenges-conducting-effective-supplemental-nutrition-assistance
Centers for Disease Control and Prevention. (2009). Recommended community strategies and measurements to prevent obesity in the United States: Implementation and measurement guide. Morbidity and Mortality Weekly Report, 58(RR-7), 1–30.
Food and Nutrition Service. (2016). Final rule: Local school wellness policy implementation under the Healthy, Hunger-Free Kids Act of 2010. U.S. Department of Agriculture. https://www.fns.usda.gov/school-meals/fr-072916c
Institute of Medicine. (2012). Accelerating progress in obesity prevention: Solving the weight of the nation. The National Academies Press. https://doi.org/10.17226/13275
Lee, E., Smathers, C., Zubieta, A. C., Ginnetti, S., Shah, A., & Freedman, D. A. (2019). Identifying indicators of readiness and capacity for implementing farm-to-school interventions. Journal of School Health, 89(5), 373–381. https://doi.org/10.1111/josh.12747
McKinnon, R. A., Reedy, J., Berrigan, D., Krebs-Smith, S. M., & NCCOR Catalogue and Registry Working Groups. (2012). The National Collaborative on Childhood Obesity Research Catalogue of Surveillance Systems and Measures Registry: New tools to spur innovation and increase productivity in childhood obesity research. American Journal of Preventive Medicine, 42(4), 433–435. https://doi.org/10.1016/j.amepre.2012.01.004
Naja-Riese, A., Keller, K. J. M., Bruno, P., Foerster, S. B., Puma, J., Whetstone, L., MkNelly, B., Cullinen, K., Jacobs, L., & Sugerman, S. (2019). The SNAP-Ed Evaluation Framework: Demonstrating the impact of a national framework for obesity prevention in low-income populations. Translational Behavioral Medicine, 9(5), 970–979. https://doi.org/10.1093/tbm/ibz115
Schroeder, M., Grannon, K., Shanafelt, A., & Nanney, M. S. (2018). Role of Extension in improving the school food environment. Journal of Extension, 56(7), Article v56-7iw2. https://www.joe.org/joe/2018december/iw2.php
Story, M., Nanney, M. S., & Schwartz, M. B. (2009). Schools and obesity prevention: Creating school environments and policies to promote healthy eating and physical activity. The Milbank Quarterly, 87(1), 71–100. https://doi.org/10.1111/j.1468-0009.2009.00548.x
U.S. Department of Agriculture. (2016). The Supplemental Nutrition Assistance Program Education (SNAP-Ed) evaluation framework: Nutrition, physical activity, and obesity prevention indicators: Interpretive guide to the SNAP-Ed evaluation framework. https://snapedtoolkit.org/framework/index/