The Journal of Extension - www.joe.org

April 2015 // Volume 53 // Number 2 // Research In Brief // 2RIB3

National 4-H Common Measures: Initial Evaluation from California 4-H

Abstract
Evaluation is a key component to learning about the effectiveness of a program. This article provides descriptive statistics of the newly developed National 4-H Common Measures (science, healthy living, citizenship, and youth development) based on data from 721 California 4-H youth. The measures were evaluated for their reliability and validity of individual items and overall measures using exploratory factor analysis. The measures overall appear to assess what they are intended to assess, but there are several methodological issues, such as cross-loading items and low variance. Recommendations for scale refinement and modifications are made.


Kendra M. Lewis
4-H Evaluation Coordinator
Division of Agriculture and Natural Resources
kmlewis@ucanr.edu
kelew@ucdavis.edu

Shannon J. Horrillo
Associate Director of 4-H Program & Policy
Division of Agriculture and Natural Resources
sjhorrillo@ucanr.edu

Keith Widaman
Distinguished Professor of Psychology
kfwidaman@ucdavis.edu

Steven M. Worker
4-H Science, Engineering, and Technology Coordinator
Division of Agriculture and Natural Resources
smworker@ucanr.edu

Kali Trzesniewski
Associate Specialist in Cooperative Extension, Human Ecology
Associate Director of Research, California 4-H Youth Development Program
ktrz@ucdavis.edu

University of California
Davis, California

Introduction

Evaluation is a key component for documenting the effectiveness of a program (Rennekamp & Arnold, 2009). A crucial element of evaluation is having well-developed, valid, and reliable measures to capture expected program outcomes (Radhakrishna, 2007). Additionally, having a set of common measures can aid comparisons across programs (Payne & McDonald, 2012). For example, comparisons of the effectiveness of 4-H across states can be made if the states are evaluating youth using the same measure. At the request of the National Institute of Food and Agriculture (NIFA), National 4-H Council led a process to develop a set of Common Measures for use across the 4-H system. These common measures assess indicators of scientific literacy, healthy living, citizenship, and youth development (e.g., positive choices, communication). A major goal for developing these measures was to have a standard instrument to assess common indicators across the 4-H Youth Development Program. Indeed, the measures are being used across the country and are being required for grants issued through 4-H National Headquarters and National 4-H Council. For more information about these measures, see http://www.4-h.org/about/youth-development-research/.

A challenge of creating any measure is developing an instrument that is valid and reliable (Van Tilburg Norland, 1990), that is, creating instruments that measure what they are intended to measure (validity) and can consistently do so (reliability; Radhakrishna, 2007). Prior to the widespread use of an instrument, information about the reliability (e.g., Cronbach's alpha, test-retest) and validity (e.g., construct, face) should be made available so that potential users of the instrument can make informed decisions about which measures to use in their evaluations (for examples, see Lackman, Neito, & Gliem, 1997; Stewart, Roberts, & Kim, 2009). To date, there is no psychometric information on the National 4-H Common Measures; our goal is to provide such information to aide in measurement refinement and use.

The study reported here had three objectives:

  1. Provide descriptive statistics from data collected from California (CA) 4-H youth that can be used for comparison with other states and future time points,
  2. Evaluate the measures in terms of reliability and validity, and
  3. Make recommendations to the system and users of the Common Measures.

Methods

Data Collection and Sample

Common Measure data were collected from 721 CA 4-H youth in 2012 through 2014, primarily through surveys embedded in the California 4-H Online Record Book (ORB) system. Implemented in 2011 in all California counties, ORB provides an online alternative to paper 4-H Record Books. Sample demographics are presented in Table 1. The sample is predominately female and non-Hispanic White, and dispersed in terms of residence. The mean age of the sample is 14.13 years.

Table 1.
Sample Demographics (N=721)
N %
Gender
Female 488 67.7
Male 233 32.3
Ethnicity
Non-Hispanic or Latino 629 87.2
Hispanic or Latino 92 12.8
Race
White 612 84.9
Black or African-American 6 0.8
Asian 33 4.6
American Indian or Alaska Native 6 0.8
Native Hawaiian or other Pacific Islander 1 0.1
Undetermined 63 8.8
Residence Type
Farm 192 26.6
Town (non-farm, rural, population <10,000) 129 17.9
Town or city (population 10,000 - 50,000) 154 21.4
Suburb of city (population > 50,000) 108 15.0
Central city (population > 50,000) 138 19.9
Mean (SD)
Age
Overall sample 14.13 (2.42)
Grades 4-7 (N=361) 12.11 (1.21)
Grades 8-12 (N=360) 16.15 (1.44)

Measures

Table 2 provides a list of the measures included in this article. For each scale, the development team for the Common Measures proposed different items for younger (Grades 4-7) and older (Grades 8-12) youth. In California, for science, youth of all grades received the measure designed for older youth. Therefore, some of the items from the National 4-H science measure for youth in grades 4-7 were not collected.

Table 2.
Scales and Number of Items for Each Age Group
Measure Number of Items
Grades 4-7 Grades 8-12
Science
Attitudes 5 6
Interest 4 4
Skills 3 5
Application -- 4
Total 12 19
Healthy Living
Follow Dietary Guidelines 18 18
Physical Activity 5 3
Risk Prevention Behaviors 12 12
Total 35 33
Citizenship
Awareness of Community Issues -- 7
Cultural Diversity 1 5
Community Engagement 5 9
Understanding the Democratic Process 2 3
Total 8 24
Youth Development
Make Positive Choices 9 10
Effectively Communicate 7 9
Build Connections 4 5
Contribution 5 5
Total 25 29

Analysis Steps

Prior to analyses, we (1) standardized (created z-scores for) items from subscales that varied in response options (e.g., some healthy living items had four response options and some had five response options) and (2) dropped items with extremely low variance (e.g., "When you use a firearm, how often do you follow safety rules"). Exploratory factor analysis (EFA; e.g., Santos & Clegg, 1999) was conducted for each measure to test whether the hypothesized subscales emerged (Table 2). Five steps were followed:

  1. Ran an unconstrained EFA (i.e., did not specify how many subscales were expected).
  2. Dropped any items that did not clearly belong on at least one subscale (i.e., items that had factor loadings lower than .30 on all factors).
  3. Items that loaded onto a factor with only one other item were combined to make a new item that was the average of the two items to avoid problems with reliability of two-item subscales (cf. Widaman, Gibbs, & Geary, 1987).
  4. Re-ran the EFA and repeated steps 2 and 3 as needed.
  5. Final subscales were retained when a solution was found in which all remaining items worked well.

The CA recommended scales (hereby referred to as "CA scales") were compared with the proposed National scales (hereby referred to as "National scales"). We tested whether the CA scales appear to be as strong and tap the same proposed constructs as the National scales by comparing the alpha reliabilities and correlates with other constructs. In this comparison, the average (across the subscales) was also tested for each scale as this is another common option that evaluators use in analyzing data.

Appendix A shows the full list of items and indicates which items were retained. Appendix B shows correlations between National and CA scales and other outcomes collected by CA. Appendix C shows correlations between the National and CA scales.

Results

Science

  • For younger youth, National proposed two subscales; the EFA revealed one factor.
  • For older youth, National proposed four subscales; the EFA revealed three.
  • Skills and Application subscales came out as proposed by National for older youth.
  • For both age groups, Attitudes and Interest emerged as one factor. Subsequent analyses further suggest these two subscales measure the same construct. Specifically,
    • The two subscales were highly correlated (r = .72 for younger youth and .77 for older youth),
    • For both ages, Attitudes and Interest show the same pattern of correlations (see Appendix B), and
    • Multiple regression analyses revealed that for both age groups, the Interest subscale did not significantly add information over and above Attitudes.
  • CA recommends using the Skills and Application subscales as proposed by National, and using only the Attitudes subscale because the Interest subscale appears to provide redundant information (younger youth) or is unrelated to outcomes when controlling for Attitudes (older youth).
  • Table 3 presents the reliability and descriptive statistics for these results.
Table 3.
Reliability and Descriptive Statistics for the Two Science Measures
Subscale Sample Size Number of Items Alpha Mean (SD) Range
Grades 4-7
Attitudes 287 5 .84 3.00 (.58) 1.20, 4.00
Interest 291 4 .84 3.24 (.58) 1.00, 4.00
Average of all items 276 9 .90 3.10 (.54) 1.44, 4.00
Grades 8-12
Attitudes 331 6 .91 2.95 (.63) 1.00, 4.00
Interest 321 4 .86 3.08 (.61) 1.00, 4.00
Skills 336 5 .92 2.92 (.83) 1.00, 4.00
Application 328 4 .60 0.56 (.34) 0.00, 1.00
Average of all itemsa 337 19 .91 0.00 (.59) -2.19, 1.27
asubscale that requires item standardization prior to analyses for the National versions.

Citizenship

  • National proposed three factors for younger youth and four factors for the older youth, but the EFA suggested one overall Citizenship Scale for both age groups.
  • Because an overall Citizenship factor emerged for both older and younger youth, CA recommends using the items that overlap between the age groups to remove unnecessary items and provide a common measure across age (see Appendix A).
  • The correlations patterns were similar between the National and CA scales for younger and older youth (see Appendix B).
  • The CA and National subscales were highly correlated, suggesting that these measure the same construct (see Appendix C).
  • Table 4 presents the reliability and descriptive statistics for these results.
Table 4.
Reliability and Descriptive Statistics for the Two Citizenship Measures
Sample Size Number of Items Alpha Mean (SD) Range
Subscale Nat CA Nat CA Nat CA Nat CA Nat CA
Citizenship-Grades 4-7
Cultural Diversity 268 -- 1 -- -- -- 3.32 (.59) -- 1.00, 4.00 --
Community engagement 270 -- 5 -- .76 -- 3.45 (.47) -- 1.80, 4.00 --
Democratic process 266 -- 2 -- .54 -- 3.38 (.49) -- 2.00, 4.00 --
Average of items 270 -- 8 8 .83 .83 3.41 (.42) 3.41 (.42) 2.25, 4.00 2.25, 4.00
Citizenship-Grades 8-12
Awareness of community issues 191 -- 7 -- .84 -- 3.12 (.44) -- 2.00, 4.00 --
Cultural Diversity 195 -- 5 -- .77 -- 3.32 (.46) -- 2.20, 4.00 --
Community engagement 196 -- 9 -- .81 -- 3.27 (.46) -- 1.00, 4.00 --
Understanding of democratic process 191 -- 3 -- .76 -- 3.36 (.48) -- 1.67, 4.00 --
Average of items 196 196 24 8 .92 .87 3.24 (.41) 3.41 (.42) 1.00, 4.00 2.00, 4.00
Note. Nat = National Scale; CA = California Scale. For scales with only two items, the correlation for the two items is presented instead of an alpha.

Healthy Living

  • National proposed three subscales for each age group.
  • All items for both age groups had large amounts of missing data, possibly due in part to the non-applicability of the item; e.g., "I follow safety rules when using a firearm" and in part due to the length of the scale.
  • The sample size for both age groups was not large enough to conduct an EFA, as few youth completed all the questions within the measure. Therefore, we only present descriptive statistics for the National recommended subscales in Table 5.
  • For all youth, the National subscales show low to moderate correlations with related CA outcomes (see Appendix B) and low to moderate correlations among themselves (see Appendix C). This is expected from scales that tap separate but related constructs.
  • CA will continue to collect data on these measures. Additionally, 4-H evaluators should review the measures for item applicability before utilizing the measure. For example, the item regarding firearm usage may be applicable only to a few youth that are in a shooting sports project.
Table 5.
Descriptive Statistics for the Two Healthy Living Measures
Subscale Sample Size Number of Items Alpha Mean (SD) Range
Healthy Living-Grades 4-7
Follow Dietary Guidelinesa 228 18 .81 -0.00 (.50) -2.04, 2.16
Physical Activitya 227 5 .70 -0.04 (.74) -5.71, 0.46
Risk Prevention Behaviors 277 12 .93 3.42 (.49) 1.00, 4.00
Average of all itemsa 229 35 -- 0.00 (.45) -2.33, 1.44
Healthy Living-Grades 8-12
Follow Dietary Guidelinesa 169 18 -- -0.04 (.69) -2.63, 1.29
Physical Activitya 181 3 .49 -0.06 (.73) -2.14, 1.88
Risk Prevention Behaviorsa 183 12 -- -0.03 (.54) -1.77, 1.14
Average of all itemsa 185 33 -- -0.05 (.52) -2.63, 1.24
Note. Because alpha reliabilities and EFA are conducted on the sample of youth that completed all the items within a subscale, some sample sizes were too low for analysis; therefore alpha and EFA results are not presented. We note that all subscales for all measures were computed such that a youth received a score for the subscale if they answered at least one question within the subscale. For example, if youth A answered two questions in the "Risk Prevention Subscale" and youth B answered all 12, both youth received a score based on the mean of the items to which they responded. This represents a liberal approach and other researchers may wish to calculate more conservative scores.
aindicates subscales that require item standardization prior to analyses for the National versions.

Youth Development

  • National proposed four subscales for each age group.
  • All items for both age groups had large amounts of missing data. As with healthy living, the sample size for both age groups was not large enough to conduct an EFA. Descriptive statistics for the National recommended subscales are presented in Table 6.
  • For all youth, the National measures show low to high correlations with related CA outcomes (see Appendix B).
  • Both age groups showed high correlations with a measure of positive youth development, suggesting that the youth development measure is tapping into a similar construct.
  • For all youth, correlations of National's subscales among themselves were high (see Appendix C).
  • CA will continue to collect data on these measures for future analyses.
Table 6.
Descriptive Statistics for the Two Youth Development Measures
Subscale Sample Size Number of Items Alpha Mean (SD) Range
Grades 4-7
Positive choices 173 9 .86 3.30 (0.45) 2.22, 4.00
Communication 173 7 .78 3.25 (0.48) 2.00, 4.00
Connections 173 4 .76 3.43 (0.48) 1.75, 4.00
Contribution 166 5 .84 3.35 (0.48) 2.00, 4.00
Average of all items 174 25 -- 3.30 (0.43) 2.00, 4.00
Grades 8-12
Positive choices 86 10 -- 3.38 (0.42) 2.50, 4.00
Communication 86 9 -- 3.34 (0.46) 2.33, 4.00
Connections 86 5 -- 3.46 (0.48) 2.40, 4.00
Contribution 82 5 -- 3.27 (0.55) 1.00, 4.00
Average of all items 86 29 -- 3.35 (0.42) 2.13, 4.00
Note. Because alpha reliabilities and EFA are conducted on the sample of youth that completed all the items within a subscale, some sample sizes were too low for analysis; therefore this information is not presented.

Recommendations and Best Practices

We recommend that the measures undergo further refinement before broad use. CA recommendations are the following.

  1. Keep item response options consistent across all items for each scale. The current measures propose subscales that include items with varying response categories, in terms of both labels and the number of categories. Items with different response categories cannot be used to form a scale unless the items are first standardized. This requirement may result in a loss of valuable information, and it will be easy for researchers to miss this step and conduct analyses that are not meaningful.
  2. Whenever possible do not use items that vary across age groups. When items vary across age groups it is not possible to study developmental trends (unless very sophisticated statistical models and analyses are used). That is, when item content changes across age it is no longer possible to disentangle differences due to age versus differences due to the change in wording or item content. Thus, long-term impacts of a program cannot be assessed. We note, however, that longitudinal analysis was not one of the original goals of National 4-H, and states may not be interested in evaluating youth outcomes over time.
  3. Make item wording more general. The wording of items is specific to youth involved in 4-H. Unless the items are reworded, potentially changing the psychometrics of the items, comparisons of youth in other programs cannot be made. Again, we note that this was not one of the original goals, and states may not be interested in comparing youth in other programs.
  4. Have a core set of items that are applicable for most youth, with additional "supplemental" items that may be project-specific. For example, the item "I follow safety rules when using a firearm" in not applicable to most youth, lending to the extremely low variance of the item.

For 4-H evaluators and practitioners utilizing the measures, we recommend the following.

  1. Science: Use the Attitudes, Skills, and Application subscales.
  2. Citizenship: Use the items present in the scale for youth ages nine to 13 years for all ages.
  3. Healthy Living: Use the measure as described by National 4-H until further analyses can be conducted. Items with limited relevance to youth (e.g. "I wear a helmet when riding an all-terrain vehicle") should only be included when relevant to the 4-H project.
  4. Youth Development: Use the measure as described by National 4-H until further analyses can be conducted.

Summary

A common set of tools to measure youth outcomes across programs and states is useful for program evaluation (Payne & McDonald, 2012). In the study reported here, we found support for several of the subscales as proposed by National 4-H; however, some subscales did not emerge, and missing data and extremely low variance on some items made it difficult to conduct analyses. Data were drawn from the community club program of 4-H, and that the demographics of youth who participate in this delivery mode are consistent throughout the state. However, this is a limitation of our study that may limit the generalizability of the findings to more diverse populations. Other programs that pilot these measures on more diverse samples should publish their psychometric findings for comparison. Appendices and EFA and scale creation syntax are available from http://4h.ucanr.edu/Research/4HPublications/commonmeasure/.

References

Lackman, L., Nieto, R. D., & Gliem, R. (1997). Instrument development for low literacy audiences: Assessing Extension program personnel teaching effectiveness. Journal of Extension [On-line], 35(1) Article 1RIB1. Available at: http://www.joe.org/joe/1997february/rb1.php

Payne, P. B., & McDonald, D. A. (2012). Using common evaluation instruments across multi-site community programs: A pilot study. Journal of Extension [On-line], 50(4) Article 4RIB2. Available at http://www.joe.org/joe/2012august/rb2.php

Radhakrishna, R. B. (2007). Tips for developing and testing questionnaires/instruments. Journal of Extension [On-line], 45(1) Article 1TOT2. Available at: http://www.joe.org/joe/2007february/tt2.php

Rennekamp, R. A., & Arnold, M. E. (2009). What progress, program evaluation? Reflections on a quarter-century of Extension evaluation practice. Journal of Extension [On-line], 47(3) Article 3COM1. Available at: http://www.joe.org/joe/2009june/comm1.php

Santos, J. R. A., & Clegg, M. D. (1999). Factor analysis adds new dimension to extension surveys. Journal of Extension [On-line], 37(5) Article 5RIB6. Available at: http://www.joe.org/joe/1999october/rb6.php

Stewart, P. K., Roberts, M. C., & Kim, K. L. (2010). The psychometric properties of the Harter Self-perception Profile for Children with at-risk African American females. Journal of Child and Family Studies, 19, 326-333. doi: 10.1007/s10826-009-9302-x

Van Tilburg Norland, E. (1990). Controlling error in evaluation instruments. Journal of Extension [On-line], 28(2) Article 2TOT2. Available at: http://www.joe.org/joe/1990summer/tt2.php

Widaman, K. F., Gibbs, K. W., & Geary, D. C. (1987). Structure of adaptive behavior: I. Replication across fourteen samples of nonprofoundly mentally retarded people. American Journal of Mental Deficiency, 91, 348-360. Retrieved from: http://psycnet.apa.org/psycinfo/1987-15966-001

Appendix A

List of Common Measure Items

Science Literacy-Grades 4 to 7
National Scale California Scale
Item # Item Subscale Factor Loading CA recommended edits Subscale Factor Loading
1. I like science Attitudes .82 -- Attitudes .82
2. I am good at science Attitudes .76 -- Attitudes .71
3. I would like to have a job related to science Attitudes .81 -- Attitudes .67
4. I do science activities that are not for school Attitudes .77 -- Attitudes .70
5. I think science is useful for solving everyday problems Attitudes .80 -- Attitudes .67
6. I like to see how things are made or invented Interest .85 -- Attitudes .69
7. I like experimenting and testing ideas Interest .80 -- Attitudes .67
8. I get excited about new discoveries Interest .83 -- Attitudes .69
9. I want to learn more about science Interest .80 -- Attitudes .83
10. I can do an experiment to answer a question Skills Data not collected on this item
11. I can tell others how to do an experiment Skills Data not collected on this item
12. I can explain why things happen in an experiment Skills Data not collected on this item
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor.
Science Literacy-Grades 8 to 12
National Scale California Scale
Item # Item Subscale Factor Loading CA recommended edits Subscale Factor Loading
1. I like science Attitudes .87 -- Attitudes .89
2. I am good at science Attitudes .85 -- Attitudes .84
3. I would like to have a job related to science Attitudes .85 -- Attitudes .79
4. I do science activities that are not for school Attitudes .81 -- Attitudes .83
5. I think science will be important in my future Attitudes .87 -- Attitudes .82
6. I think science is useful for solving everyday problems Attitudes .77 -- Attitudes .55
7. I like to see how things are made or invented Interest .83 Exclude -- --
8. I like experimenting and testing ideas Interest .87 Exclude -- --
9. I get excited about new discoveries Interest .84 Exclude -- --
10. I want to learn more about science Interest .82 Exclude -- --
11. I can use scientific data to form a question Skills .83 -- Skills .70
12. I can design a scientific procedure to answer a question Skills .89 -- Skills .90
13. I can use data to create a graph for presentation to others Skills .88 -- Skills .86
14. I can create a display to communicate my data and observations Skills .85 -- Skills .83
15. I can use science terms to share my results Skills .87 -- Skills .85
16. I have helped with a community service project that relates to science (for example: planted trees or garden, road or stream clean-up, recycling) Application .79 -- Application .69
17. In my 4-H program, I used science tools to help in the community (for example: mapped with GIS, tested water quality) Application .92 -- Application .92
18. I taught others about science (for example: demonstrated, gave presentation, led a project) Application .89 -- Application .86
19. I organized or led science-related events (for example: science fair, environmental festival) Application .92 -- Application .90
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor.
Citizenship-Grades 4-7
National Scale California Scale
Item # Item Subscale Factor Loading CA recommended edits Subscale Factor Loading
1. I enjoyed learning about people who are different from me Cultural Diversity -- -- Citizenship .64
2. I can make a difference in my community through community service Community Engagement .78 -- Citizenship .69
3. I help make sure everyone gets an opportunity to say what they think Understanding Democratic Process .88 -- Citizenship .69
4. I can apply knowledge in ways that solve "real-life" problems through community service Community Engagement .79 -- Citizenship .68
5. I gained skills through serving my community that will help me in the future Community Engagement .79 -- Citizenship .67
6. I treat everyone fairly and equally when I am in charge of a group Understanding Democratic Process .88 -- Citizenship .56
7. I plan to work on projects to better my community Community Engagement .71 -- Citizenship .52
8. I am encouraged to volunteer more Community Engagement .91 -- Citizenship .52
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor.
Citizenship-Grades 8-12
National Scale California Scale
Item # Item Subscale Factor Loading CA recommended edits Subscale Factor Loading
1. I pay attention to news events that affect my community Awareness of Community & Community Issues .57 Exclude -- --
2. I am aware of the important needs in my community Awareness of Community & Community Issues .75 Exclude -- --
3. I really care about my community Awareness of Community & Community Issues .68 Exclude -- --
4. I talk to my friends about issues affecting my community, state, or world Awareness of Community & Community Issues .68 Exclude -- --
5. I'm interested in others' opinions about public issues Awareness of Community & Community Issues .78 Exclude -- --
6. I listen to everyone's views whether I agree or not Awareness of Community & Community Issues .65 Exclude -- --
7. When I hear about an issue, I try to figure out if they are just telling one side of the story Awareness of Community & Community Issues .65 Exclude -- --
8. I explore cultural differences Cultural Diversity .78 Exclude -- --
9. I value learning about other cultures Cultural Diversity .81 Exclude -- --
10. I respect people from different cultures Cultural Diversity .73 Exclude -- --
11. I have learned about people who are different from me Cultural Diversity .73 -- Citizenship .56
12. I can make a difference in my community through community service Community Engagement .72 -- Citizenship .71
13. I help make sure everyone gets an opportunity to say what they think Understanding Democratic Process .81 -- Citizenship .75
14. I can apply knowledge in ways that solve "real-life" problems through community service Community Engagement .71 -- Citizenship .73
15. I gained skills through serving my community that will help me in the future Community Engagement .73 -- Citizenship .80
16. I treat everyone fairly and equally when I am in charge of a group Understanding Democratic Process .86 -- Citizenship .73
17. I am able to lead a group in making a decision Understanding Democratic Process .71 Exclude -- --
18. I would enjoy hosting someone from another culture Cultural Diversity .44 Exclude -- --
19. I can contact someone I've never met before to get their help with a problem Community Engagement .50 Exclude -- --
20. I plan to work on projects to better my community Community Engagement .78 -- Citizenship .65
21. I am encouraged to volunteer more Community Engagement .70 -- Citizenship .51
22. After high school I will continue to work to better my community Community Engagement .73 Exclude -- --
23. I am interested in a career that helps others Community Engagement .55 Exclude -- --
24. I am interested in working in government (such as school board, Director of parks and rec, legislator, legislative aide, intern) Community Engagement .38 Exclude -- --
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor.
Healthy Living-Grades 4-7
Item # Item Subscale Factor Loading
1. Eat fruit for a snack Dietary Guidelines -.10
2. Eat vegetables for a snack Dietary Guidelines -.17
3. Choose water instead of soda pop or Kool-Aid when I am thirsty. Dietary Guidelines -.01
4. Drink 1% or skim milk instead of 2% or whole milk Dietary Guidelines -.23
5. Choose a small instead of a large order of French fries Dietary Guidelines -.07
6. Eat smaller servings of high fat foods like French fries, chips, snack cakes, cookies, or ice cream Dietary Guidelines -.17
7. Eat a low-fat snack like pretzels instead of chips Dietary Guidelines -.04
8. Drink less soda pop Dietary Guidelines -.00
9. Drink less Kool-Aid Dietary Guidelines .01
10. I do moderate physical activities like walking, helping around the house, raking leaves, or using the stairs Physical Activity .38
11. I exercise 30-60 minutes every day Physical Activity .22
12. Being active is fun Physical Activity .76
13. Being active is good for me Physical Activity .89
14. Physical activity will help me stay fit Physical Activity .91
15. I learned the foods that I should eat every day Dietary Guidelines .81
16. I learned what makes up a balanced diet Dietary Guidelines .80
17. I learned why it is important for me to eat a healthy diet Dietary Guidelines .89
18. I learned how to make healthy food choices Dietary Guidelines .88
19. I eat more fruits and vegetables Dietary Guidelines .77
20. I eat more whole grains Dietary Guidelines .75
21. I eat less junk foods Dietary Guidelines .60
22. I drink more water Dietary Guidelines .73
23. I encourage my family to eat meals together Dietary Guidelines .71
24. When I cook food I am safe and careful Risk Prevention .88
25. If I am sick, I ask an adult before taking medicine Risk Prevention .84
26. I wear a helmet when I ride a bicycle Risk Prevention .61
27. I wear a helmet when I rollerblade or ride a skateboard Risk Prevention .62
28. I wear a helmet when riding an All-Terrain Vehicle Risk Prevention .69
29. I follow safety rules when using a firearm or bow Risk Prevention .86
30. I wear reflective clothing when walking after dark Risk Prevention .64
31. I use a pedestrian crossing when crossing the road Risk Prevention .85
32. I tell my friends what I think when they are going to do something unsafe Risk Prevention .75
33. I avoid using substances that could harm me Risk Prevention .86
34. I wear a seat belt when riding in a car Risk Prevention .88
35. I avoid riding in cars with unsafe drivers Risk Prevention .86
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor.
Healthy Living-Grades 8-12
Item # Item Subscale Factor Loading
1. I learned about the foods that I should eat every day Dietary Guidelines .78
2. I learned what makes up a balanced diet Dietary Guidelines .82
3. I learned why it is important for me to eat a healthy diet Dietary Guidelines .73
4. I learned how to make healthy food choices Dietary Guidelines .78
5. I learned how many calories I need to eat each day Dietary Guidelines .60
6. I learned the importance of fruits and vegetables in my diet Dietary Guidelines .74
7. I learned the importance of whole grains in my diet Dietary Guidelines .66
8. I think about what foods my body needs during the day Dietary Guidelines .77
9. I make food choices based on what I know my body needs Dietary Guidelines .71
10. I make healthy food choices whenever I can Dietary Guidelines .69
11. I match my food intake to the number of calories I need to eat each day Dietary Guidelines .43
12. I eat more fruits and vegetables Dietary Guidelines .55
13. I eat more whole grains Dietary Guidelines .63
14. I eat less junk foods Dietary Guidelines .63
15. I drink less soda Dietary Guidelines .59
16. I drink more water Dietary Guidelines .45
17. I encourage my family to eat meals together Dietary Guidelines .41
18. When I cook food, I am safe and careful Risk Prevention .68
19. If I am sick, I ask an adult before taking medicine Risk Prevention .69
20. I wear reflective clothing when walking after dark Risk Prevention .71
21. I use pedestrian crossings when crossing the road Risk Prevention .79
22. I tell my friends what I think when they are going to do something unsafe Risk Prevention .68
23. I avoid using substances that could harm me Risk Prevention .63
24. My family eats at least one meal a day together Dietary Guidelines -.06
25. During the past 7 days, on how many days were you physically active for a total of at least 60 minutes per day (add up all the time you spent in any kind of physical activity that increased your heart rate and made you breathe hard some of the time) Physical Activity .54
26. On an average school day, how many hours do you spend watching television? Physical Activity .76
27. On an average school day, how many hours do you play video games, looking at a computer, smartphone or tablet for something that is not for school? Physical Activity .79
28. When you ride a bicycle how often do you wear a helmet Risk Prevention .64
29. When you rollerblade or skateboard how often do you wear a helmet? Risk Prevention .47
30. When you ride an All-Terrain Vehicle (ATV) how often do you wear a helmet? Risk Prevention -.11
31. When you use a firearm, how often do you follow safety rules? Risk Prevention .32
32. How often do you use a seatbelt when riding in a car? Risk Prevention -.01
33. Have you ever ridden in a car driven by someone who had been drinking alcohol? Risk Prevention .13
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor.
Youth Development-Grades 4-7
Item # Item Subscale Factor Loading
1. I use information to make decisions Positive choices .59
2. I set goals for myself Positive choices .62
3. I take responsibility for my actions Positive choices .59
4. I listen well to others Communication .68
5. I am respectful of others Communication .66
6. I have the confidence to speak in front of groups Communication .59
7. I can work things out when others don't agree with me Communication .75
8. I work well with other youth Connection .67
9. I am comfortable making my own decisions Positive choices .75
10. I have a plan for reaching my goals Positive choices .74
11. I know how to deal with stress in positive ways Positive choices .76
12. I can explain my decisions to others Positive choices .72
13. I can change my plan when I need to Positive choices .77
14. I don't let my friends talk me into doing something I don't want to do Positive choices .66
15. I am comfortable sharing my thoughts and feelings with others Communication .74
16. I can use technology to help me express my ideas Communication .67
17. I know who I can go to if I need help with a problem Communication .55
18. I can work successfully with adults Connection .79
19. I have friends who care about me Connection .81
20. I am connected to adults who are not my parents Connection .80
21. I am someone who wants to help others Contribution .80
22. I like to work with others to solve problems Contribution .82
23. I have talents I can offer to others Contribution .79
24. I learned things that helped me make a difference in my community Contribution .77
25. I helped with a project that made a difference in my community Contribution .72
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor
Youth Development-Grades 8-12
Item # Item Subscale Factor Loading
1. I use information to make decisions Positive choices .56
2. I set goals for myself Positive choices .64
3. I take responsibility for my actions Positive choices .63
4. I can explain why my decision is a good one Positive choices .68
5. I consider the consequences of my choices Positive choices .71
6. I can resist negative social pressures Positive choices .60
7. I listen well to others Communication .59
8. I am respectful of others Communication .67
9. I have the confidence to speak in front of groups Communication .59
10. I can resolve differences with others in positive ways Communication .66
11. I work well with other youth Connection .43
12. I am comfortable making my own decisions Positive choices .53
13. I have a plan for reaching my goals Positive choices .72
14. I know how to deal with stress in positive ways Positive choices .71
15. I can make alternative plans if something doesn't work Positive choices .76
16. I am comfortable sharing my thoughts and feelings with others Communication .80
17. I can use technology to help me express my ideas Communication .67
18. I know who I can go to if I need help with a problem Communication .74
19. I am willing to consider the ideas of others even if they are different than mine Communication .77
20. I can stand up for things that are important to me Communication .75
21. I can work successfully with adults Connection .82
22. I have friends who care about me Connection .83
23. I know community leaders who support me Connection .91
24. I have adults in my life who care about me and are interested in my success Connection .89
25. I am someone who wants to help others Contribution .89
26. I like to work with others to solve problem Contribution .90
27. I have talents I can offer to others Contribution .82
28. I learned things that helped me make a difference in my community Contribution .90
29. I led a project that made a difference in my community Contribution .78
Note: Factor loadings for the National scales derived from an exploratory factor analysis for each subscale with the specification to retain one factor.

Appendix B

Correlations of the National Scales and California Scales with other Outcomes

Science-Grades 4 to 7
Mindset Stress Depression Self Esteem Spark Goals YSOC PYD
National
Attitudes .29** -.17** -.06 .22** .24** .15 -.00 .35**
Interest .25** -.16** -.03 .20** .22** .17* .02 .44**
Average .30** -.18** -.05 .23** .25** .19* .01 .42**
California
Attitudes .30** -.18** -.05 .23** .25** .19* .01 .42**
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.
Science-Grades 8 to 12
Mindset Stress Depression Self Esteem Spark Goals YSOC PYD
National
Attitudes .10 -.24** -.09 .16** .19** .23** .10 .27**
Interest .10 -.20** -.02 .16** .16** .19** .13* .31**
Skills .13* -.21** -.08 .16** .22** .23** -.06 .25**
Application -.02 -.05 -.03 .08 -.01 .03 .30** .10
Average .12* -.26** -.08 .21** .21** .26** .15** .35**
California
Attitudes .10 -.24** -.09 .16** .19** .23** .10 .27**
Skills .13* -.21** -.08 .16** .22** .23** -.06 .25**
Application -.02 -.05 -.03 .08 -.01 .03 .30** .10
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.
Citizenship-Grades 4 to 7
Mindset Stress Depression Self Esteem Spark Goals YSOC PYD
National
Cultural Diversity .19** -.20** -.07 .19* .24** .21* .08 .50**
Community Engagement .23** -.22* .03 .23** .40** .24** .10 .61**
Democratic Process .34** -.26** -.10 .38** .17* .36** .14* .59**
Average .29** -.25** -.06 .29** .36** .30** .13* .67**
California
Citizenship .29** -.25** -.06 .29** .36** .30** .13* .67**
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.
Citizenship-Grades 8 to 12
Mindset Stress Depression Self Esteem Spark Goals YSOC PYD
National
Awareness of community issues .20** -.24** -.18* .27** .30** .48** .06 .58**
Cultural Diversity .19* -.06 .09 .19** .31** .18 -.02 .46**
Community engagement .15* -.24** -.08 .20** .38** .41** .07 .57**
Democratic Process .24** -.18* -.04 .24** .30** .25** .03 .66**
Average .17* -.25** -.09 .25** .39** .45** .04 .65**
California
Citizenship .19** -.18* -.02 .23** .40** .42** .06 .70**
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.
Healthy Living-Grades 4 to 7
Mindset Stress Depression Self Esteem Spark Goals YSOC PYD
National
Follow Dietary Guidelines .10 -.05 .14* .08 .10 .14 .16* .19**
Physical Activity .12 -.14* -.09 .19** .06 .28** .02 .25**
Risk Prevention Behaviors .20** -.23** -.07 .19* .21** .28** .10 .42**
Average .17* -.14* .10 .18** .16* .27** .13 .34**
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.
Healthy Living-Grades 8 to 12
Mindset Stress Depression Self Esteem Spark Goals YSOC PYD
National
Follow Dietary Guidelines .18* -.15 -.20 .21** .27** .35** .13 .47**
Physical Activity -.05 -.12 -.10 -.09 -.08 .24* .05 .25**
Risk Prevention Behaviors .23** -.16* -.13 .06 .19* .33** .15* .33**
Average .16* -.19* -.07 .13 .29** .45** .20** .50**
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.
Youth Development-Grades 4 to 7
Mindset Stress Depression Self Esteem Spark Goals YSOC PYD
National
Choices .13 -.36** -.14 .33** .31** .52** .11 .67**
Communication .16** -.30** -.16* .36** .35** .47** .05 .64**
Connections .21** -.36** -.20* .37** .30** .48** .06 .69**
Contribution .07 -.33** -.14 .29** .27** .36** .19* .62**
Average .17** -.40** -.19* .36** .34** .49** .11 .72**
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.
Youth Development-Grades 8 to 12
Mindset Stress Depression Self Esteem Spark Goals YSOC PYD
National
Choices .31** -.26* -.11 .24* .42** .58** .00 .62**
Communication .25* -.14 -.04 .23* .40** .48** -.05 .59**
Connections .29** -.24* -.17 .28* .16 .38** -.09 .43**
Contribution .28* -.13 -.01 .22* .29* .40** .01 .45**
Average .31** -.22* .09 .28** .39** .53** -.03 .63**
*p < .05, **p < .01. YSOC= Youth Selection, Optimization, and Compensation; PYD=Positive Youth Development.

Appendix C

Correlations between National Scales and California Scales

Science-Grades 4 to 7
Nat-Attitudes Nat-Interest Nat-Average/CA-Attitudes
Nat-Attitudes -- .72** .94**
Nat-Interest .72** -- .91**
Nat-Average/CA-Attitudes .94** .91** --
*p < .05, **p < .01. Nat = National Scales, CA= California Scales.
Science-Grades 8 to 12
Nat/CA-Attitudes Nat-Interest Nat/CA-Skills Nat/CA-Application Nat-Average
Nat/CA-Attitudes -- .77** .53** .03 .85**
Nat-Interest .77** -- .50** .05 .83**
Nat/CA-Skills .53** .50** -- -.01 .77**
Nat/CA-Application .03 .05 -.01 -- .28**
Nat-Average .85** .83** .77** .28** --
*p < .05, **p < .01. Nat = National Scales, CA= California Scales.
Citizenship-Grades 4 to 7
Nat-Cultural Diversity Nat-Community Engagement Nat-Democratic Process Nat-Average/ CA Citizenship
Nat-Cultural Diversity -- .52** .50** .67**
Nat-Community Engagement .52** -- .56** .94**
Nat-Democratic Process .50** .56** -- .77**
Nat-Average/CA-Citizenship .69** .94** .77** --
*p < .05, **p < .01. Nat = National Scales, CA= California Scales.
Citizenship-Grades 8 to 12
Nat-Awareness of community issues Nat-Cultural Diversity Nat-Community Engagement Nat- Democratic Process Nat-Average CA-Citizenship
Nat-Awareness of community issues -- .64** .66** .60** .88** .67**
Nat-Cultural Diversity .64** -- .51** .56** .75** .63**
Nat-Community Engagement .67** .51** -- .62** .91** .86**
Nat- Democratic Process .60** .56** .62** -- .77** .80**
Nat-Average .88** .75** .91** .77** -- .89**
CA-Citizenship .67** .63** .86** .80** .89** --
*p < .05, **p < .01. Nat = National Scales, CA = California Scales.
Healthy Living-Grades 4 to 7
Nat-Follow Dietary Guidelines Nat-Physical Activity Nat-Risk Prevention Behaviors Nat-Average
Nat-Follow Dietary Guidelines -- .09 .45* .85**
Nat-Physical Activity .09 -- .17* .42**
Nat-Risk Prevention Behaviors .45** .17* -- .77**
Nat-Average .85** .42** .77** --
*p < .05, **p < .01. Nat = National Scale.
Healthy Living-Grades 8 to 12
Nat-Follow Dietary Guidelines Nat-Physical Activity Nat-Risk Prevention Behaviors Nat-Average
Nat-Follow Dietary Guidelines -- .03 .49** .85**
Nat-Physical Activity .03 -- .03 .30**
Nat-Risk Prevention Behaviors .49** .03 -- .77**
Nat-Average .85** .30** .77** --
*p < .05, **p < .01. Nat = National Scale.
Youth Development- Grades 4 to 7
Nat-Choices Nat-Communicate Nat-Connections Nat-Contribution Nat-Average
Nat-Choices -- .78** .74** .74** .93**
Nat-Communicate .78** -- .74** .69** .91**
Nat-Connections .74** .74** -- .78** .87**
Nat-Contribution .74** .69** .78** -- .85**
Nat-Average .93** .91** .87** .85** --
*p < .05, **p < .01. Nat = National Scale.
Youth Development- Grades 8 to 12
Nat-Choices Nat-Communicate Nat-Connections Nat-Contribution Nat-Average
Nat-Choices -- .82** .68** .61** .91**
Nat-Communicate .82** -- .78** .70** .94**
Nat-Connections .68** .78** -- .72** .84**
Nat-Contribution .61** .70** .72** -- .83**
Nat-Average .91** .94** .84** .83** --
*p < .05, **p < .01. Nat = National Scale.