The Journal of Extension - www.joe.org

August 2012 // Volume 50 // Number 4 // Tools of the Trade // 4TOT3

Content Appraisal as a Method for Measuring the Effectiveness & Usability of Online Content

Abstract
Content appraisal is a simple, qualitative system to identify modifications to make website material more useful to clientele. This system provides a comprehensive evaluation of content, focuses on content strategy, identifies weaknesses, and provides recommendations for improvement. The criteria examined included knowledge level, interrelatedness, relevance, usability, actionability, and differentiation. HorseQuest, a pioneer Community of Practice (CoP) for eXtension, was the first to apply a content appraisal process in an attempt to document the efficacy and impact of their web content. This appraisal system can be effectively used by other groups to help improve a website's usefulness to clientele.


Jenifer Nadeau
Equine Extension Specialist and Associate Professor
University of Connecticut
Storrs, Connecticut
Jenifer.nadeau@uconn.edu

Neely Heidorn
Equine Extension Specialist and Assistant Professor
Louisiana State University
Baton Rouge, Louisiana
NHeidorn@agcenter.lsu.edu

Nick Broady
Technology Trainer
University of Kentucky
Lexington, Kentucky
nick.broady@extension.org

Jennifer Whittle
Graduate Student
HorseQuest Community of Practice
University of Kentucky
Lexington, Kentucky
Jennifer.whittle@uky.edu

Introduction

Content appraisal, a simple qualitative system, will provide Extension personnel and other website users with a method that is easy to use that ensures relevance and usefulness to clientele, increasing its effectiveness. Previous website analysis has focused mainly on the survey of users and the use of Web analytics software (Parish, 2011) or Neilsen's heuristics (Nielsen, 1994; Nielsen, 2005). Web analytics software, such as Google Analytics, provides detailed information regarding the use of a the website, including page views, average time users spend on pages, and bounce rate. Nielsen's heuristics are the 10 general principles for the design of the user interface, which focus on usability of a website. In contrast, the content appraisal system involves the thorough evaluation of content and results in a report that focuses on content strategy and identification of weaknesses and provides recommendations for improvement.

HorseQuest, an eXtension Community of Practice (CoP), used this method to evaluate content on their site. The criteria examined included knowledge level, interrelatedness, relevance, usability, actionability, and differentiation. HorseQuest was the first CoP to create a content appraisal report based on these criteria. The purpose of analyzing the content on eXtension.org was to review the existing site for overall effectiveness and adherence to HorseQuest's goals. In the strategic content foundation for this project, these three organizational goals were identified:

  • Empower people to make smart decisions that improve the quality of their lives.
  • Provide credible, reliable, research-based information, tools, and solutions people can use.
  • Bestow the knowledge from the best minds at land grant universities to the community-at-large.

In addition, we've identified the following key guiding principles for content creation:

  • Content developed by HorseQuest becomes part of that community.
  • Content creators will be credited for their contributions, which may affect promotions and tenure.
  • Content contributors should believe in collaborating on content—including letting others in the HorseQuest CoP make decisions about content they've created.
  • Duplication of existing content should be avoided.
  • Content contributors should strive to understand who their content is for—local audience vs. national audience; novice reader vs. expert reader.
  • CoP content will need to change and evolve to meet the needs of the HorseQuest CoP audiences. (adapted from Brain Traffic's eXtension Qualitative Report, 2010)

Materials and Methods

The content appraisal report was organized by sections. Section one included the goals of the organization, a "what we did" section which described the scope, content selection, process of content appraisal, and appraisal attributes (criteria). Appraisal attributes included:

  • Knowledge level—the level of specialized knowledge needed to understand website content.
  • Interrelatedness—how well pieces of content link to other content on the website.
  • Relevance—how current and of interest the contest is to the intended audience.
  • Usability—the readability and effectiveness of the presentation of the content.
  • Actionability—the explanation of a next step to be taken by the user.
  • Differentiation—how the content fills a unique need or offers a unique perspective.

Section two focused on HorseQuest's content performance, including the format of the content (including answers from our experts, articles, audio, learning lesson, video, news, or other such as glossary terms); appraisal attributes, which included knowledge level, interrelatedness, relevance, usability, actionability, and differentiation; and other observations including overall key findings. Section three discussed the number of pages appraised, areas for improvement and strengths of the site.

Content Appraisal Results

Content appraisal for this report covered 213 webpages. The focus of these results is Section 2, related to content performance, which other Extension professionals will likely find to be most relevant. Percentages may not add up to 100 because of rounding to the nearest whole number.

Format of the material on the website was the first thing appraised (Table 1).

Table 1.
Format of the Material on the Website

Value Meaning TOTAL (of pages appraised)
Answer Content in Answers from our Experts 0.94%
Article Content that is solely or primarily text 37.09%
Audio Content that is solely or primarily sound files 0%
Learning Lesson Online lessons or tools 2.35%
News Content in In the News 0.94%
Video Content that is solely or primarily video 2.81%
Other Glossary terms, other 55.87%

Subject matter knowledge level needed to understand or use the page was appraised (Table 2).

Table 2.
Subject Matter Knowledge Level Needed

Value Meaning TOTAL (of pages appraised)
1-LOW No specialized knowledge is required or assumed. 45.07%
2-MEDIUM Some knowledge is required. 48.83%
3-HIGH A high level of subject matter knowledge is required. 6.10%

Interrelatedness (how the content fitted and linked to other content in the CoP) was appraised (Table 3).

Table 3.
Interrelatedness of Content

Value Meaning TOTAL (of pages appraised)
1-LOW Lack of working links to other content 29.11%
Tone is inconsistent with other content in a way that’s strongly distracting.
Content is off-topic
2-MEDIUM Links exist but some are broken 25.82%
Tone is moderately inconsistent with other CoP content
Tags are mostly absent (e.g., only one tag and it’s non-specific)
3-HIGH Good links, tone, tags 45.07%

Relevance (timeliness and interesting to the intended audience) was appraised (Table 4).

Table 4.
Relevance of Content

Value Meaning TOTAL (of pages appraised)
1-LOW Outdated, no longer relevant 1.88%
2-MEDIUM Appears dated/stale or is of interest only to a small geographic area 5.63%
3-HIGH Timely or "evergreen" content 92.49%

Usability, which focused on the content having clear, descriptive headings and subheadings, appropriate sentence and paragraph length, use of graphics, bullets, tables, charts, etc., where appropriate, and overall readability, was appraised (Table 5).

Table 5.
Usability of Content

Value Meaning TOTAL (of pages appraised)
1-LOW Numerous minor problems, a few moderate problems, or one major problem that seriously interferes with the content. 6.10%
2-MEDIUM Minor to moderate problem(s). 29.58%
3-HIGH No significant problems. 64.32%

Actionability, which focused on whether there was a clear next step for the current audience of the content and a means of facilitation of that step, was appraised (Table 6).

Table 6.
Actionability of Content

Value Meaning TOTAL (of pages appraised)
1-LOW No next steps given. 40.85%
2-MEDIUM Next step is implied or vague; next step depends on a link that is broken; indirect or absent call to action (CTA). 15.02%
3-HIGH Next step is clear, specific, and stated explicitly; effective CTA. 44.13%

Differentiation, which focused on whether the page filled a unique need, provided information or a perspective that is not readily available elsewhere, was appraised (Table 7).

Table 7.
Differentiation of Content

Value Meaning TOTAL (of pages appraised)
1-LOW Topic is better covered elsewhere, and page doesn't contribute to a strong body of content for HorseQuest 29.10%
2-MEDIUM Information is generally known or topic is covered elsewhere, but page provides some modest benefit over other sources, or contribute to providing comprehensive information in HorseQuest 53.52%
3-HIGH Page offers a unique perspective, information not widely known, or better explanations than other sources. 17.38%

Conclusions

The content appraisal process helped the HorseQuest CoP analyze its existing online material and make improvements. It has also helped ensure that the website adheres to the CoP's organizational goals and guiding principles for content creation. Any content that received a low or medium rating is being examined and strengthened. The appraisal attributes used here are just a guideline; website creators can create their own set of standards. A solid content appraisal process can help address problem areas of a website as well as build upon strengths of the website to make content more usable to the audience.

References

Brain Traffic. (2010). eXtension Qualitative Report.

Nielsen, J. (2005). Ten Usability Heuristics. Retrieved from: http://www.useit.com/papers/heuristic/heuristic_list.html 

Nielsen, J. (1994). Enhancing the explanatory power of usability heuristics. Paper presented at the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating Interdependence, 152-158.

Parish J. A. (2011). Website usage information for evaluating beef cattle Extension programming. Journal of Extension [On-line], 49(5) Article 5TOT9. Available at http://www.joe.org/joe/2011october/tt9.php