EVALUATION

Measuring Success - Obviously, if something does not work, we do not want to repeat it. Conversely, if something does work, we want to know not only how well, but why it worked so we can do it again. How can we measure the success of a project? There are any number of techniques and tools that may be used to measure success - and many of them are self-administered.

Evaluation is collecting and using information to answer questions about the program. It is an ongoing process and a way of helping determine how to improve a program. It helps inform decisions - it is a tool to use in problem solving. Two common questions asked about evaluation are:
  1. "Do we need an outside expert?" and
  2. "How much will this cost?"
The answer to the first question is often "no." A self-assessment tool is usually sufficient for assessing a program being done by a parent group. Expert evaluators are usually called in for large, comprehensive programs or for an effort which requires a more precise, research-based approach.

The answer to the second question is "it depends." It depends on the type of evaluation done. Rigorous evaluations of large-scale prevention projects can be expensive. However, it is possible to do self-evaluation projects with a very small financial outlay, especially if reliance on paid consultants is minimized.

Evaluation of a program should be done:
  1. During the design and development of the program,
  2. During the implementation of the program,
  3. At the completion of each activity and
  4. At the end of the program.
There are three types of evaluation:
  1. Process evaluation - describes, measures and assesses program activities and materials. It documents what was done, when, to whom and to how many. It answers questions like: What did we do? When? Where? To how many? What did we do well? What do we need to improve? Process evaluation "counts the widgets." For example, if we want to lower the substance abuse among teenagers, we know that parent education programs can do that: we plan to conduct six parent education classes of 12 sessions each, send a monthly newsletter to all parents, sponsor three billboards, conduct a symposium on parenting and put ads in the weekly shopper and weekly radio shows for the year. Process evaluation assesses to make sure we did all those things we said in our plan we would do.

  2. Outcome evaluation - measures the program achievements and describes the program's immediate effects. It answers questions like; Did the program change the knowledge level, the attitudes, or the behaviors of the participants? Did the program increase the awareness of the participants on the issue it was designed to raise? Factors that we measure are sometimes called "indicators" of the larger problem targeted for attack.

    For example, in the above case example, if we want to lower the use of substance abuse among teenagers, we know that lack of parent monitoring is a risk factor and that our parent education program gives parents skills in monitoring, so our outcome evaluation seeks to measure the change in the number of parents in our program who now monitor their children more closely after the parenting program.

  3. Impact evaluation - looks beyond the immediate results to assess and measure longer term effects and impact on the larger community as well as unintended program effects. It seeks to answer the question: Did the parenting program decrease the use of substance abuse among teenagers in our community?

Why Conduct Program Evaluations?

There are several reasons for conducting evaluations of programs:
  1. To determine the effectiveness of programs for participants;
  2. To document that program objectives have been met;
  3. To provide information about service delivery that will be useful to program staff and other audiences;
  4. To enable program staff to make changes that improve program effectiveness; and
  5. Because funding agencies require it.
Prevention programs that address drug and alcohol use are doing so in a relatively new field. We are still learning about what kinds of prevention strategies are effective, what strategies are not effective and what strategies may even be counter-productive.

For recommended resource publications for program evaluation see Bibliography section entitled "Evaluation/Data Collection" in back of this guide.