The Evaluation Exchange Newsletter
Family Support Issue, Spring 2002
Evaluations to Watch
Two evaluators from SRI
describe the benefits realized by the Parent Institute for Quality
Education when they prefaced their summative evaluation with a
formative evaluation.
In 1999, SRI International (SRI) received support from the Stuart
Foundation to conduct an in-depth formative and summative evaluation
of the Parent Institute for Quality Education (PIQE). PIQE is
a not-for-profit community organization in California that aims
to increase parents’ knowledge and skills to support their
children’s academic achievement and ultimate enrollment
in postsecondary education. In its 15-year history, PIQE has graduated
more than 200,000 parents from its eight-week Parent Institute,
many of whom were new immigrants to the United States.
PIQE, as a learning organization, values the opportunity to partcipate
in both formative evaluations (which enable them to judge the
worth of the program during the program activities) and summative
evaluations (which enable them to judge the program at the end
of the activities). The formative evaluation discussed below has
allowed PIQE to assess and strengthen its current practices. Originally
scheduled to last only one year, the formative evaluation was
expanded to two and a half years to allow time for PIQE to implement
program refinements based on the evaluation findings and to allow
for a second round of observation and feedback on the new methods
and materials.
This article describes how SRI and PIQE worked together on the
formative evaluation and how it supported refinement of the program,
identification of promising practices, and articulation of child,
parent, and teacher outcomes, which will be measured in the summative
evaluation.
Developing
a “Theory of Change”
As an initial step in the evaluation, SRI assisted PIQE in articulating
a “theory of change” to describe how PIQE expects
the program components (e.g., the eight-week class, the teacher
workshop) to produce their desired short-term and long-term outcomes
(e.g., improved parent and teacher attitudes and behaviors, child
educational achievement). All observation and interview protocols
used in the evaluation have been based on this theory of change
and developed collaboratively with PIQE staff.
The first year of formative evaluation (the observations and
interviews discussed below) raised several questions about the
goals and strategies outlined in the first conceptualization of
PIQE’s theory of change. As a result, PIQE revised their
theory of change and articulated new measurable outcomes for parents,
teachers, and students. If we had not had the opportunity to refine
the theory of change through a formative evaluation, the summative
evaluation would not have included measurement of some activities
and outcomes now considered essential for program success. In
addition to supporting a stronger summative evaluation design,
collaboration around developing and refining the theory of change
helped PIQE to refine its program objectives and description and
to identify implementation areas that are crucial for program
success.
Comprehensive
Observations and Interviews
SRI conducted observations of six complete eight-week sessions
of the Parent Institute, as well as interviews of host school
principals and teachers, parent participants, and program facilitators.
SRI and PIQE staff met several times to discuss the findings and
the ways SRI could provide further feedback or support.
SRI also developed a draft set of benchmarks (milestones that
would indicate that the desired changes had occurred) to help
support PIQE staff in tracking their progress. PIQE staff reviewed
the draft benchmarks and helped refine them so they could be used
to manage their change process. Program staff have used the formative
evaluation findings and data collection tools in several ways.
For example, they have clarified lesson objectives, developed
sample lesson plans, revised program curriculum, created new facilitator
training materials, and revised the program operations manual.
Real-Time
Feedback
To examine the extent to which the program refinements were successfully
being implemented, PIQE staff asked SRI to conduct additional
“real-time” observations (provide feedback on each
class observed within a day or two of the observation so that
they could use the information to help other instructors who were
about to teach the same lesson). SRI provided real-time feedback
on 10 class sessions, each taught by a different instructor, during
the 2000-2001 school year. After each observation, SRI researchers
distributed detailed observation notes and held telephone debriefing
meetings with PIQE staff to discuss the notes and give immediate
verbal feedback. SRI produced a summary of the real-time feedback
with descriptions of what did and did not work in program implementation
for PIQE to use as an instructor training tool.
Real-time feedback allowed SRI to look at parts of the program
that PIQE wished to have examined in more detail and areas in
which PIQE was specifically trying to apply changes based on earlier
feedback. The real-time feedback was extremely useful, but it
would not have been possible or as useful without the earlier
comprehensive observations and interviews. The combination of
these two approaches worked extremely well.
The
Value of the Formative Evaluation Method
Formative evaluation is often overlooked when organizations believe
that their programs are already running according to plan or when
resources are limited. Our experience with the PIQE program, however,
suggests that formative evaluation is valuable for mature as well
as new programs. In addition, we found that our formative work
has built a stronger foundation for the summative evaluation scheduled
to begin in May 2002. We hope that our experience encourages others
involved in program evaluations to include formative research
methods in their evaluations whenever possible.
Shari Golan, Ph.D.
Dana Petersen, M.A., M.P.H.
http://www.gse.harvard.edu/~hfrp/eval/issue18/eval3.html |