Stage
1 - Establish Your Process and Create Indicators
The first stage of a district-wide technology evaluation project
incorporates the first four process steps of:
- Creation of an evaluation committee
- Development of evaluation priorities and related evaluation
questions.
- Creation of performance indicators for each evaluation question
- Organization of indicators into assessment rubrics
It is in this stage that the groundwork is set for effective program
evaluation. It is here that the conceptualization of the
work is lodged in a committee of project stakeholders, evaluation
questions are defined, and benchmarks/indicators are created.
Developing a Committee
A district-wide evaluation committee is the central, organizing,
structure of the technology evaluations we have facilitated in
most districts. As an initial step, districts create a technology
evaluation committee composed of stakeholders from throughout the
district and local community. This committee closely parallels
the structure of the district's technology planning committee in
that in order for the committee to be effective, it must be representative
of the interests and concerns of the broad district community.
In general, the committees we work with number about 12 to 15 members
and include district level staff, a board member, and principals,
as well as classroom teachers and technology specialists from all
grade levels.
Our Committee Composition worksheet
is useful in assigning and delineating committee membership. Notice
that the emphasis in committee composition is to represent the
full range of stakeholders to the district evaluation process.
Creating Evaluation Priorities and Questions
After working through the organizing questions exercise, the committee
is prepared to begin to frame the actual evaluation questions. Here
it is important to note that some evaluations only cover specific
parts of a district's technology implementation efforts. For
example, a district may choose to evaluate just its technology
professional development efforts. Another district may choose
to focus on teacher productivity and uses of technology. Yet
another district may focus on the key question of technology's
impact on student achievement. The point is that the evaluation
effort must generate questions and answers that are relevant to
the particular district conducting the evaluation.
Most often, we have found that committees choose to base their
evaluation questions in specific technology plan goals. This
method of developing questions rooted in planning goals addresses
the district's basic need to know if they are meeting their goals
for instructional technology and the various implementation process
steps.
Basic Points for Developing Evaluation Questions
Committees should work to insure that their evaluation questions
meet the following primary criteria:
- Questions relate to the district's desired outcomes for
student and/or teacher use of technology
Once again, we must note that it is not possible to evaluate
a program's success if you do not have an image of what success
looks like. A district must have defined their desired
outcomes for technology before measuring the progress toward
meeting those outcomes. Most often, these outcomes are
part of the district's technology planning goals; but in the
event that those goals are not sufficiently related to technology's
use, then the district will first have to work on defining goals
for technology's impact.
- Questions focus on assessing the impact of specific components
of a district's technology implementation effort
A successful technology implementation effort has a number of
components such as curriculum, professional development, and
infrastructure. As the point is always made in technology
planning, all of these components must work in concert
in order to achieve success. Therefore, when assessing
district technology progress, it pays to consider the impact
and effect of individual key components of the overall effort. In
most cases, a single question of "How is technology impacting
student achievement" is simply too broad to have a meaningful
answer. On the other hand, a question that asks (for example)
if district professional development is providing teachers new
skills in the application of technology is more answerable. Then,
a related question might be to determine if teachers are using
their skills in the classroom. Finally, a third question
could be to determine if students are actually using technology
in the ways that teachers support. This chain of component
questions allows for more focused data collection and easier
analysis of possible breaks in the chain of logic that supports
technology integration.
Developing Indicators
After developing evaluation questions, the committee's work should
turn toward creating performance indicators for each question.
This work is often accomplished by dividing the full committee
into subcommittees -- one for each evaluation question. Each
subcommittee can then work to develop indicators and ultimately
an indicator rubric for their question and then reviewed the work
of each of the other two subcommittees. The product of this
review is subsequently fed back into each group's development process. In
this way every committee member is able to review each of the questions/indicator
rubrics and this review benefits the work of the whole committee.
If you are developing an evaluation for a particular program -
such as a grant-funded project - then you might want to create
a so-called "logic map" to map out the project's goals, objectives,
actions, and indicators of success. To help in this work, we've
created an Inspiration-based logic map template and
have used the template to create a sample
logic map (please note that you will need Inspiration
software to view/use either of these files). For
more information creating logic maps, see our logic
map creation resource.
For evaluations that seek to broadly assess the impact of technology
on teaching and learning (e.g,. evaluations of a district's technology
plan implementation), we have developed a worksheet to
help subcommittees frame general indicators of technology's impact
on teaching and learning. The categories we have used on the worksheet
are simply representative of the most common types of evaluation
questions. We also have provided some basic tips for developing
rubrics that organize indicators.
In the end, you want to develop a rubric that details multiple
(we suggest four) levels of success for meeting your developed
indicator. Our indicator rubric template is
designed to help format your work. We've also included a sample of
what a rubric should look like when complete.
Some Tips for Creating Effective Rubrics
A rubric must be found in, and built from a picture of excellence to
establish a valid target and anchor for scoring. This is
true whether or not the work you are scoring ranks at exemplary
levels.
- Establish a first draft of the possible criteria to be used
in scoring the work
- Decide which of the criteria are most important for the purpose
and nature of this particular assessment vs. the feasibility
of using those criteria or that many criteria.
- Decide whether there will be one holistic rubric or various
analytic rubrics for each of the priority criteria
- Begin by trying to build a 4 point or 6 point rubric, regardless
of how many points on a scale you want the rubric to eventually
have.
- Though your rubric(s) should eventually minimize the use of
comparative and evaluative language, begin at first to sketch
out the rubric language for each point on the scale by using
words like excellent/good/fair/poor so as to set the right tone
for each point on the scale.
- Always build your rubric from the top, starting from a description
of exemplary performance.
- Carefully craft the language of the cut point on the scale,
so as to make clear what the performer must do for minimal success.
Resist the urge to describe the lowest points on the scale in
negative terms only. (What is accomplished at even the lowest
levels?)
- Edit and amplify your rubric, based on feedback from use and
peer review.
Next Steps?
Now that you have your rubric(s), it's time to move on to Stage
2, Data Collection.
|
