set 3, 2005 59
Validity and reliability are t wo key ideas in
assessment. In the last issue of set I looked at
the concept of validity and how it might inform
the assessment decisions we make as classroom
practitioners and school leaders. In this article
I address the issue of reliability, and how it too
can help inform our assessment strategies and
practices.
What is reliability?
Reliability refers to the consistency of t he
results we obtain from an assessment. This
may mean:
• Consistency across time—would the results
have been the same if the test or assessment
had take n plac e on anot her day, or at
another time?
• Consistency across tasks —would the result
have been the same if other tasks had been
chosen to assess the learning?
• Consistenc y across markers—would the
results have been similar if another marker
had scored the assessment?
The higher the level of consistency, the more
reliable are the results. No results, however, can
be completely reliable. There is always some
random variation that affects the assessment.
Chase (1974) helps us understand reliability
by using an an alog y fr om ev er yday li fe.
When we measure the length of a room, the
consistency of the results we get will var y
depending on what instrument we use to take
the measurement. For instance, a conventional
metre ruler will give us much more consistent
results than an elastic tape measure. The ruler
is rigid and stable; however many times we use
it to measure the room, there will be a high
degree of agreement from one measurement to
the next. The elastic tape measure, on the other
hand, will need to be stretched just the right
amount to show an exact metre, and so will
produce much less consistent mea surements.
Sometimes we will stretch the tape too far and
end up underestimating the length; sometimes
we will not stretch the tape far enough, and
overest imate it. The elasticity of the tape
measure introduces an element of random
variation into our measuring process that will
make our measurements less consistent, and
so less reliable.
When the resu lts of an assessment are
reliable, we can be confident that repeated or
equivalent assessments will provide consistent
results. This puts us in a better position to make
generalised statements about a student’s level
of achievement, which is especially important
when we are using the results of an assessment
to make decisions about teaching and learning,
or when we are reporting back to students and
their parents or caregivers.1
Determining reliability
Deter mi ni ng reliabi lity has trad ition al ly
been seen a s a statistica l exercise. It usually
involve s calculating a reliabilit y coefficient
to indicate how well assessment results agree
over repeate d uses of the asse ssment tool.
Reliability coefficients vary between zero and
one, with zero indicating no agreement and
one, total agreement (a result that is never
actually obtained in educational assessment).
Test developers use several methodologies to
calculate reliability coefficients, depending on
the type of consistency they are interested in.
Some of these are briefly described below.
Reliability coefficients on standardised tests
are often greater than 0.9, indicating a high
degree of reliability. The question of how high
the reliability for a set of assessment results
should be, however, depends very much on
what level of decision making will be based on
our assessment results. When we are dealing
with assessments that are highly significant we
need high reliability. When the decisions do
not have lasting consequences, a high reliability
measure is not as important.
The Standard Error of
Measurement
The reliability coefficient for a set of test results
is sometimes used to calculate what is called
the standard error of measurement (SEm). This
can be used to describe a band of achievement
with in which we c an be reasonably sure a
student’s true level of achievement actually
lies. A true level of achievement can be thought
of as the level that would be achieved by the
student if the test was perfectly reliable. For
example, when a score is reported as 30 with a
SEm of 3, we can be reasonably confident that
the true achievement level of the individual
would fall somewhere in the range 27 to 33.
In educational testing it is considered good
practice to report the SEm. In the Progressive
Achievement Test of Reading, for instance,
the manual writers provide an estimate of the
SEm for each individual test (which is usually
around 3 marks). Being aware of the SEm helps
us regard a student’s test score as describing a
range rather than a precise point.
Reliability in the classroom
It is un lik ely th at te ach ers wi ll sp end
time calculating reli abi lit y coef ficients or
reporting sta ndard errors of measurement
for the assessment tasks or tests they develop
A hitchhiker’s guide to reliability
Charles Darr
Table 1 METHODOLOGIES FOR DETERMINING RELIABILITY
Method Proces s
Test/re test The same test, or an equivalent version, is administered at two different times to
the same group of students. The two sets of results are then compared to calculate
the reliability coefficient. This method provides an indication of how consistent the
results are over time or between equivalent forms of the same test.
Internal consiste ncy The results on different tasks or sections of an assessment are compared to
see how well they r elate. Several different methods can be used, including
dividing the test into t wo halves and comparing the results on each half ( split-
half method). O ther instances of this type of reliability coefficient involve more
sophisticated statistical methods. In test manuals it is common to see what
is called Cronbach’s alpha, or the use of Kuder Richardson formula 20 or 21.
Calculating this type of reliability coefficient provides an indication of how
consistently the items or tasks within an assessment promote the same result.
Inter-ra ter reliabilit y Results from different markers can be compared to ascertain the level of
agreement. This method used to show how consistently two or more assessors
are scoring the same tasks, is called moderation when it is used in the context of
assessment for qualifications.
