Agenda for weeks of Aug 17, Aug 24 and Sept 01 , 2020
[updated August 17, 2020 --please notify JH if you encounter any glitches]
Topic: The Quality of Measurements and the Effects of Measurement Error.
Preamble: This topic of measurement is probably new for you, as it was for JH
when he began in cancer clinical trials in 1973, and oncologists (cancer doctors)
were judging responses of advanced cancer to chemotherapy
by measuring tumours by
'palpation'.
Re Q1 and Q2: In early measurement courses JH gave, students measured readability manually
by counting the lengths of words and
sentences, and the number of syllables in words.
Early on in bios601, the task became much easier using online (and online textxs) and online tools, or those in Microsoft Word.
From 3 measurements of readability, they calculated the standard error of measurement as the SD of the 3.
The CV is the SD divided by the mean of the 3, expressed as a percentage.
standard error of measurementsince everyone knows what 0.7 of a grade is.
But if the scale is arbitrary (running from say 0 to 70) a SEE of 9 'points' is more difficult to judge,
unless one knows well what a '40' or a '20' is. In this case the ICC is more useful,
but it requires that one has > 1 measurement each on each of several texts of
different difficulty .. so one can judge how much is genuine 'between-text'
and within-text' variation.
For validity students went to the library (or looked around at home, or online) for books of
different difficulty so that we could see if the measurements agreed well with
what difficulty experts though the difficulty of each book was. It was not like
the study in Q16 where 500 meant 500 or 1500 meant 1550 and all would agree on
this 'gold standard'.
Unlike in physical measurements, this issue of
an independent gold standard' is a challenging one in psychometric measurement.
It's not like you can order 'a grade 6' book from the US
National Institute of Standards and Technology (NIST)
the way you can order a substance with a known cholesterol concentration, or a 1Kg weight.
One strategy we used more recently was to look online for lists of
books recommended by
teachers for children in different grades, and
our job was made easier if we could find
the texts themselves online, and simply cut and paste
samples of them into MS Word or an online 'readability'
tool. Some years,
we used the full range from 'the Cat in the Hat' to
university texts,
and plotted the measurements against the grade or age level.
If you measure a text with different instruments or tools, and if they have a
common scale (e.g. grade level) then you could use
a linear model to estimate how systematically (if any) they vary from one to another.
If you think of these tools as the only ones available (like iPhone vs Android)
then you should treat them as 'fixed' effects.
If one the other hand they are a sample of the many tools 'out there' then a
random
effects model might be more appropriate.
Even though we will use examples involving the measurement of
physical quantities such as activity and biochemical parameters,
JH kept the readability example in Q1 and Q2 as a reminder of the early psychometric
focus on the quality of measurements.
Week of Aug 17 (see NOTE)
Send JH your answers to however much/little you are able to do of
Q3, Q4, Q5, Q6, Q7[PhD]
by Sun August 23.
Q3 [ 'm-s' is short for 'math-stat' ].
The point of asking you to derive the link
is to emphasize that the
Standard Error of Measurement and R are not ENTIRELY
separate. Yes, the Standard Error of Measurement is more limited, because it does
not tell you how much variation there is from
person to person (or object to object). But you
can
think of R or the ICC as the proportion of the
OBSERVED variance that is 'real' (i.e. due to
genuine person to person variation),
and think of the remainder, 1 - R,
as the square of the Standard Error of Measurement.
(1) The square of the Standard Error of Measurement is ONE of the TWO
components in the observed variance. (2) The
square of the genuine between-person variance
is the other. The ICC is (2) as a fraction
of (1)+(2).
A good example of the 2 concepts
can be found in the blurb from the Educational
Testing Service called
INTERPRETING YOUR GRE SCORES, page 8 of the Notes.
Q4
Relationship between test-retest correlation
and ICC(X). The point here is to see the same
concept from two different perspectives.
Q5
Relationship between correlation(X,X')
and ICC(X): Some people like this explanation
of the ICC, since it echoes what was said above
about the ICC as the proportion of the variance we observe
in an imperfectly measured characteristic that is 'real'.
Think of a correlation as another way to measure
how strongly an imperfectly measured characteristic
correlates (agrees) with the perfectly version.
If you were trying to explain the ICC to a lay person, you
would probably have better success using 'correlation'
than 'variance'. To explain correlation you don't
have to get into as many details as you would have to if
you take the 'variance' route. If we are willing
to cheat a little bit (and tell people that
the SD is like the typical or average absolute deviation),
you might get away with using the concept of a SD,
but the concept of a typical or mean squared
deviation will for sure lose more people.
Q6
Galton's data:
Have a look at the Family album
respondents used to report
the family heights.
Who, if anyone, did the measuring? Who
did the 'reporting'?
Do you know how tall your parents and grandparents
are (were?)
Q7
'Increasing Reliability by averaging several
measurements'
This is a very topical and 'charged' issue at funding agencies, such as
the Canadian Institues of Health Research, where each
grant application used to be reviewed by 2 primary reviewers, and then
an average is made of the scores of up to 20 panel members (incl. the 2)
who heard and discussed the
2 reviews, and had also looked through the application themselves.
The new system uses 5 reviewers who do not meet/communicate, and
their scores are averaged.
If in the old system, where the ICC was say 0.4, what would be the ICC
if we used the average of 2 raters? 3 raters? 4 raters?
You can manipulate the algebra as you wish, but you might also
think of it as follows:
if we average m raters, the true sigma-sq-between is not affected,
but the true sigma-sq-within now gets reduced from
sigma-sq-within / 1 if 1 random rater, to sigma-sq-within / 2 if we average 2,
... sigma-sq-within / m if we average the scores of m raters.
So with an average of m raters, the observed variance of these averages is now
sigma-sq-between + sigma-sq-within / m
The fraction that this that is signal is
sigma-sq-between / [ sigma-sq-between + sigma-sq-within / m]
SO what the question is asking is what if we use N*m raters
so we have fractions
sigma-sq-between / [ sigma-sq-between + sigma-sq-within / N*m]
and
sigma-sq-between / [ sigma-sq-between + sigma-sq-within / 1*m]
The algebra is a matter of manipulation this ratio, so as ro remove the
'm' that is there to start with, and end with the basic ICC[1]
(ie what if m=1) and the scaling factor N.
Another example, if a 3 hour GRE exam, done by a paper and pencil, has
a reliability of 0.9, what reliability would a 6-hour or 12-hour exam have?
Taking 3 hours as the unit of effort, it is
0.9/ (0.9 + 0.1 ) for 3 hours
0.9/ (0.9 + 0.1/2) for 6 hours
0.9/ (0.9 + 0.1/4) for 12 hours
etc.
Geoff Norman was part of a group who developed
McMaster's 'Multiple Mini Interview' system.
McMaster, and many other schools since then have
abandoned the traditional interview
and use this instead
see Med Educ. 2004 Mar;38(3):314-26.
An admissions OSCE: the multiple mini-interview.
Eva KW, Rosenfeld J, Reiter HI, Norman GR.
and subsequent publications
that evaluated its measurement properties.
Week of Aug 24 (see NOTE)
Send JH your answers to however much/little you are able to do of
Q8, Q9, Q17 parts a and b, Q20
by Sun August 30.
Q8
Just because (random) measurement errors tend to cancel out in
averages doesn't mean that errors in measurement can be ignored. For example,
how comfortable would you be in measuring how much physical activity JH does
by having him wear a 'step-counter' for a randomly selected week of the
year, and using that 1-week
measurement as an 'x' in a multiple or logistic or Cox
regression? See slides 7 and 8 from part of JH's
"Scientific reasoning, statistical thinking, measurement issues, and use of
graphics: examples from research on children"
at Royal Children's Hospital in Melbourne, earlier this year.
pdf
Some of the the terminology will be new to you, and so (as you will discover
when you do run the simulations in Q8 of how well you can estimate the conversion factors between
degrees F and degrees C) will some of the consequences of measurement error.
The "animation (in R) of effects of errors in X on slope of Y on X" might be of interest,
as might the java applet accompanying "Random measurement error
and regression dilution bias".
These consequences are rarely touched on, yet alone emphasized, in theoretical courses on regression, where all
'x' values are assumed to be measured without error! Welcome to the REAL world.
Q9
The point is to 'smooth' the decay curve.
But (as the hint says) its form should
not be a big surprise : it was the
subject of a question earlier on in the math-stat.
questions.
Q17
* Measuring Heart Rate is more challenging
Link
Link
* Measuring Environmental Noise:
Link
* Measuring Physical Activity: interpreting the statistical REPORT
is challenging!
Link
Q20
!!
Week of Sept 1
Send JH your answers to however much you are able to do of
[Q3, Q4, Q5, Q6, Q8, Q9, Q17 parts a and b], and Q21, Q23[PhD], Q24[PhD], Q25, Q26, Q27
by
Saturday September 5.
(If you have already send JH your work on [Q3-Q17], then unless he asked you to resubmit it,
no need to send it again.
But it may be easier to attempt the later ones if you have already gone through Q3, Q4, Q5, Q6, Q8, Q9, Q17 parts a and b).
NOTE: JH doesn't
like to talk about what gets counted in grades, etc, but in case any students ask:
he will only 'count' your
effort with Q21, Q23[PhD], Q24[PhD], Q25, Q26, Q27. He realizes that some
of you have other commitments/plans for the last 2 weeks of August and wants you to put these first, especially if they involving
re-recharging batteries!
Q21
A real example to convince you that
MEASUREMENT ERROR MATTERS.
By the way, section 4.7.4 Measurement error in regression,
of the Cox-Donnelly book cited in question 21 gives a nice visual
(rather than algebraic) explanation for the flattening of the slope.
Its the same visual explanation that JH's co-authors give
in their BMJ tutorial, here
J Hutcheon, A Chiolero and J Hanley Random measurement error and regression dilution bias
The AIDS example in Cox and Donnelly is a very nice illustration of how measurement errors
led to a delay in figuring out how HIV was transmitted. This re-inforces the message
that measurement errors can cause quite big but
subtle distortions.
This Q, new in 2019, is meant to show that the
dilutions you have calculated/seen are not just in toy math examples,
but also present in just about all research involving regression -- since it
is near impossible to measure all X's perfectly.
Clearly, some authors like the ICC, while others make it a bit more complicated
by using a ratio of the 2 components in the ICC. In the end, it comes to the same thing.
JH prefers the ICC version, since it is more easily remembered.
Q23
Unlike the ICC or R, or the variance-ratio used in Q21, the magnitude of the errors in this Q,
new in 2020, is not naturally quantified by a
variance, or a fraction of the overall variance. But the effect
is actually easier to explain to a lay person than the variance-based attenuations we have focused on.
The 1856 publication is not well known, and the description of the effects of the errors in the addresses is even less well known.
Thus, a short note to a modern day Epidemiology journal, explaining
this early attention to measurement , and recounting Snow's clear description of the consequences of errors,
would make a
nice contribution to the teaching of epidemiology and its history!
JH is hoping that your answers to this Q can be the start of such a note... authored by the class!
Q24
Will be interested to read your suggestions.
Q25
A hand-drawn version of the diagram will be fine.
Q26
No need to go beyond a 2 point X, or a 2-point errors. If you understand this case, you understand the more general case.