Establishing
Causal Links
CAUSAL STUDIES
The search for causal explanations is of
central importance in every area of scientific research. The first step in
understanding something often involves speculating about what its cause or
causes might be and then finding a way to test those speculations. In the last
few years, for example, there has been a dramatic increase in the number of
American children who are obese. What factors might be responsible for this
increase? Too much fast food? Too little exercise? Some other factor? Some
combination of factors? Causal experiments or, as they are usually called,
causal studies are the main tool by which researchers confront such questions.
The design of causal studies and the assessment of their
results present researchers with a series of problems over and above those we
have encountered in thinking about how to design an experiment. The first stems
from the fact that causation is not an all-or-nothing matter. Not every subject
exposed to a cause will necessarily yield the effect. To establish a causal link,
what must be shown is that significantly different levels of the effect
obtained in those exposed and not exposed to the cause. And this can be a
problem. Causal studies generally involve limited number of subjects. How can
researchers be sure that a difference in levels of effect in their study groups
is not due to the fact that relatively large differences often occur by chance
in relatively small groups? How can they be sure, that is, that whatever effect
a study uncovers is due to the suspected cause, and not random chance?
Second, effects are rarely associated with a single causal
factor. How, then, can researchers be sure that other causal factors have not
influenced the results? This problem is all the more severe because
experimental and control subjects will usually be drawn from a population that
may have been exposed to these other factors.
80
Finally, causal research cannot always be undertaken in a way
that conforms to the model of a tightly controlled experiment outlined in
Chapter 4. Researchers cannot, for example, investigate the possible link
between diet and obesity in children by exposing young children to unhealthy
food. Some other way must be found to test for this link.
Causal studies typically investigate the impact of suspected
causes on the members of large populations, as in the example above. Often they
will deal with issues of some considerable practical importance, like the
problem of obesity in American children. For this reason, causal studies are
the most widely covered of all science stories in the popular media. Near the
end of the chapter we will turn out attention to the ways in which causal
studies are handled in the mass media. Unfortunately, media coverage of causal
research leaves a lot to be desired. Facts needed to make sense of a study will
often be omitted and study outcomes oversimplified. And more often than not,
these shortcomings result from a failure to appreciate the issues we will
consider in this chapter.
RULING OUT CHANCE
In 2001 a study was undertaken to determine
whether St. John’s wort, a popular herbal remedy, can counter the effects of
depression. The study involved 200 women, most in their early 40s, who had
suffered from major depression for two years, though none were classified as
being severely depressed. The women were assigned at random to receive either
an extract of St. John’s wort or a placebo. Those taking the St. John’s wort
began with three tablets a day, each containing a commonly recommended dosage.
If they did not improve after four weeks, the dose was increased to four
tablets. After eight weeks, the patients were evaluated by psychiatrists who
did not know whether the patients were in the experimental or control groups.
Twenty-seven percent of those who took St John’s wort showed marked improvement
compared with 19% in the placebo group. Is this difference large enough to show
that St. John’s wort is an effective treatment for depression?
Before we can answer this question, we need to know something
about the thinking involved in estimating the accuracy of samples taken at
random from large populations. This is because the subjects in a causal study
are a sample. In the St. John’s wort study, the 200 hundred subjects had been
selected from the larger population composed of women suffering from
depression. And the question at issue is whether, based on the study’s findings
about this sample, we can conclude that St. John’s wort counters depression in
the larger population. Or is it that the difference reported in the study is
nothing more than the kind of chance variation often found in small samples?
Statisticians have been able to determine that the accuracy
of a sample is, to a large extent, a function of sample size. The larger a
sample, the greater the chances it will accurately mirror what is true of the
population from which it was taken. This is a rough approximation of something
statisticians call the law of large
numbers. Flip a fair coin 10 times, and it should come down “heads” half
the time. But in a series of 10 flips, chances are not all that bad that “heads”
will come up three or four times or perhaps six or seven times. Now flip the
same coin 1,000 times. Chances are very high that the coin will come down
“heads” somewhere in the neighborhood of 500 times; chances are slim that we
will get a result very much higher or lower than 500. (To see why, imagine what
the chances would be of flipping a coin 1,000 times and getting only one or two
“heads” as opposed to flipping it 10 times with the same result.)
This fact about samples accounts for a notion that is
indispensable in estimating the chances that a sample is accurate—margin of error. Imagine that a poll has
just been taken of registered voters in your state. Five hundred voters were
selected at random, telephoned, and asked if they intend to vote in the
upcoming election. 52% said yes. In reporting this result, the pollsters will
probably say something like “this sample has a margin of error of +/− (plus or
minus) 4%.” As a general rule, what this means is that if a sample of this size
(500) were taken 20 times, in 19 of the 20 samples − 95% of the samples − the
outcome would be within 4%, one way or the other, of the result obtained in the
sample actually taken. So, in the case of our poll, there is a 95% chance that
between 48% and 56% (52% +/− our 4% margin of error) of all registered voters
will vote in the upcoming election.
Suppose instead that the sample had involved 1,500 voters
with the same results. Remember, the larger a random sample, the greater the
chances its results willbe accurate.What this meansisthat as
samplesizeincreases,themargin oferror decreases. For a sample of 1,500, the
margin of error is about +/− 2%. Chances are 19 out of 20 that something
between 50% and 54% of the registered voters will turn out in the next
election.
Table 5.1 gives the margins of error for a number of common
sample sizes. In each case, there is a 95% chance that the sample outcome will
reflect that of the population from which it was taken. Our choice of the 95%
confidence level is somewhat, though not entirely, arbitrary. For example, we
could just as easily have given margins of error at a lower confidence level,
say, the 80% level. As you might suspect, the margins of error for this lower
confidence level would be smaller. Ifwe are willing to tolerate a greater
chance that we are wrong, we can venture a more accurate estimate. Table 5.1
tells us that in a sample of 100, for example, we can be 95% sure that the
outcome will be within +/−10% of the sample outcome. Were we to restrict
ourselves to a conclusion we could be 80% sure would be true, the margin of
error would shrink to about 7% either way. However, most sampling is based on
the 95% confidence level. Unless we have information to the contrary, it is a
safe bet that a reported margin of error is at this level.
To assess the results of a causal study, we will need to make
use of what we have found out about margin of error. As we noted earlier,
experimental and control groups are, in a sense, samples. The 200 women in the
St. John’s wort study are representative of the larger population of all people
who fit the profile of the study subjects: women in their 40s who suffer from
moderate levels of depression. (Unless there is good reason to think this
particular group is unusually susceptible to bouts of depression, the
population can perhaps be
T A B L E 5.1
Sample Size Approximate Margin of
Error(%)*
25 +/−22 50 +/−14 100 +/−10 250 +/−6
500 +/−4
1000 +/−3
1500 +/−2.5
2000 +/−2
*The
interval surrounding the actual sample outcome containing 95% of all possible
sample outcomes.
expanded to include all people who suffer
moderate depression.) So the issue we must resolve is whether the difference in
the outcomes for samples that make up the experimental and control group are
large enough to indicate a causal link and not just the kind of random
statistical variation associated with sampling.
In the St. John’s wort study, 27% of 100 experimental
subjects improved as did 19% of the control subjects. Is this difference
enough? Look back to Table 5.1. The margin of error for samples of 100 is about
+/− 10%. This tells us there is a 95% chance that in the population from which
the sample was taken, somewhere between 17% and 37% could be expected to
improve. In the population corresponding to the control group, somewhere
between 9% and 29% would improve. Figure 5.1 shows that there is considerable
overlap between these two intervals. This tells us is that chances are quite
high that the difference we have discovered is due to random statistical
fluctuations in the sampling process. This result does not mean that there is
no link between the suspected causal agent and the effect we are testing. It is
entirely possible that a causal link exists but that the level of effect is too
small to measure using
|
Control Group |
|
Experimental Group |
|
9% |
29% |
|
|
17% |
37% |
F I G U R E 5.1
groups of this size. What we can conclude,
however, is that this particular experiment has not conclusively established
such a link. Were the difference between levels of effect in our two groups to
have been 20% or more, we would have concluded that the difference is due to
something other than the random statistical fluctuations associated with
sampling. Quite possibly, it is due to the fact that St. John’s wort can
prevent depression.
Interestingly enough, the difference in levels of effect
found in the St. John’s wort study would have been sufficient to suggest a
causal link if the experimental and control groups had been larger! Imagine if
the study had been carried out on groups containing 1,000 subjects each. Table
5.1 tells us the margin of error for groups of this size is +/− 3%. The
corresponding intervals are represented in Figure 5.2. Note there is a clear
gap between the two intervals. It is not unusual for causal researchers to
expand study sizes when their initial evidence for a causal link is tentative.
A borderline result may become much less ambiguous if it can be shown to
persist as larger samples are investigated. Nor is it unusual for researchers
to combine the results of several small studies in an attempt to create groups
of sufficient size to suggest a link that may be less clearly indicated in any
of the individual studies. This approach is called meta-analysis. The findings of any such analysis are at best
tentative, since their experimental and control groups come from many studies,
and thus are likely to minimize differences in the studies over which they
range.
Causal experiments do not always involve experimental and
control groups of the same size. Even where the groups differ in size, we set
minimal levels of difference in much the same way. Suppose, for example, that
we have an experimental group of 50 subjects and a control group of 100. In
constructing our intervals we need only make sure to work with the proper
margins of error, which will be different in each case. Since we are working
with percentages, we should encounter no difficulty in comparing the intervals.
When the results of causal experiments are reported,
researchers often speak of differences that are or are not statistically significant. A difference in the outcome of two
samples will be statistically significant when there is little or no overlap
between the confidence intervals for the experimental and control groups. Thus
a difference that is statistically significant is one which is highly unlikely
to be due to normal sample fluctuations; chances are slim that two groups,
chosen at random, would accidentally differ by the amount we observed in our
experiment. Conversely, a result that is not statistically significant suggests
there is a great deal of overlap and that the observed difference in levels of
effect may well be due to random sample fluctuations.
As we have noted, causal research is nearly always conducted
at the 95% confidence level. A statistically significant result, then, is one
that would occur by chance only one time in twenty. In working with the results
of a study done at this confidence level, Table 5.1 can help us to decide
whether a result is or is not statistically significant. But a note of caution
is in order here. The intervals in Table 5.1 can give us a rough approximation
of whether a difference in experimental and control group outcomes is
significant. However, they are a bit off. The percentage difference required to
achieve statistical significance is a bit less than the difference suggested by
Table 5.1. For example, a difference of just over 13% will be statistically
significant for groups of 100 or so. (The required differences decrease even
more when levels of the effect are very near to 0% or 100%.) Table 5.1 suggests
that a 20% difference would be required. The amount of overestimation in Table
5.1 decreases as the size of experimental and control groups increases. Table
5.1 suggests a 6% difference is required to achieve statistical significance
for samples of about 1,000, when in fact just over a 4% difference will do the
trick. We can correct for the inaccuracy in Table 5.1 if we adopt the following
rules of thumb in working with reported differences between experimental and
control groups:
1. If
there is no overlap in the intervals for the two, the difference is
statistically significant.
2. If
there is some overlap in the intervals (in the intervals have less than
onethird of their values in common), the difference is probably statistically
significant. The greater the overlap, the smaller the chances the difference is
significant.
3. If
there is a good deal of overlap (more than one-third of all values), the
difference is probably not statistically significant.
In the jargon of the causal researcher, failure to establish
a causal link is often called a failure to reject the null hypothesis. The null hypothesis is simply the claim that there
is no difference between levels of effect in the real populations from which
the samples were taken. An experiment that succeeds in establishing a large
enough difference in levels of effect between experimental and control groups
will often be said to reject the null hypothesis. But in the study of St.
John’s wort, such a difference has not been observed. On the basis of the
study, in other words, we cannot reject the null hypothesis.
Imagine now that we are about to design a causal study. Does
A cause B in Cs? We select a number of Cs, assign them at random into
experimental and control groups, and administer A to the experimental subjects
and a placebo to the controls. How large of a difference in levels of B should
we expect to find at the end of the study if there is a causal link? The answer
now should be clear. Whatever it takes to allow us to reject the null
hypothesis for samples of the size with which we are working.
MULTIPLE CAUSAL FACTORS
As a veteran teacher with years of
experiences observing students, I’m convinced that students who attend class
regularly generally do better on tests that do those who attend sporadically.
But then personal observation can be misleading. Maybe I have just remembered
those good test takers who always came to class, since I would like to think my
teaching makes some difference. Is there really a causal link between my
teaching and the performance of my students? We can determine this by doing a
test. I will teach two courses in the same subject next semester, each
containing 100 students. The only difference between the two courses will be
that in the first, attendance will be mandatory, while in the second it will be
voluntary. All material to be tested will be covered either in the textbook or
in lecture notes to be supplied to all students. Course grades will be based on
a single, comprehensive final exam given to all students in both courses.
Suppose now that we have performed this experiment, and at the end of the term
we discover a statistically significant difference between the test scores of
the two groups. The experimental group, the group required to attend, scored
much higher, on average, than the control group, most of whom took advantage of
the attendance policy and rarely appeared in class. To ensure accuracy we have
excluded the five highest and lowest scores from each group and the average
difference remains statistically significant.
Despite the care we have taken in designing our experiment,
it nonetheless suffers from a number of shortcomings. Perhaps the most obvious
is the fact that it involves no control of factors other than attendance which
might influence test scores. One such factor, obviously, is the amount of time
that each subject studies outside of class. Remember, tests were based solely
on material available to all subjects. What if a much higher percentage of the
subjects in the experimental group than the control group spent considerable
time preparing for the final? If this is the case, we would expect the
experimental group to do better on the final but for reasons having little to
do with class attendance.
The way to avoid this sort of difficulty is by matching
within the experimental and control groups for factors, other than the
suspected cause, which may contribute to the level of the effect. Matching
involves manipulating subjects in an attempt to ensure that all factors that
may contribute to the effect are equally represented in the two groups. There
are several ways of matching. One is simply to make sure that all other
contributing factors are equally represented within both groups. This we might
accomplish in our experiment by interviewing the students beforehand to
determine the number of hours on average studied per week. Presuming we can
find an accurate way of getting this information, we can then disqualify
students from one or the other of our groups until we have equal numbers of
good, average, and poor studiers in both groups. Another way of matching is to
eliminate all subjects who exhibit a causal factor other than that for which we
are testing. Suppose we discover that a few students in each group are
repeating the course. We might want to remove them altogether from our study.
The final way to match is to include only subjects who
exhibit other possible causal factors. We might do this by restricting our
study to students, all of whom study roughly the same amount each week. If all
of our experimental and control subjects have additional factors which
contribute to the effect in question, the factor for which we are testing
should increase the level of the effect in the experimental group, provided
that it is actually a causal factor. Matching in this last way can be
problematic if there is any chance that the effect may be caused by a
combination of factors. Thus we may end up with an experiment which suggests
that A causes B in Cs when, in point of fact, it is A in combination with some
other factor which causes B in Cs.
By matching within our two groups we can frequently account
for causal factors other than the factor we are investigating. However, there
is a way that unwanted causal factors can creep into an experiment which
matching will not prevent. We must be on guard against the possibility that our
subjects will themselves determine whether they are experimental or control
subjects. Imagine, for example, a student who has enrolled in the course that
requires attendance but then hears from a friend about the course that does not
require attendance. It seems at least likely that poor students will opt for
the course that requires less. Thus, we may find that poor students have a
better chance of ending up in the control section rather than in the
experimental section. We could, of course, control for this possibility by
making sure students do not know the attendance policy prior to enrolling and
by allowing no movement from course to course. Another problem we might have
here is that poor students in the experimental group, upon hearing of the
attendance policy, might drop out, again leaving us with an experimental group
not well matched to the control group. In any event, it is worth taking
whatever precautions are possible, in designing a causal experiment, to insure
that subjects have no way to influence the composition of the experimental and
control groups.
RANDOMIZED, PROSPECTIVE, AND RETROSPECTIVE CAUSAL
STUDIES
Randomized Causal
Studies. The causal studies we have discussed so far have several
things in common. Each began with a set of subjects who, prior to the study,
were very much alike. (In the second study—the one about student test
performance—matching was used to make sure both groups were similar in makeup.)
In particular, none had been exposed to the suspected causal agent. In both,
subjects were randomly assigned to experimental and control groups. Only then
were experimental subjects exposed to the suspected cause. Randomized studies, as studies of this sort are called, conform
nicely to the criteria for a decisive experiment discussed in Chapter 4. Any
pronounced differences in levels of effect that emerge over the course of
either study would be highly likely if there is a causal link and highly
unlikely otherwise.
The great advantage of randomized studies is that they are
capable of providing unequivocal evidence for a causal link. But they have
several disadvantages as well. First, they tend to be quite expensive,
particularly if it is necessary to work with large groups of subjects. Second,
unless the suspected effect follows reasonably soon after exposure to the
casual agent, randomized studies may take a great deal of time to carry out.
In 1981, a large-scale randomized study was begun to
determine whether low doses of aspirin could decrease the risk of myocardial
infarction (heart attack). The study was undertaken by a team of investigators
from Harvard Medical School and a Harvard teaching hospital. Over 260,000 male
U.S. physicians between the ages of 40 and 84 were invited to participate. Of
the nearly 60,000 who agreed to participate, about 26,000 were excluded because
they reported a history of heart problems. 33,223 willing and eligible
physicians were enrolled in a “run-in” phase in which all were given low-dose
aspirin. After 18 weeks, they were sent a questionnaire asking about their health
status, any side effects of the aspirin, compliance with study protocols, and
their willingness to continue in the trial. About 11,000 changed their mind,
reported a reason for exclusion or did not reliably take their aspirin. The
remaining 22,071 physicians were then randomly assigned to receive either an
aspirin or an aspirin placebo. Follow-up questionnaires were sent after six and
twelve months and then yearly. Though the study was scheduled to run until
1995, it was stopped after only six years because it was clear that aspirin had
a significant effect on the risk of a first myocardial infarction.
The details of this story attest to the amount of time,
effort, and expense that may be required to undertake an ambitious randomized
causal study. The selection of candidates and subsequent efforts at matching
alone took nearly two years. The study then took another six years to yield
decisive results. And the entire project required millions of dollars in
funding from four National Institutes of Health grants.
Time and expense are not the only impediments to randomized
studies. Many suspected causal links cannot be subjected to randomized testing
without putting their subject at undue risk. Do high rates of cholesterol in
the blood cause heart disease? Imagine what a randomized experiment might
involve. We might begin with a large number of young children. Having divided
them at random into two groups, we will train one group to eat and drink lots
of fatty, starchy, and generally unhealthy foods of the sort we suspect may be
associated with high levels of cholesterol. I’m sure you can see the problem.
Not coincidentally, much medical research is carried out on laboratory animals
precisely because we tend to have much less hesitation about administering potentially
hazardous substances to members of nonhuman species. But there is another
approach, one that allows a more humane use of human and animal subjects: work
with subjects who have been exposed to suspected causes and their effects. Such
studies fall into two broad categories, called prospective and retrospective
studies.
|
QUICK
REVIEW 5.1 Randomized
Studies |
|
Population from which experimental and control groups are drawn, not yet exposed to suspected cause.
Experimental Group Control
Group All are exposed to the suspected None
are exposed to cause the
suspected cause. |
Prospective Causal
Studies. Prospective studies begin with subjects who have already
been exposed to the causal agent under investigation. Over the term of the
study these experimental subjects are compared to control subjects who have not
been exposed to the suspected cause. One of the most well-known prospective
studies began in 1976 at Harvard Medical School. 122,000 nurses, aged 30 to 55,
agreed to fill out a questionnaire every two years about diseases and
heath-related topics including smoking, hormone use, and menopausal status.
More than 90% of the respondents still answer a questionnaire every other year,
providing information about what they eat, what medicines they take, what
illness they have had, and whether they drink, smoke, exercise, or take
vitamins, among other things. At various points in the study, blood samples
have been obtained by sending a request and a few collection supplies to the
nurses, something that could never have been done with the general public. Over
the years researchers have been able to identify risk factors for diseases such
as breast, colon, and lung cancer, diabetes, and heart disease. Among other
things, researchers have discovered that those nurses who regularly take
vitamin E have significantly lower rates of heart disease and those who have
undergone hormone replacement therapy have significantly higher rates of breast
cancer. As the study subjects age, researchers hope to be able to determine the
extent to which diet and exercise lead to a longer, healthier life.
Even the best of prospective studies run up against one major
stumbling block. At the onset of any prospective study, subjects in the
experimental group have already been exposed to the suspected causal factor but
as we have found, most things can be caused by a variety of factors. It is
always possible that other causal factors are responsible for some part of the
effect in both the experimental and control groups. For example, in the study
above, nurses who took vitamin E had lower rates of heart disease. But it may
be that those who take vitamin E tend to take good care of themselves by
exercising, watching their weight, and eating a healthy diet, all factors that
contribute to heart disease. And this is the problem. By concentrating on a
single causal factor in the selection process, we leave open the possibility
that whatever difference in levels of effect we observe in our two groups may
be due in part to other factors. This, of course, is precisely where
prospective studies differ from randomized studies. By randomly dividing
subjects into experimental and control groups prior to administering the
suspected cause, we greatly decrease the chances that other factors will account
for differences in level of effect. In prospective studies it is always
possible that other factors will come into play precisely because we begin with
subjects who may have been exposed to other causal factors.
Matching can be used to control for extraneous causal factors
in prospective studies. Suppose we find, in the long-term nurses study, that
about 30% more experimental than control subjects exercise regularly. We can
easily subtract some subjects from our experimental group or add some to the control
group to achieve similar percentages of this obvious causal factor. It is not
an oversimplification to say that the reliability of a prospective study is in
direct proportion to the degree to which such matching is successful. Thus, in
assessing the results of a prospective study, we need to know what factors have
been controlled for via matching. In addition, it is always wise to be on the
lookout for other factors that might influence the study’s outcome yet which
have not been controlled for. In general, a properly done prospective study can
provide some strong indication of a causal link, though not as strong as that
provided by a randomized study.
In some respects prospective studies offer advantages over
randomized causal studies. For one thing, prospective studies require much less
direct manipulation of experimental subjects and thus tend to be easier and
less expensive to carry out. Their principle advantage, however, lies in the
fact that they can involve very large groups, as in the doctors and nurses
studies discussed above. Causal factors often result in differences in level of
effect that are so small as to require large samples to detect reliably.
Moreover, greater size alone increases the chances that the samples will be
representative with respect to other causal factors. This is crucial when an
effect is associated with several causal factors. If a number of factors cause
B in Cs, we increase our chances of accurately representing the levels of these
other factors in our two groups as we increase their size. In addition,
prospective studies allow us to study potential causal links we would not want
to investigate in randomized studies. For example, the nurses study discussed
above uncovered a link between hormone replacement therapy and breast cancer.
No researcher who suspected such a link would undertake a randomized experiment
that would involve exposing women to this potential hazard. A study that
involves merely tracking women who are already undergoing replacement therapy
is much less objectionable.
Retrospective Causal
Studies. Retrospective studies begin with two groups, our familiar
experimental and control groups, but the two are composed of subjects who do
and do not have the effect in question. The study then involves looking into
the subjects’ backgrounds in an attempt to uncover different levels of the
potential causal factor in the two groups.
The following retrospective study investigated the effects of
lead on children. A government health survey of 4,000 U.S. children between the
ages
|
QUICK
REVIEW 5.2 Prospective
Studies |
|
Population from which experimental Preliminary Preliminary Experimental Group Control
Group All have been exposed
to the None
have been exposed to
Modified so that other potential causal factors are equally
represented in both groups. |
of 4 and 15, done between 1999 and 2002,
included 135 children with attention deficit hyperactivity disorder (ADHA).
Blood tests were done on all 4,000. Among the 135 children with ADHA, blood
lead levels of more than two micrograms per deciliter occurred at a much
greater rate than in the other children. The ADHA children were four times more
likely to have elevated lead levels in their blood, leading researchers to conclude
that exposure to lead may cause ADHA.
Even the best of retrospective studies can provide only weak
evidence for a causal link. This is because, in retrospective studies, it is
exceedingly difficult to control for other potential causal factors. Subjects
are selected because they either do or do not have the effect in question, so
other potential causal factors may automatically be built into our two groups.
A kind of backwards matching is sometimes possible in retrospective studies. In
the study above, if other sources of increased lead level can be identified,
some attempt can be made to insure that these other factors are equally
represented in the two groups. However, even if the two groups can be
configured so that they exhibit similar levels of other suspected causes, we
have at most very tentative evidence for the causal link in question. The
process of re-configuring the two groups may leave us with a distorted picture
of the extent to which the causal factor under investigation is responsible for
the effect. That the two groups now appear to be alike with respect to other
causal factors is, thus, largely because they are contrived to appear that way.
The particular study we are discussing suffers from an
unusual weakness in the selection process for retrospective studies.
Essentially, retrospective studies involve looking for something common in the
background of subjects who exhibit an effect: in this case, elevated lead
levels in the blood. It is at least a
|
QUICK
REVIEW 5.3 Retrospective
Studies |
|
Population from which experimental
All exhibit the suspected None
exhibit the suspected effect.effect. Sometimes modified after all data are collected to equalize the
occurrence of other potential causal factors in the two groups. |
possibility that cause and effect have been
reversed! In the ADHA study, researchers noted that children with ADHA are more
likely than others to eat old leaded paint chips or inhale leaded paint dust
because of their hyperactivity. It may be that ADHA is in part responsible for
the increased lead levels.
All we are in a position to conclude, as the result of a
retrospective study, is that we have looked into the background of subjects who
have a particular effect and we have found that a suspected cause occurs more
frequently than in subjects who do not have the effect in question. Whether the
effect is due to the suspected cause or whether cause and effect are reversed
may be difficult to say even when pains are taken to control for other
potential causal factors.
One final limitation of retrospective studies is that they
provide no way of estimating the level of difference of the effect being
studied. The very design of retrospective studies insures that 100% of the
experimental group, but none of the control group, will have the effect. In the
ADHA study, it was reported that ADHA children were “four times more likely” to
have elevated lead levels in their blood. No doubt this difference is
statistically significant. This tells us the difference is probably not due to
chance. But it does not tell us that those children with elevated blood levels
are “four times more likely” to have ADHA. Though this difference is striking,
it tells us nothing about the efficacy of the suspected cause. By contrast, a
prospective study—in which the experimental subjects had levels of the
suspected effect four times higher than control subjects—would provide much
stronger evidence of a causal link. Here, the evidence would be strengthened if
other possible causes could be ruled out by matching prior to the onset of the
study.
Because of these difficulties, retrospective studies are best
regarded as a tool for uncovering potential causal links. The major advantage
to retrospective studies, by contrast with randomized and prospective studies,
is that they can be carried out quickly and inexpensively since they involve
little more than careful analysis of data that is already available. And
sometimes alacrity is of the essence. Imagine that there has been a rash of
cases of food poisoning in your community. Before much of anything can be done,
health authorities need some sense of what might be causing the problem.
Interviews with the victims reveal that most had eaten at a particular
restaurant within the last few days. This bit of retrospectively gained
information may provide just the clue needed to get at the cause or causes of
the problem.
READING BETWEEN THE LINES
The results of causal research are reported
in specialized scientific journals. Typically an article will include full
information about the design of the experiment, the results, and a complete
statistical analysis where appropriate. Many medical journals, like The Journal of the American Medical
Association and The New England
Journal of Medicine, also provide full disclosure of the funding sources of
the research. Conclusions will be carefully qualified, and the article will
probably contain a brief history of other relevant research. When research
uncovers a result which may have an impact on the general public, it will often
be reported in the popular media in newspaper, magazines, the Internet, and on
television. And it is here—in the mass media—that most of us encounter the
findings of causal research.
Unfortunately, the popular media tend to do a poor job of
reporting the outcomes of causal research. Media reports will often leave out
crucial information, no doubt in the name of brevity; A 20- or 30-page journal
article usually will be covered in a few paragraphs. Such reports tend also to
dispatch with the kind of careful qualifications that normally accompany the
original write-up of the results. For these reasons, it is important to learn
to read between the lines of popular reports if we are to make sense of the
research on which they are based.
Here, for example, is the complete text of a newspaper story
about an important piece of causal research:
Lithium, which is widely prescribed for manic-depressive
disorders, may be the first biologically effective drug treatment for
alcoholism, according to studies at St. Luke’s Medical Center. The new evidence
indicates that the drug appears to have the unique ability to act on the brain
to suppress an alcoholic’s craving for alcohol. The St. Luke’s study involved
84 patients, ranging from 20 to 60 years of age, who had abused alcohol for an
average of 17 years. Eighty-eight percent were male. Half the patients were
given lithium while the other half took a placebo, a chemically inactive
substance. Seventy-five percent of the alcoholics who regularly took their
daily lithium pills did not touch a drop of liquor for up to a year and a half
during the follow-up phase of the experiment. This abstinence rate is at least
50% higher than that achieved by the best alcohol treatment centers one to five
years after treatment. Among the alcoholics who did not take their lithium regularly,
only 35% were still abstinent at the end of 18 months. Among those who stopped
taking the drug altogether, all had resumed drinking by the end of six months.
(Researchers tested the level of lithium in the blood of the subjects to
determine if they were taking the drug regularly.)
Just what are we to make of this story and the research it
describes? Is lithium effective in the treatment of alcoholism? (Note that the
story begins by claiming that lithium “may be” the first effective treatment
for alcoholism.) In trying to make sense of an article like this one, it is
necessary to try to answer a number of questions, all based on our findings in
this chapter:
What is the causal hypothesis at issue?
What kind of causal experiment is
undertaken?
What crucial facts and figures are missing
from the report?
Given the information you have at your
disposal, can you think of any major flaws in the design of the experiment?
Given the information available, what
conclusion can be drawn about the causal hypothesis?
Let’s consider again the news article about the lithium
study, now in light of our five questions.
What is the causal
hypothesis at issue? The hypothesis is that lithium suppresses the
alcoholic’s craving for alcohol.
What kind of causal
experiment is undertaken? Randomized. Subjects are divided into
experimental and control groups prior to the experiment and only the
experimental subjects are exposed to the suspected causal agent.
What crucial facts and
figures are missing from the report? The passage gives us no
information about what happened to the members of the control group. Nor does
it tell us the number of subjects from the experimental group who “regularly took
their daily lithium pills.” We know that 75% of these subjects did so, but this
could be as few as three out of four. All we are told of the remaining members
of the experimental group is that 35% remained abstinent and that some stopped
taking the drug altogether. We are not told how many are in each of these
subgroups. It is possible that the majority of experimental subjects did not
remain abstinent. Given the information we have at our disposal, we just cannot
say for sure, one way or the other. Though we are given no information about
the control group, we are provided with some information against which to
assess the results in the experimental group: we are told that the 75%
abstinence rate is “at least 50% higher than that achieved by the best alcohol
treatment centers one to five years after treatment.” However, we are not told
whether the success rate for treatment centers is a percentage of people who
entered treatment or people who completed treatment. If the former is the case,
there is a strong possibility treatment centers have a higher rate of success
than that established in the experiment. Once again, we can draw no conclusions
since we are not provided with the key comparative information.
Given the information you
have at your disposal, can you think of any major flaws in the design of the
experiment? One possible flaw comes to mind. It may be that the
subjects who continued to take their medication (lithium or placebo) throughout
the entire 18 months of the experiment were more strongly motivated to quit
drinking than the other subjects. And this may have influenced the outcome of
the experiment. Precautions need to be taken to ensure either that no subjects
lacked this motivation or that they were equally represented in experimental and
control groups. Here, information about the results of the control group would
be helpful. If roughly equal numbers of people dropped out of both groups, we
would have some initial reason to think that we had controlled for motivation.
Given the information
available, what conclusion can be drawn about the causal hypothesis? We
can conclude very little particularly because we are given no information about
what happened to the control group. This is not to say that the experiment
itself warrants no conclusion about the possible link between lithium and
alcoholism. However the report about the study with which we have been working
has presented us with so little information that we can draw no conclusion.
Media accounts of causal studies may make reference to a
possible causal mechanism. The article about lithium and alcoholism does
mention one: lithium, it is hypothesized, may act on the brain in some way to
suppress an alcoholic’s craving for alcohol. Though in this case the proposed
mechanism is vague, a well-understood, well-established mechanism can provide
an additional piece of information to suggest that a study is on to something.
.png)
