Nowadays, it is a common practice to
employ complex sampling strategies—sampling strategies that involve
stratification, clustering and weighting—in public health survey researches.
This is so because the use of simple random sampling (SRS) in population-based
surveys imposes both technical and financial difficulties. Besides, objectives of
a study such as making sub-population inferences may necessitate using such sampling
schemes.
In many public health surveys implementing
the foregoing sampling strategy, it is common to see the sample size needed if
SRS were used multiplied by an anticipated design effect (i.e., the measure of
loss of precision due to complex sampling). That is in fact a necessary step in
studies planning complex sampling procedures.
But sadly enough, in many such
studies the sampling scheme is completely forgotten during the analysis phase
of the study and data are analyzed as if they were generated through SRS
technique. In effect, analysis outputs of interest such population descriptive statistics
and model parameters are biased.
Briefly, here is what happens due to
failure to take account of the sampling features during the analysis.
- Sampling weights help to attenuate bias that results from unequal probability of selection of sampling units. Thus, ignoring weighting during the analysis results in underestimation of population descriptive statistics such as means and proportions.
- Clustering generally decrease precision (i.e., increases the standard error of estimates). Hence, failing to take account of clustering during the analysis underestimates the standard error (because such analyses are based on assumption of SRS). In effect, the α-error is inflated.
- Stratification increases the efficiency of the sample. That means, it reduces the standard error of estimates. When it is ignored during analysis, the standard error is increased. In effect, β-error is inflated.
The joint effect of weighting,
clustering and stratification is a tug-of-war with weighting and clustering on
one side pulling the standard error up and stratification on the other side
pulling the standard error down. The net effect depends on which side succeeds
in dragging the other. Often, the pulling effect of clustering and weighting outweighs
and standard errors are increased as a net effect. Thus, all sampling features
should be explicitly taken account of in the analysis of data obtained based on
the complex sampling scheme. Otherwise, the bias becomes paramount.
The pitfall of failing to take account
of design features in studies that have considered design effect during the
sample size calculation is this. The sample size required to attain a given
precision using a complex sample is much higher than the corresponding SRS
sample size. Hence, when the sampling features are ignored during the analysis,
an SRS of large sample size is wrongly assumed. Thus, the standard errors of the
estimates are highly underestimated (though there is also a possibility of
overestimation based on the net effect of stratification, clustering and
weighting). As a result, statistical tests are likely to be significant while
they are not (i.e., tests of significance are biased away from the null; the
possibility of bias towards the null cannot be precluded, however.)
To conclude, I would say instead of
considering design effect during the sampling design and then ignore all design
features during the analysis, it may be better to ignore the design effect from
the outset. Or else, analysis techniques that properly take account of the sampling
scheme should be applied.
Reference
For a detailed treatment of the
complex sampling techniques and associated analytic issues readers are referred
to:
- Heeringa SG, West BT, Berglund PA. Applied Survey Data Analysis. Boca Raton, FL: Chapman & Hall/CRC; 2010.
