The pitfall of considering design effect during the design phase of a study but ignoring it during the analysis

Nowadays, it is a common practice to employ complex sampling strategies—sampling strategies that involve stratification, clustering and weighting—in public health survey researches. This is so because the use of simple random sampling (SRS) in population-based surveys imposes both technical and financial difficulties. Besides, objectives of a study such as making sub-population inferences may necessitate using such sampling schemes.

In many public health surveys implementing the foregoing sampling strategy, it is common to see the sample size needed if SRS were used multiplied by an anticipated design effect (i.e., the measure of loss of precision due to complex sampling). That is in fact a necessary step in studies planning complex sampling procedures.

But sadly enough, in many such studies the sampling scheme is completely forgotten during the analysis phase of the study and data are analyzed as if they were generated through SRS technique. In effect, analysis outputs of interest such population descriptive statistics and model parameters are biased.

Briefly, here is what happens due to failure to take account of the sampling features during the analysis.

1. Sampling weights help to attenuate bias that results from unequal probability of selection of sampling units. Thus, ignoring weighting during the analysis results in underestimation of population descriptive statistics such as means and proportions.
2. Clustering generally decrease precision (i.e., increases the standard error of estimates). Hence, failing to take account of clustering during the analysis underestimates the standard error (because such analyses are based on assumption of SRS). In effect, the α-error is inflated.
3. Stratification increases the efficiency of the sample. That means, it reduces the standard error of estimates. When it is ignored during analysis, the standard error is increased. In effect, β-error is inflated.

The joint effect of weighting, clustering and stratification is a tug-of-war with weighting and clustering on one side pulling the standard error up and stratification on the other side pulling the standard error down. The net effect depends on which side succeeds in dragging the other. Often, the pulling effect of clustering and weighting outweighs and standard errors are increased as a net effect. Thus, all sampling features should be explicitly taken account of in the analysis of data obtained based on the complex sampling scheme. Otherwise, the bias becomes paramount.

The pitfall of failing to take account of design features in studies that have considered design effect during the sample size calculation is this. The sample size required to attain a given precision using a complex sample is much higher than the corresponding SRS sample size. Hence, when the sampling features are ignored during the analysis, an SRS of large sample size is wrongly assumed. Thus, the standard errors of the estimates are highly underestimated (though there is also a possibility of overestimation based on the net effect of stratification, clustering and weighting). As a result, statistical tests are likely to be significant while they are not (i.e., tests of significance are biased away from the null; the possibility of bias towards the null cannot be precluded, however.)

To conclude, I would say instead of considering design effect during the sampling design and then ignore all design features during the analysis, it may be better to ignore the design effect from the outset. Or else, analysis techniques that properly take account of the sampling scheme should be applied.

Reference

For a detailed treatment of the complex sampling techniques and associated analytic issues readers are referred to:

• Heeringa SG, West BT, Berglund PA. Applied Survey Data Analysis. Boca Raton, FL: Chapman & Hall/CRC; 2010.