Weighted versus unweighted estimates: Is it wrong if they are different?

A while back, I and a colleague submitted a manuscript for publication on a peer-reviewed journal. Our manuscript was based on a weighted analysis of data to attenuate bias due to unequal probability of selection of sub-populations^[1].  To be more transparent, we reported both the weighted and unweigthed frequency distribution of the population by important variables.

After the manuscript underwent a thorough anonymous peer-review, we received a peer-review report which states that because the weighted and unweighted frequencies did have substantial difference, the analysis weight we used was wrong. Does obtaining different weighted and unweighted frequencies render the assigned analysis weight invalid?

As pointed out above, if weighting serves to attenuate bias due to disproportionate sampling of sub-populations, then weighted and unweighted frequencies (and other estimates such as means and regression coefficients)  are expected to differ. The magnitude of the difference depends on the extent of the disproportion in the sampling.

To clear the above issue, let’s see an example. Let’s say we have a population of size 100 comprised of 25 males and 75 females. That is, males comprise 25% of the population and females comprise 75%.  If we take a sample of size 30 using simple random sampling and if this sample is comprised of 15 males and 15 females, then based on our sample, males comprise 50% of the population and females comprise the remaining 50%. However, the probability of selection for males was 60% (15/25*100) whereas that for females was only 20% (15/75*100). Clearly, the sampling is disproportionate. Hence, the analysis should take account of this disproportion in the sampling. If not, results will be biased.

In this simplest scenario, we need to calculate an analysis weight as the inverse of the selection probabilities of males and females. Accordingly, the analysis weight (non-normalized) for males will be 1.67 and for females 5. If we declare a complex sample survey design (using svyset in Stata) and re-do the analysis (complex sample analysis), then the frequency of males will be 25% and that of females will be 75%.

Hence, it is not abnormal if the weighted and unweighted frequencies differ. That is rather what must be expected given the weight variable has been properly computed.

Reference

1. Heeringa SG, West BT, Berglund PA. Applied Survey Data Analysis. Boca Raton, FL: Chapman & Hall/CRC; 2010. 

The pitfall of considering design effect during the design phase of a study but ignoring it during the analysis

Nowadays, it is a common practice to employ complex sampling strategies—sampling strategies that involve stratification, clustering and weighting—in public health survey researches. This is so because the use of simple random sampling (SRS) in population-based surveys imposes both technical and financial difficulties. Besides, objectives of a study such as making sub-population inferences may necessitate using such sampling schemes.

In many public health surveys implementing the foregoing sampling strategy, it is common to see the sample size needed if SRS were used multiplied by an anticipated design effect (i.e., the measure of loss of precision due to complex sampling). That is in fact a necessary step in studies planning complex sampling procedures.

But sadly enough, in many such studies the sampling scheme is completely forgotten during the analysis phase of the study and data are analyzed as if they were generated through SRS technique. In effect, analysis outputs of interest such population descriptive statistics and model parameters are biased.

Briefly, here is what happens due to failure to take account of the sampling features during the analysis.

1. Sampling weights help to attenuate bias that results from unequal probability of selection of sampling units. Thus, ignoring weighting during the analysis results in underestimation of population descriptive statistics such as means and proportions.
2. Clustering generally decrease precision (i.e., increases the standard error of estimates). Hence, failing to take account of clustering during the analysis underestimates the standard error (because such analyses are based on assumption of SRS). In effect, the α-error is inflated.
3. Stratification increases the efficiency of the sample. That means, it reduces the standard error of estimates. When it is ignored during analysis, the standard error is increased. In effect, β-error is inflated.

The joint effect of weighting, clustering and stratification is a tug-of-war with weighting and clustering on one side pulling the standard error up and stratification on the other side pulling the standard error down. The net effect depends on which side succeeds in dragging the other. Often, the pulling effect of clustering and weighting outweighs and standard errors are increased as a net effect. Thus, all sampling features should be explicitly taken account of in the analysis of data obtained based on the complex sampling scheme. Otherwise, the bias becomes paramount.

The pitfall of failing to take account of design features in studies that have considered design effect during the sample size calculation is this. The sample size required to attain a given precision using a complex sample is much higher than the corresponding SRS sample size. Hence, when the sampling features are ignored during the analysis, an SRS of large sample size is wrongly assumed. Thus, the standard errors of the estimates are highly underestimated (though there is also a possibility of overestimation based on the net effect of stratification, clustering and weighting). As a result, statistical tests are likely to be significant while they are not (i.e., tests of significance are biased away from the null; the possibility of bias towards the null cannot be precluded, however.)

To conclude, I would say instead of considering design effect during the sampling design and then ignore all design features during the analysis, it may be better to ignore the design effect from the outset. Or else, analysis techniques that properly take account of the sampling scheme should be applied.

Reference

For a detailed treatment of the complex sampling techniques and associated analytic issues readers are referred to:

• Heeringa SG, West BT, Berglund PA. Applied Survey Data Analysis. Boca Raton, FL: Chapman & Hall/CRC; 2010.