Background and objectives: Despite the growing popularity of the vignette methodology to deal with self-reported data and to address the related issue of reporting heterogeneity, the formal evaluation of the validity of this methodology is still a topic of research. Some critical assumptions need to hold in order for this method to be valid. In this paper we perform some analysis on the assumption of “vignette equivalence” (implying that individuals perceive the latent level of the variable represented in the vignette in the same way), using data about responsiveness contained within the World Health Survey.
Data: We use data taken from the WHS, a survey launched by the WHO in 2001 and comprising seventy countries. All surveys were drawn from nationally representative frames with known probability resulting in sample sizes of between 600 and 10,000 respondents across the countries surveyed. We refer to the module on health system responsiveness contained in the WHS. In broad terms, health system responsiveness has been defined as the way in which individuals are treated and the environment in which they are treated, encompassing the notion of an individual’s experience of contact with the health system (Valentine et al. 2003)
Method: First, we perform a non parametric analysis to test the assumption of vignette equivalence. We use a test based on the global ordering of the vignettes. The global ordering of the vignettes for a domain can be obtained pooling all the responses across countries and considering the average categorical response for each vignette. (Murray et al. 2003). A minimal condition for the assumption of vignette equivalence to hold is that individual responses are consistent with the global ordering of vignettes. This test has already been utilized, with reference to other datasets, by Kapteyn et al. (2007), Kristensen and Johansson (2008) and Kapteyn et al. (2008). Secondly, after having estimated a hierarchical ordered probit (HOPIT) model for responsiveness on the pool of countries, we perform some sensitivity analysis, as suggested by Kristensen and Johansson (2008). We re-estimate our model considering groups of countries with similar cultures, stratified according to the Inglehart-Welzel chart of values (www.worldvaluessurvey.org). We also consider groups of countries that differ for their level of socio-economic development, as represented by the Human Development Index (Unite Nations Development Programme). If the results of the sensitivity analysis are robust, the assumption of vignette equivalence is not contradicted. Thirdly, we model the reporting behaviour of the respondents through a two-step regression procedure to evaluate the possibility of multidimensionality in the constructs described by the vignettes.
Results: The non-parametric analysis, based on the global ordering of the vignettes, does not contradict the assumption of vignette equivalence. In almost all countries more than 90% of respondents reports vignette orderings that are consistent with the global ordering. The assumption of vignette equivalence is also not contradicted by our parametric analysis. The results of the sensitivity analysis appear to be robust to the stratification of countries according to both the Inglehart-Welzel chart of values and the Human Development Index. Moreover, the analysis of the reporting behaviour of respondents appears to reject the hypothesis of multidimensionality in the constructs described by the vignettes, and thus provide some support to the assumption of vignette equivalence.
Conclusions: The results derived from our analysis do not contradict the assumption of vignette equivalence, and, accordingly, provide tentative support for the use of the vignette methodology to correct for the presence of reporting heterogeneity.
(Joint work with Nigel Rice and Peter Smith.)