CE-DAT quality checklist
Since 2004, the CE-DAT initiative compiles the results of cross-sectional sample surveys from complex emergencies. By providing estimates of mortality, nutrition and vaccination indicators, those surveys have been adopted as a means of rapidly assessing the direct impact of a crisis on human populations (Guha-Sapir, 1991). They are further used to guide policy decisions and humanitarian interventions or to provide data for advocacy purposes.
Regarding the multiplicity of actors carrying out such surveys (i.e. NGOs, UN agencies, academics), there has been an increasing need of standardization of epidemiological methods. Moreover, while compiling the data, the CE-DAT team has observed several recurrent issues relating to the reporting of methodological elements, such as omitted confidence intervals, lack of interpretation, absence of report of biases and limitations. Past reviews of survey quality have also shown varied levels of comparability of results due to variation in reporting, the use of different methodologies and poor methodological rigor (Boss et al., 1994 ; Garfield, 2000 ; Spiegel et al., 2004 ; Prudhon and Spiegel, 2007).
In order to advise the CE-DAT team on such issues within daily operations, CRED has conducted Technical Advisory Group (TAG) meetings to bring together NGOs, UN agencies and development agencies. The TAGs were supplemented by Expert Group Meetings (EGM), which have taken place in 2007 and 2008 and during which epidemiologists, data specialists with specific expertise and the CE-DAT team analyzed issues of a more methodological nature, including indices of survey quality and reporting.
A completeness checklist was first developed in order to identify the methodological elements required for a survey to be considered valid and publicly available in the CE-DAT database. The second EGM has further developed an evaluation system to assess the quality of surveys, proposing guidelines for consistent reporting and methodological quality.
Those guidelines were convened by the Expert Group and further justified by evidence from the literature. They were rendered practical through the use of a checklist of elements to be included in a survey report. They are currently used internally and were pilot tested on surveys in the CE-DAT database for further refinement of the criteria.
The checklist will consist of a tool for field organizations in the preparation of reports and for the CE-DAT team in screening surveys for quality purposes, as it has been doing regarding completeness criteria. Additionally, the guidelines might assist users of the literature in understanding components and interpreting the evidence that contribute to a valid survey report. In conclusion, they should be viewed as a step forward in the process of standardization of epidemiological methods in complex emergency settings. In the future, the CE-DAT team hopes to use the CE-DAT initiative to maintain and strengthen the dialogue with users, researchers and fields investigators regarding the application of the guidelines in order to further refine the criteria, as well as to explore more delineations of methodological quality.
A consensus paper on these guidelines will be published shortly.