Development of a qualitative method to assist in interpretation of results from disproportionality analyses in the SAFEGUARD project.
Lorna Hazell, Niklas Schmedt, Lorenza Scotti, Ingrid Leal, Gianluca Trifiro, Miriam Sturkenboom and Saad Shakir.
Background
Disproportionality analyses are used in spontaneous reporting systems to support signal detection. Limitations of these systems, however, present challenges for interpretation and comparisons with other types of data. The purpose of this study was to develop an output suitable for integration with other data, specifically epidemiological studies investigating the safety of diabetes drugs in the context of the SAFEGUARD project.
Objectives
To categorise associations from disproportionality analyses in the SAFEGUARD project and assess their level of uncertainty.
Methods
Disproportionality analyses were performed for 29 diabetes drugs and 9 outcomes of interest using 8 different strategies: two databases (FAERS and Eudravigilance), two outcome definitions (broad and narrow) and two reference groups (all drugs and diabetes drugs only). An algorithm was defined, depending on which analysis strategies detected disproportionality, to categorise each drug-outcome pair (n=261) as low, medium, high or unclear evidence of association. Each categorisation was qualitatively assessed using 9 ‘uncertainty factors’: 1) specificity of outcome definition, 2) potential for stimulated reporting, 3) time on the market, 4) consistency between databases, 5) multiple testing, 6) reporter status, 7) level of drug use, 8) masking bias and, 9) use of statistical thresholds. An overall ‘uncertainty score’ for each categorisation from 0 (low uncertainty) to 1 (high uncertainty) was derived from the proportion of the 9 uncertainty factors that applied.
Results
Twenty-three pairs (8.8%) were classified as high evidence of an association, 48 (18.4%) as medium, 122 (46.7%) as low and 68 (26.1%) as unclear. Examples of those categorised as ‘high’ are shown in Table 1.
Table 1 Examples of Pairs Categorised as ‘High’ Evidence of Association with their Uncertainty Scores
| Drug-outcome pair | Uncertainty Scores |
|---|---|
| Incretin-based therapies & pancreatic outcomes | From 0.25 for liraglutide to 0.625 for vildagliptin |
| Glitazones & heart failure | 0.57 for pioglitazone; 0.14 rosiglitazone |
| Pioglitazone & bladder cancer | 0.14 |
| Rosiglitazone & cardiovascular outcomes | 0.14 |
Conclusion
Disproportionality analyses should be interpreted in the context of their limitations due to potential biases in the data. We have used an uncertainty score, as a measure of these limitations, to qualify the categorisation of the associations detected. The assessment criteria were not exhaustive, were topic-specific and required considerable computation. Thus further refinement and evaluation is required before such methods could be recommended for routine use. Nevertheless the categorisation and uncertainty score provide an output that can be compared and integrated with similarly formatted results from independent studies.