91³ÉÈ˰涶Òô

Case Study - the story of INTREPID

INTREPID is a scoping review examining the use of instrumental variables (IVs) in medicines research. The project was originally conceived by MI-CRE Chief Investigator and previous Methods and Data Portfolio Chair, Professor Nicole Pratt, in response to the increasing application of IV methods in observational studies in the medical field.

Unmeasured confounding is a principal concern when estimating causal effects from observational data, and IV methods are used to control for this confounding. There are assumptions underlying the application of IVs but a lack of reporting guidelines means their utility and validity in medicines research is not well understood. A better understanding of the contemporary application of IVs – how often and the types of instruments used, the types of studies using IVs, and whether assumption testing is reported - will help address these knowledge gaps, making this project critically important for advancing medicines research within Australia and globally.

INTREPID brings together 16 early-, mid- and senior-career researchers across all MI-CRE nodes as well as international expertise from Professor Kristian Filion, McGill University (Canada). The project has provided a wealth of opportunities for learning, including development of methodological skills in the design and conduct of a scoping review, increased knowledge of instrumental variables in medicines research, technical skills in data management and dynamic database builds, and transferable skills such as project management and leading/participating in big team science. INTREPID has empowered strong cross-node collaborative learning and relationships through active participation from researchers with diverse methods and research backgrounds. Further, with E/MCRs providing day-to-day leadership and management on the project, INTREPID is also developing local leadership and fostering self-sufficiency so that the medicines research community can continue to grow independently.

Beyond the direct skills and capacity building opportunities, INTREPID will deliver long-term impact. Publication of findings from the scoping review will provide future researchers with guidance for the conduct and reporting of IV methods in medicines research. The reach and significance of the published work will contribute to greater public awareness of MI-CRE and its activities, with INTREPID having already been presented to the MI-CRE Policy and Translation Reference Group (consumers and policymakers) and an international audience (International Conference on Pharmacoepidemiology 2024).

INTREPID has fostered strong relationships within MI-CRE, as well as with international researchers, further leading to the design of subsequent Capacity Building and Training/research activities. In this way, INTREPID serves as a cornerstone of MI-CRE's broader capacity-building efforts, driving growth and collaboration across the medicines research community.

Case Study - PyRrHiC: Pharmaceutical Benefits Scheme Replication and Harmonisation Challenge using Australian dispensing data

At the end of 2022 the Methods and Data Portfolio set themselves a challenge — if different researchers attempt to answer the same question, with the same dataset, will they get the same results? The outcome of this challenge was an Australia-wide collaborative project known as the ‘PyRrHiC Study’, or the Pharmaceutical Benefits Scheme Replication and Harmonisation Challenge.

The aim of the experiment was to identify how different decisions made during data preparation and analysis impact study results. The experiment involved 15 MI-CRE collaborators from four nodes: University of South Australia (UniSA), University of New South Wales (UNSW), University of Sydney (USyd) and University of Western Australia (UWA). This working group developed a for a medicine utilisation study that described the analyses to be conducted, and specified how the results were to be collated, but did not describe all operational details.

Working independently, each node developed a detailed analysis protocol using the template and kept these confidential from the other sites. Analysis was then conducted, based on this protocol, using the PBS 10% sample, a dataset commonly used for Australian pharmacoepidemiology studies and familiar to all the involved researchers. Following the completion of analyses, the results were shared amongst the study team and each node nominated a representative to be part of a small working group to compare and quantify the variability across the four sets of results.

The findings were surprising. Although the four groups generated datasets with comparable demographic results, the time-to-event estimates were all completely different. When comparing the methods used to conduct the study, the working group found that different analytical decisions at multiple points were made in each group leading to different results. This included differences in how exposure duration was calculated and operational definitions of treatment events.

The next steps of this project are to use PyRrHiC’s findings to help improve replicability in medicine utilisation studies. Specifically, we intend to develop guidance on data preparation and analysis for these types of studies; and to develop a 'user guide' for studies using the PBS 10% dataset.

Medicine utilisation studies, like the one replicated in this experiment, are used by decision-makers to inform policy and clinical practice. By demonstrating that four groups answering the same question produced four different sets of results this study showed that replicability is a very real issue that needs to be addressed. This is important work for pharmacoepidemiology researchers in Australia using PBS data as well as researchers conducting medicine utilisation studies internationally.

The PyRrHiC study is an exemplar of the types of capacity-building projects initiated through MI-CRE, involving students, E/MCRs and senior investigators from across the network, developing skills in collaborative design, standardised protocols, writing, coding, and project management. The PyRrHiC study was in the journal Pharmacoepidemiology and Drug Safety in March 2024. The work has also been presented by members of the working group at national and international conferences.