Wiktoria Lawniczak’s research while affiliated with University of Leeds and other places

Understanding causality, over mere association, is vital for researchers wishing to inform policy and decision making – for example, when seeking to improve population health outcomes. Yet, contemporary causal inference methods have not fully tackled the complexity of data hierarchies, such as the clustering of people within households, neighbourhoods, cities, or regions. However, complex data hierarchies are the rule rather than the exception. Gaining an understanding of these hierarchies is important for complex population outcomes, such as non-communicable disease, which is impacted by various social determinants at different levels of the data hierarchy. The alternative of analysing aggregated data could introduce well-known biases, such as the ecological fallacy or the modifiable areal unit problem. We devise a hierarchical causal diagram that encodes the multilevel data generating mechanism anticipated when evaluating non-communicable diseases in a population. The causal diagram informs data simulation. We also provide a flexible tool to generate synthetic population data that captures all multilevel causal structures, including a cross-level effect due to cluster size. For the very first time, we can then quantify the ecological fallacy within a formal causal framework to show that individual-level data are essential to assess causal relationships that affect the individual. This study also illustrates the importance of causally structured synthetic data for use with other methods, such as Agent Based Modelling or Microsimulation Modelling. Many methodological challenges remain for robust causal evaluation of multilevel data, but this study provides a foundation to investigate these.

Background Government bodies, private enterprises, and researchers increasingly use ‘big data’ to monitor, evaluate interventions, make future predictions, and seek causal understanding. Such data are often complex in structure (i.e., hierarchical), which creates challenges for methods that work for a single homogeneous population, but which mislead if applied to data with substructure. If causal insights are sought, this usually pertains to the individual, yet most datasets are aggregated due to issues surrounding sensitive personal information, which is why it is common to encounter simulation approaches, such as agent-based modelling (ABM), or ecological analyses that evaluate only marginal (i.e., clustered) information. Contemporary causal inference methods are yet to tackle the full complexities of multilevel data structure, beyond longitudinal repeated measures. There is thus a gap in our understanding and methods capabilities surrounding causal analysis of structured data, which this study examines. Methods 1) devise a hierarchical causal diagram that encodes a multilevel data generating mechanism with prespecified cross-level causal relationships; 2) simulate multilevel data from the causal diagram and obtain aggregated data; 3) contrast multilevel and ecological estimates of a simulated individual-level causal effect, to assess the presence and extent of potential biases. Results Unlike a multilevel analysis of the full data, ecological analyses of cluster-level data do not generally yield robust causal effect estimates. While it is known that ecological analyses invoke the ‘ecological fallacy’ (i.e., where attributing features of clusters to units within clusters may mislead), this study quantifies this for the first time within a formal causal framework. An algorithm to simulate causally structured multilevel data is also demonstrated. Conclusion Insights into the limitations of common analytical practices were made possible by simulating causally structured hierarchical data, demonstrating the value of causal diagrams in both simulation and causal analysis. Methodological challenges remain for robust causal evaluation of big data, but this study shows how to investigate these challenges. Results reveal the need for individual-level data with application of multilevel analyses to achieve robust causal inquiry; ecological analyses do not generally provide sound causal effect estimation. If individual-level data are unavailable, synthetic data (informed by available marginal data) becomes necessary to answer causal questions and this study provides a tool to generate synthetic population data that reflects multilevel causal structures, which in turn will then better inform the use of methods such as ABMs. This study has enormous implications for the use of big data when seeking causal insights.