BMC Medical Research Methodology

Purpose The process of searching for and selecting clinical evidence for systematic reviews (SRs) or clinical guidelines is essential for researchers in Traditional Chinese medicine (TCM). However, this process is often time-consuming and resource-intensive. In this study, we introduce a novel precision-preferred comprehensive information extraction and selection procedure to enhance both the efficiency and accuracy of evidence selection for TCM practitioners. Methods We integrated an established deep learning model (Evi-BERT combined rule-based method) with Boolean logic algorithms and an expanded retrieval strategy to automatically and accurately select potential evidence with minimal human intervention. The selection process is recorded in real-time, allowing researchers to backtrack and verify its accuracy. This innovative approach was tested on ten high-quality, randomly selected systematic reviews of TCM-related topics written in Chinese. To evaluate its effectiveness, we compared the screening time and accuracy of this approach with traditional evidence selection methods. Results Our finding demonstrated that the new method accurately selected potential literature based on consistent criteria while significantly reducing the time required for the process. Additionally, in some cases, this approach identified a broader range of relevant evidence and enabled the tracking of selection progress for future reference. The study also revealed that traditional screening methods are often subjective and prone to errors, frequently resulting in the inclusion of literature that does not meet established standards. In contrast, our method offers a more accurate and efficient way to select clinical evidence for TCM practitioners, outperforming traditional manual approaches. Conclusion We proposed an innovative approach for selecting clinical evidence for TCM reviews and guidelines, aiming to reduce the workload for researchers. While this method showed promise in improving the efficiency and accuracy of evidence-based selection, its full potential required further validation. Additionally, it may serve as a useful tool for editors to assess manuscript quality in the future.

Background Focus groups (FGs) are an established method in health research to capture a full range of different perspectives on a particular research question. The extent to which they are effective depends, not least, on the composition of the participants. This study aimed to investigate how published FG studies plan and conduct the recruitment of study participants. We looked at what kind of information is reported about recruitment practices and what this reveals about the comprehensiveness of the actual recruitment plans and practices. Methods We conducted a systematic search of FG studies in PubMed and Web of Science published between 2018 and 2024, and included n = 80 eligible publications in the analysis. We used a text extraction sheet to collect all relevant recruitment information from each study. We then coded the extracted text passages and summarised the findings descriptively. Results Nearly half (n = 38/80) of the studies were from the USA and Canada, many addressing issues related to diabetes, cancer, mental health and chronic diseases. For recruitment planning, 20% reported a specific sampling target, while 6% used existing studies or literature for organisational and content planning. A further 10% reported previous recruitment experience of the researchers. The studies varied in terms of number of participants (range = 7–202) and group size (range = 7–20). Recruitment occurred often in healthcare settings, rarely through digital channels and everyday places. FG participants were most commonly recruited by the research team (21%) or by health professionals (16%), with less collaboration with public organisations (10%) and little indication of the number of people involved (13%). A financial incentive for participants was used in 43% of cases, and 19% reported participatory approaches to plan and carry out recruitment. 65 studies (81%) reported a total of 58 limitations related to recruitment. Conclusions The reporting of recruitment often seems to be incomplete, and its performance lacking. Hence, guidelines and recruitment recommendations designed to assist researchers are not yet adequately serving their purpose. Researchers may benefit from more practical support, such as early training on key principles and options for effective recruitment strategies provided by institutions in their immediate professional environment, e.g. universities, faculties or scientific associations.

Background The integration of real-world evidence (RWE) from real-world data (RWD) in clinical research is crucial for bridging the gap between clinical trial results and real-world outcomes. Analyzing routinely collected data to generate clinical evidence faces methodological concerns like confounding and bias, similar to prospectively documented observational studies. This study focuses on additional limitations frequently reported in the literature, providing an overview of the challenges and biases inherent to analyzing routine clinical care data, including health claims data (hereafter: routine data). Methods We conducted a literature search on routine data studies in four high-impact journals based on the Journal Citation Reports (JCR) category “Medicine, General & Internal” as of 2022 and three oncology journals, covering articles published from January 2018 to October 2023. Articles were screened and categorized into three scenarios based on their potential to provide meaningful RWE: (1) Burden of Disease, (2) Safety and Risk Group Analysis, and (3) Treatment Comparison. Limitations of this type of data cited in the discussion sections were extracted and classified according to different bias types: main bias categories in non-randomized studies (information bias, reporting bias, selection bias, confounding) and additional routine data-specific challenges (i.e., operationalization, coding, follow-up, missing data, validation, and data quality). These classifications were then ranked by relevance in a focus group meeting of methodological experts. The search was pre-specified and registered in PROSPERO (CRD42023477616). Results In October 2023, 227 articles were identified, 69 were assessed for eligibility, and 39 were included in the review: 11 on the burden of disease, 17 on safety and risk group analysis, and 11 on treatment comparison. Besides typical biases in observational studies, we identified additional challenges specific to RWE frequently mentioned in the discussion sections. The focus group had varied opinions on the limitations of Safety and Risk Group Analysis and Treatment Comparison but agreed on the essential limitations for the Burden of Disease category. Conclusion This review provides a comprehensive overview of potential limitations and biases in analyzing routine data reported in recent high-impact journals. We highlighted key challenges that have high potential to impact analysis results, emphasizing the need for thorough consideration and discussion for meaningful inferences.

Background Peripartum depression is a common but potentially debilitating pregnancy complication. Mobile applications can be used to collect data throughout the pregnancy and postpartum period to improve understanding of early risk indicators. Aim This study aimed to improve understanding of why women drop out of a peripartum depression mHealth study, and how we can improve the app design. Method Participants who dropped out of the Mom2B study (n = 134) answered closed and open questions on their motives for dropping out of the study, suggestions for improvement, and preferred timeframe of the study. A mix of quantitative and qualitative strategies was used to analyze the responses. Results The most common reasons for discontinuation were lack of time, problems with or loss of the pregnancy, the use of other pregnancy applications, surveys being too lengthy, the app draining too much battery, and participants incorrectly believing that their answers were irrelevant for the study. Participants suggested fewer survey moments, more reminders, and a need for more unique content compared to commercially available apps. Conclusions Researcher who want to use mHealth designs in peripartum studies need to ensure that their study designs are as time-efficient as possible, remind participants about the study, manage expectations about the study and what is expected of participants throughout the study, design their apps to be attractive in a competitive market, and follow-up with participants who are excluded from the study due to pregnancy complications.

Time-to-event data are very common in medical applications. Regression models have been developed on such data especially in the field of survival analysis. Kernels are used to handle even more complicated and enormous quantities of medical data by injecting non-linearity into linear models. In this study, a Multiple Kernel Learning (MKL) method has been proposed to optimize survival outcomes under the Accelerated Failure Time (AFT) model, a useful alternative to the Proportional Hazards (PH) frailty model. In other words, a survival parametric regression framework has been presented for clinical data to effectively integrate kernel learning with AFT model using a gradient descent optimization strategy. This methodology involves applying four different parametric models, evaluated using 19 distinct kernels to extract the best fitting scenario. This culminated in a sophisticated strategy that combined these kernels through MKL. We conducted a comparison between the Frailty model and MKL due to their shared fundamental properties. The models were assessed using the Concordance index (C-index) and Brier score (B-score). Each model was tested on both a case study and a replicated/independent dataset. The outcomes showed that kernelization enhances the performance of the model, especially by combining selected kernels for MKL. Supplementary Information The online version contains supplementary material available at 10.1186/s12874-024-02455-4.

Background We aimed to develop and validate an algorithm for identifying women with polycystic ovary syndrome (PCOS) in the French national health data system. Methods Using data from the French national health data system, we applied the International Classification of Diseases (ICD-10) related diagnoses E28.2 for PCOS among women aged 18 to 43 years in 2021. Then, we developed an algorithm to identify PCOS using combinations of clinical criteria related to specific drugs claims, biological exams, international classification of Diseases (ICD-10) related diagnoses during hospitalization, and/or registration for long-term conditions. The sensitivity, specificity and positive predictive value (PPV) of different combinations of algorithm criteria were estimated by reviewing the medical records of the Department of Reproductive Medicine at a university hospital for the year 2022, comparing potential women identified as experiencing PCOS by the algorithms with a list of clinically registered women with or without PCOS. Results We identified 2,807 (0.01%) women aged 18 to 43 who received PCOS-related care in 2021 using the ICD-10 code for PCOS in the French National health database. By applying the PCOS algorithm to 349 women, the positive and negative predictive values were 0.90 (95%CI (83–95) and 0.93 (95%CI 0.90–0.96) respectively. The sensitivity of the PCOS algorithm was estimated at 0.85 (95%CI 0.77–0.91) and the specificity at 0.96 (95%CI 0.92–0.98). Conclusion The validity of the PCOS diagnostic algorithm in women undergoing reproductive health care was acceptable. Our findings may be useful for future studies on PCOS using administrative data on a national scale, or even on an international scale given the similarity of coding in this field.

Objective To assess whether the outcome generation true model could be identified from other candidate models for clinical practice with current conventional model performance measures considering various simulation scenarios and a CVD risk prediction as exemplar. Study design and setting Thousands of scenarios of true models were used to simulate clinical data, various candidate models and true models were trained on training datasets and then compared on testing datasets with 25 conventional use model performance measures. This consists of univariate simulation (179.2k simulated datasets and over 1.792 million models), multivariate simulation (728k simulated datasets and over 8.736 million models) and a CVD risk prediction case analysis. Results True models had overall C statistic and 95% range of 0.67 (0.51, 0.96) across all scenarios in univariate simulation, 0.81 (0.54, 0.98) in multivariate simulation, 0.85 (0.82, 0.88) in univariate case analysis and 0.85 (0.82, 0.88) in multivariate case analysis. Measures showed very clear differences between the true model and flip-coin model, little or none differences between the true model and candidate models with extra noises, relatively small differences between the true model and proxy models missing causal predictors. Conclusion The study found the true model is not always identified as the “outperformed” model by current conventional measures for binary outcome, even though such true model is presented in the clinical data. New statistical approaches or measures should be established to identify the casual true model from proxy models, especially for those in proxy models with extra noises and/or missing causal predictors.

Background The majority of phase 3 clinical trials are implemented in multiple sites or centres, which inevitably leads to a correlation between observations from the same site or centre. This correlation must be carefully considered in both the design and the statistical analysis to ensure an accurate interpretation of the results and reduce the risk of biased results. This scoping review aims to provide a detailed statistical method used to analyze data collected from multicentre HIV randomized controlled trials in the African region. Methods This review followed the methodological framework proposed by Arksey and O’Malley. We searched four databases (PubMed, EBSCOhost, Scopus, and Web of Science) and retrieved 977 articles, 34 of which were included in the review. Results Data charting revealed that the most used statistical methods for analysing HIV endpoints in multicentre randomized controlled trials in Africa were standard survival analysis techniques (24 articles [71%]). Approximately 47% of the articles used stratified analysis methods to account for variations across different sites. Out of 34 articles reviewed, only 6 explicitly considered intra-site correlation in the analysis. Conclusions Our scoping review provides insights into the statistical methods used to analyse HIV data in multicentre randomized controlled trials in Africa and highlights the need for standardized reporting of statistical methods.

Background Spinal pain affects up to 30% of school-age children and can interfere with various aspects of daily life, such as school attendance, physical function, and social life. Current assessment tools often rely on parental reporting which limits our understanding of how each child is affected by their pain. This study aimed to address this gap by developing MySpineData-Kids (“MiRD-Kids”), a tailored patient-reported questionnaire focusing on children with spinal pain in secondary care (Danish hospital setting). Methods The process and development of MiRD-Kids followed a structured, multi-phase approach targeted children in outpatient care. The first phase involved evidence-synthesis, expert consultations, and item formulation, resulting in the first version. The second phase involved pilot testing among pediatric spinal pain patients, leading to modifications for improved clarity and relevance. The third phase involved implementation at the Pediatric outpatient track at The Spine Centre of Southern Denmark, University Hospital of Southern Denmark. Results MiRD-Kids was based on selected items from seven questionnaires, encompassing 20 items across six domains. Pilot testing with 13 pediatric patients facilitated modifications and finalized the questionnaire. The questionnaire includes sections for parents/legal guardians and six domains for children covering pain, sleep, activities, trauma, concerns, and treatment, following the International Classification of Functioning, Disability, and Health (ICF). Implementation challenges were overcome within a 2-month period, resulting in the clinical questionnaire MiRD-Kids a comprehensive tool for assessing pediatric spinal pain in hospital outpatient settings. Conclusion MiRD-Kids is the first comprehensive questionnaire for children with spinal pain seen in outpatient caresetting and follows the ICF approach. It can support age-specific high-quality research and comprehensive clinical assessment of children aged 12 to 17 years, potentially, contributing to efforts aimed at mitigating the long-term consequences of spinal pain.

Background In this work, we implement a data-driven approach using an aggregation of several analytical methods to study the characteristics of COVID-19 daily infection and death time series and identify correlations and characteristic trends that can be corroborated to the time evolution of this disease. The datasets cover twelve distinct countries across six continents, from January 22, 2020 till March 1, 2022. This time span is partitioned into three windows: (1) pre-vaccine, (2) post-vaccine and pre-omicron (BA.1 variant), and (3) post-vaccine including post-omicron variant. This study enables deriving insights into intriguing questions related to the science of system dynamics pertaining to COVID-19 evolution. Methods We implement a set of several distinct analytical methods for: (a) statistical studies to estimate the skewness and kurtosis of the data distributions; (b) analyzing the stationarity properties of these time series using the Augmented Dickey-Fuller (ADF) tests; (c) examining co-integration properties for the non-stationary time series using the Phillips-Ouliaris (PO) tests; (d) calculating the Hurst exponent using the rescaled-range (R/S) analysis, along with the Detrended Fluctuation Analysis (DFA), for self-affinity studies of the evolving dynamical datasets. Results We notably observe a significant asymmetry of distributions shows from skewness and the presence of heavy tails is noted from kurtosis. The daily infection and death data are, by and large, nonstationary, while their corresponding log return values render stationarity. The self-affinity studies through the Hurst exponents and DFA exhibit intriguing local changes over time. These changes can be attributed to the underlying dynamics of state transitions, especially from a random state to either mean-reversion or long-range memory/persistence states. Conclusions We conduct systematic studies covering a widely diverse time series datasets of the daily infections and deaths during the evolution of the COVID-19 pandemic. We demonstrate the merit of a multiple analytics frameworks through systematically laying down a methodological structure for analyses and quantitatively examining the evolution of the daily COVID-19 infection and death cases. This methodology builds a capability for tracking dynamically evolving states pertaining to critical problems.

Background Graft loss is a major health concern for kidney transplant (KTx) recipients. It is of clinical interest to develop a prognostic model for both graft function, quantified by estimated glomerular filtration rate (eGFR), and the risk of graft failure. Additionally, the model should be dynamic in the sense that it adapts to accumulating longitudinal information, including time-varying at-risk population, predictor-outcome association, and clinical history. Finally, the model should also properly account for the competing risk by death with a functioning graft. A model with the features above is not yet available in the literature and is the focus of this research. Methods We built and internally validated a prediction model on 3,893 patients from the Wisconsin Allograft Recipient Database (WisARD) who had a functioning graft 6 months after kidney transplantation. The landmark analysis approach was used to build a proof-of-concept dynamic prediction model to address the aforementioned methodological issues: the prediction of graft failure, accounted for competing risk of death, as well as the future eGFR value, are updated at each post-transplant time. We used 21 predictors including recipient characteristics, donor characteristics, transplant-related and post-transplant factors, longitudinal eGFR, hospitalization, and rejection history. A sensitivity analysis explored a less conservative variable selection rule that resulted in a more parsimonious model with reduced predictors. Results For prediction up to the next 1 to 5 years, the model achieved high accuracy in predicting graft failure, with the AUC between 0.80 and 0.95, and moderately high accuracy in predicting eGFR, with the root mean squared error between 10 and 18 mL/min/1.73m2 and 70%-90% of predicted eGFR falling within 30% of the observed eGFR. The model demonstrated substantial accuracy improvement compared to a conventional prediction model that used only baseline predictors. Conclusion The model outperformed conventional prediction model that used only baseline predictors. It is a useful tool for patient counseling and clinical management of KTx and is currently available as a web app.

Background The prediction of coronavirus disease in 2019 (COVID-19) in broader regions has been widely researched, but for specific areas such as urban areas the predictive models were rarely studied. It may be inaccurate to apply predictive models from a broad region directly to a small area. This paper builds a prediction approach for small size COVID-19 time series in a city. Methods Numbers of COVID-19 daily confirmed cases were collected from November 1, 2022 to November 16, 2023 in Xuzhou city of China. Classical deep learning models including recurrent neural network (RNN), long and short-term memory (LSTM), gated recurrent unit (GRU) and temporal convolutional network (TCN) are initially trained, then RNN, LSTM and GRU are integrated with a new attention mechanism and transfer learning to improve the performance. Ten times ablation experiments are conducted to show the robustness of the performance in prediction. The performances among the models are compared by the mean absolute error, root mean square error and coefficient of determination. Results LSTM outperforms than others, and TCN has the worst generalization ability. Thus, LSTM is integrated with the new attention mechanism to construct an LSTMATT model, which improves the performance. LSTMATT is trained on the smoothed time series curve through frequency domain convolution augmentation, then transfer learning is adopted to transfer the learned features back to the original time series resulting in a TLLA model that further improves the performance. RNN and GRU are also integrated with the attention mechanism and transfer learning and their performances are also improved, but TLLA still performs best. Conclusions The TLLA model has the best prediction performance for the time series of COVID-19 daily confirmed cases, and the new attention mechanism and transfer learning contribute to improve the prediction performance in the flatten part and the jagged part, respectively.

Background Health services research often relies on secondary data, necessitating quality checks for completeness, validity, and potential errors before use. Various methods address implausible data, including data elimination, statistical estimation, or value substitution from the same or another dataset. This study presents an internal validation process of a secondary dataset used to investigate hospital compliance with minimum caseload requirements (MCR) in Germany. The secondary data source validated is the German Hospital Quality Reports (GHQR), an official dataset containing structured self-reported data from all hospitals in Germany. Methods This study conducted an internal cross-field validation of MCR-related data in GHQR from 2016 to 2021. The validation process checked the validity of reported MCR caseloads, including data availability and consistency, by comparing the stated MCR caseload with further variables in the GHQR. Subsequently, implausible MCR caseload values were corrected using the most plausible values given in the same GHQR. The study also analysed the error sources and used reimbursement-related Diagnosis Related Groups Statistic data to assess the validation outcomes. Results The analysis focused on four MCR procedures. 11.8–27.7% of the total MCR caseload values in the GHQR appeared ambiguous, and 7.9–23.7% were corrected. The correction added 0.7–3.7% of cases not previously stated as MCR caseloads and added 1.5–26.1% of hospital sites as MCR performing hospitals not previously stated in the GHQR. The main error source was this non-reporting of MCR caseloads, especially by hospitals with low case numbers. The basic plausibility control implemented by the Federal Joint Committee since 2018 has improved the MCR-related data quality over time. Conclusions This study employed a comprehensive approach to dataset internal validation that encompassed: (1) hospital association level data, (2) hospital site level data and (3) medical department level data, (4) report data spanning six years, and (5) logical plausibility checks. To ensure data completeness, we selected the most plausible values without eliminating incomplete or implausible data. For future practice, we recommend a validation process when using GHQR as a data source for MCR-related research. Additionally, an adapted plausibility control could help to improve the quality of MCR documentation.

Background Autoimmune disorders have primary manifestations such as joint pain and bowel inflammation but can also have secondary manifestations such as non-infectious uveitis (NIU). A regulatory health authority raised concerns after receiving spontaneous reports for NIU following exposure to Remicade®, a biologic therapy with multiple indications for which alternative therapies are available. In assessment of this clinical question, we applied validity diagnostics to support observational data causal inferences. Methods We assessed the risk of NIU among patients exposed to Remicade® compared to alternative biologics. Five databases, four study populations, and four analysis methodologies were used to estimate 80 potential treatment effects, with 20 pre-specified as primary. The study populations included inflammatory bowel conditions Crohn’s disease or ulcerative colitis (IBD), ankylosing spondylitis (AS), psoriatic conditions plaque psoriasis or psoriatic arthritis (PsO/PsA), and rheumatoid arthritis (RA). We conducted four analysis strategies intended to address limitations of causal estimation using observational data and applied four diagnostics with pre-specified quantitative rules to evaluate threats to validity from observed and unobserved confounding. We also qualitatively assessed post-propensity score matching representativeness, and bias susceptibility from outcome misclassification. We fit Cox proportional-hazards models, conditioned on propensity score-matched sets, to estimate the on-treatment risk of NIU among Remicade® initiators versus alternatives. Estimates from analyses that passed four validity tests were assessed. Results Of the 80 total analyses and the 20 analyses pre-specified as primary, 24% and 20% passed diagnostics, respectively. Among patients with IBD, we observed no evidence of increased risk for NIU relative to other similarly indicated biologics (pooled hazard ratio [HR] 0.75, 95% confidence interval [CI] 0.38–1.40). For patients with RA, we observed no increased risk relative to similarly indicated biologics, although results were imprecise (HR: 1.23, 95% CI 0.14–10.47). Conclusions We applied validity diagnostics on a heterogenous, observational setting to answer a specific research question. The results indicated that safety effect estimates from many analyses would be inappropriate to interpret as causal, given the data available and methods employed. Validity diagnostics should always be used to determine if the design and analysis are of sufficient quality to support causal inferences. The clinical implications of our findings on IBD suggests that, if an increased risk exists, it is unlikely to be greater than 40% given the 1.40 upper bound of the pooled HR confidence interval.

Background The aim of this study is to develop a method we call “cost mining” to unravel cost variation and identify cost drivers by modelling integrated patient pathways from primary care to the palliative care setting. This approach fills an urgent need to quantify financial strains on healthcare systems, particularly for colorectal cancer, which is the most expensive cancer in Australia, and the second most expensive cancer globally. Methods We developed and published a customized algorithm that dynamically estimates and visualizes the mean, minimum, and total costs of care at the patient level, by aggregating activity-based healthcare system costs (e.g. DRGs) across integrated pathways. This extends traditional process mining approaches by making the resulting process maps actionable and informative and by displaying cost estimates. We demonstrate the method by constructing a unique dataset of colorectal cancer pathways in Victoria, Australia, using records of primary care, diagnosis, hospital admission and chemotherapy, medication, health system costs, and life events to create integrated colorectal cancer patient pathways from 2012 to 2020. Results Cost mining with the algorithm enabled exploration of costly integrated pathways, i.e. drilling down in high-cost pathways to discover cost drivers, for 4246 cases covering approx. 4 million care activities. Per-patient CRC pathway costs ranged from $10,379 AUD to$41,643 AUD, and varied significantly per cancer stage such that e.g. chemotherapy costs in one cancer stage are different to the same chemotherapy regimen in a different stage. Admitted episodes were most costly, representing 93.34% or $56.6 M AUD of the total healthcare system costs covered in the sample. Conclusions Cost mining can supplement other health economic methods by providing contextual, sequence and timing-related information depicting how patients flow through complex care pathways. This approach can also facilitate health economic studies informing decision-makers on where to target care improvement or to evaluate the consequences of new treatments or care delivery interventions. Through this study we provide an approach for hospitals and policymakers to leverage their health data infrastructure and to enable real time patient level cost mining.

Background Missing observations within the univariate time series are common in real-life and cause analytical problems in the flow of the analysis. Imputation of missing values is an inevitable step in every incomplete univariate time series. Most of the existing studies focus on comparing the distributions of imputed data. There is a gap of knowledge on how different imputation methods for univariate time series affect the forecasting performance of time series models. We evaluated the prediction performance of autoregressive integrated moving average (ARIMA) and long short-term memory (LSTM) network models on imputed time series data using ten different imputation techniques. Methods Missing values were generated under missing completely at random (MCAR) mechanism at 10%, 15%, 25%, and 35% rates of missingness using complete data of 24-h ambulatory diastolic blood pressure readings. The performance of the mean, Kalman filtering, linear, spline, and Stineman interpolations, exponentially weighted moving average (EWMA), simple moving average (SMA), k-nearest neighborhood (KNN), and last-observation-carried-forward (LOCF) imputation techniques on the time series structure and the prediction performance of the LSTM and ARIMA models were compared on imputed and original data. Results All imputation techniques either increased or decreased the data autocorrelation and with this affected the forecasting performance of the ARIMA and LSTM algorithms. The best imputation technique did not guarantee better predictions obtained on the imputed data. The mean imputation, LOCF, KNN, Stineman, and cubic spline interpolations methods performed better for a small rate of missingness. Interpolation with EWMA and Kalman filtering yielded consistent performances across all scenarios of missingness. Disregarding the imputation methods, the LSTM resulted with a slightly better predictive accuracy among the best performing ARIMA and LSTM models; otherwise, the results varied. In our small sample, ARIMA tended to perform better on data with higher autocorrelation. Conclusions We recommend to the researchers that they consider Kalman smoothing techniques, interpolation techniques (linear, spline, and Stineman), moving average techniques (SMA and EWMA) for imputing univariate time series data as they perform well on both data distribution and forecasting with ARIMA and LSTM models. The LSTM slightly outperforms ARIMA models, however, for small samples, ARIMA is simpler and faster to execute.

Background The connection between participants and their research team can affect how safe, informed, and respected a participant feels, and their willingness to complete a research project. Communication between researchers and participants is key to developing this connection, but there is little published work evaluating how communication during clinical research is conducted. Purpose This paper explores what communications happen (and how) with research participants in Australia post consenting to participate in clinical research. It provides reflections from Australians working in clinical research about their current strategies, or those they would like to use, to communicate with research participants. Methods This exploratory, qualitative descriptive study reports findings associated with twenty semi-structured interviews that were undertaken with people who work in clinical research in Australia (such as staff in participant facing, site management, or sponsor representative roles). These interviews were conducted and analysed inductively using thematic analysis. Findings Research staff reported using a range of communication strategies which varied in implementation, uptake, and suitability between clinical research studies and sites. Four major themes were identified in the interviews: [1] staff use innovative pragmatism to communicate; [2] staff tailor the communication strategies to fit the participants’ context; [3] the site, its systems, and staff training all impact communication; [4] successful communication requires collaboration between stakeholders. Conclusion There are a variety of communication strategies, methods and activities research staff currently employ with trial participants, which vary in purpose, method, resources required, and suitability between studies and sites. Thorough consideration of the participants’ contexts and the capacity of research sites is crucial for the design of studies which allow for effective communication between the research team and participants. The authors encourage those developing clinical research projects to involve site staff and consumer representatives early in planning for communication with participants.

Background Undetected atrial fibrillation (AF) poses a significant risk of stroke and cardiovascular mortality. However, diagnosing AF in real-time can be challenging as the arrhythmia is often not captured instantly. To address this issue, a deep-learning model was developed to diagnose AF even during periods of arrhythmia-free windows. Methods The proposed method introduces a novel approach that integrates clinical data and electrocardiograms (ECGs) using a colorization technique. This technique recolors ECG images based on patients' demographic information while preserving their original characteristics and incorporating color correlations from statistical data features. Our primary objective is to enhance atrial fibrillation (AF) detection by fusing ECG images with demographic data for colorization. To ensure the reliability of our dataset for training, validation, and testing, we rigorously maintained separation to prevent cross-contamination among these sets. We designed a Dual-input Mixed Neural Network (DMNN) that effectively handles different types of inputs, including demographic and image data, leveraging their mixed characteristics to optimize prediction performance. Unlike previous approaches, this method introduces demographic data through color transformation within ECG images, enriching the diversity of features for improved learning outcomes. Results The proposed approach yielded promising results on the independent test set, achieving an impressive AUC of 83.4%. This outperformed the AUC of 75.8% obtained when using only the original signal values as input for the CNN. The evaluation of performance improvement revealed significant enhancements, including a 7.6% increase in AUC, an 11.3% boost in accuracy, a 9.4% improvement in sensitivity, an 11.6% enhancement in specificity, and a substantial 25.1% increase in the F1 score. Notably, AI diagnosis of AF was associated with future cardiovascular mortality. For clinical application, over a median follow-up of 71.6 ± 29.1 months, high-risk AI-predicted AF patients exhibited significantly higher cardiovascular mortality (AF vs. non-AF; 47 [18.7%] vs. 34 [4.8%]) and all-cause mortality (176 [52.9%] vs. 216 [26.3%]) compared to non-AF patients. In the low-risk group, AI-predicted AF patients showed slightly elevated cardiovascular (7 [0.7%] vs. 1 [0.3%]) and all-cause mortality (103 [9.0%] vs. 26 [6.4%]) than AI-predicted non-AF patients during six-year follow-up. These findings underscore the potential clinical utility of the AI model in predicting AF-related outcomes. Conclusions This study introduces an ECG colorization approach to enhance atrial fibrillation (AF) detection using deep learning and demographic data, improving performance compared to ECG-only methods. This method is effective in identifying high-risk and low-risk populations, providing valuable features for future AF research and clinical applications, as well as benefiting ECG-based classification studies.

Background Propensity score matching has become a popular method for estimating causal treatment effects in non-randomized studies. However, for time-to-event outcomes, the estimation of hazard ratios based on propensity scores can be challenging if omitted or unobserved covariates are present. Not accounting for such covariates could lead to treatment estimates, differing from the estimate of interest. However, researchers often do not know whether (and, if so, which) covariates will cause this divergence. Methods To address this issue, we extended a previously described method, Dynamic Landmarking, which was originally developed for randomized trials. The method is based on successively deletion of sorted observations and gradually fitting univariable Cox models. In addition, the balance of observed, but omitted covariates can be measured by the sum of squared z-differences. Results By simulation we show, that Dynamic Landmarking provides a good visual tool for detecting and distinguishing treatment effect estimates underlying built-in selection or confounding bias. We illustrate the approach with a data set from cardiac surgery and provide some recommendations on how to use and interpret Dynamic Landmarking in propensity score matched studies. Conclusion Dynamic Landmarking is a useful post-hoc diagnosis tool for visualizing whether an estimated hazard ratio could be distorted by confounding or built-in selection bias.

Background The choice of a single primary outcome in randomised trials can be difficult, especially in mental health where interventions may be complex and target several outcomes simultaneously. We carried out a systematic review to assess the quality of the analysis and reporting of multiple outcomes in mental health RCTs, comparing approaches with current CONSORT and other regulatory guidance. Methods The review included all late-stage mental health trials published between 1st January 2019 to 31st December 2020 in 9 leading medical and mental health journals. Pilot and feasibility trials, non-randomised trials, and early phase trials were excluded. The total number of primary, secondary and other outcomes was recorded, as was any strategy used to incorporate multiple primary outcomes in the primary analysis. Results There were 147 included mental health trials. Most trials (101/147) followed CONSORT guidance by specifying a single primary outcome with other outcomes defined as secondary and analysed in separate statistical analyses, although a minority (10/147) did not specify any outcomes as primary. Where multiple primary outcomes were specified (33/147), most (26/33) did not correct for multiplicity, contradicting regulatory guidance. The median number of clinical outcomes reported across studies was 8 (IQR 5–11 ). Conclusions Most trials are correctly following CONSORT guidance. However, there was little consideration given to multiplicity or correlation between outcomes even where multiple primary outcomes were stated. Trials should correct for multiplicity when multiple primary outcomes are specified or describe some other strategy to address the multiplicity. Overall, very few mental health trials are taking advantage of multiple outcome strategies in the primary analysis, especially more complex strategies such as multivariate modelling. More work is required to show these exist, aid interpretation, increase efficiency and are easily implemented. Registration Our systematic review protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) on 11th January 2023 (CRD42023382274).

Background In cohort studies with time-to-event outcomes, covariates of interest often have values that change over time. The classical Cox regression model can handle time-dependent covariates but assumes linear effects on the log hazard function, which can be limiting in practice. Furthermore, when multiple correlated covariates are studied, it is of great interest to model their joint effects by allowing a flexible functional form and to delineate their relative contributions to survival risk. Methods Motivated by the World Trade Center (WTC)-exposed Fire Department of New York cohort study, we proposed a partial-linear single-index Cox (PLSI-Cox) model to investigate the effects of repeatedly measured metabolic syndrome indicators on the risk of developing WTC lung injury associated with particulate matter exposure. The PLSI-Cox model reduces the dimensionality of covariates while providing interpretable estimates of their effects. The model’s flexible link function accommodates nonlinear effects on the log hazard function. We developed an iterative estimation algorithm using spline techniques to model the nonparametric single-index component for potential nonlinear effects, followed by maximum partial likelihood estimation of the parameters. Results Extensive simulations showed that the proposed PLSI-Cox model outperformed the classical time-dependent Cox regression model when the true relationship was nonlinear. When the relationship was linear, both the PLSI-Cox model and classical time-dependent Cox regression model performed similarly. In the data application, we found a possible nonlinear joint effect of metabolic syndrome indicators on survival risk. Among the different indicators, BMI had the largest positive effect on the risk of developing lung injury, followed by triglycerides. Conclusion The PLSI-Cox models allow for the evaluation of nonlinear effects of covariates and offer insights into their relative importance and direction. These methods provide a powerful set of tools for analyzing data with multiple time-dependent covariates and survival outcomes, potentially offering valuable insights for both current and future studies.

Background Treatment switching in randomized clinical trials introduces challenges in performing causal inference. Intention To Treat (ITT) analyses often fail to fully capture the causal effect of treatment in the presence of treatment switching. Consequently, decision makers may instead be interested in causal effects of hypothetical treatment strategies that do not allow for treatment switching. For example, the phase 3 ALTA-1L trial showed that brigatinib may have improved Overall Survival (OS) compared to crizotinib if treatment switching had not occurred. Their sensitivity analysis using Inverse Probability of Censoring Weights (IPCW), reported a Hazard Ratio (HR) of 0.50 (95% CI, 0.28-0.87), while their initial ITT analysis estimated an HR of 0.81 (0.53-1.22). Methods We used a directed acyclic graph to depict the clinical setting of the ALTA-1L trial in the presence of treatment switching, illustrating the concept of treatment-confounder feedback and highlighting the need for g-methods. In a re-analysis of the ALTA-1L trial data, we used IPCW and the parametric g-formula to adjust for baseline and time-varying covariates to estimate the effect of two hypothetical treatment strategies on OS: “always treat with brigatinib” versus “always treat with crizotinib”. We conducted various sensitivity analyses using different model specifications and weight truncation approaches. Results Applying the IPCW approach in a series of sensitivity analyses yielded Cumulative HRs (cHRs) ranging between 0.38 (0.12, 0.98) and 0.73 (0.45,1.22) and Risk Ratios (RRs) ranging between 0.52 (0.32, 0.98) and 0.79 (0.54,1.17). Applying the parametric g-formula resulted in cHRs ranging between 0.61 (0.38,0.91) and 0.72 (0.43,1.07) and RRs ranging between 0.71 (0.48,0.94) and 0.79 (0.54,1.05). Conclusion Our results consistently indicated that our estimated ITT effect estimate (cHR: 0.82 (0.51,1.22) may have underestimated brigatinib’s benefit by around 10-45 percentage points (using IPCW) and 10-20 percentage points (using the parametric g-formula) across a wide range of model choices. Our analyses underscore the importance of performing sensitivity analyses, as the result from a single analysis could potentially stand as an outlier in a whole range of sensitivity analyses. Trial registration Clinicaltrials.gov Identifier: NCT02737501 on April 14, 2016.

Background Liver injury from drug-drug interactions (DDIs), notably with anti-tuberculosis drugs such as isoniazid, poses a significant safety concern. Electronic medical records contain comprehensive clinical information and have gained increasing attention as a potential resource for DDI detection. However, a substantial portion of adverse drug reaction (ADR) information is hidden in unstructured narrative text, which has yet to be efficiently harnessed, thereby introducing bias into the research. There is a significant need for an efficient framework for the DDI assessment. Methods Using a Chinese natural language processing (NLP) model, we extracted 25,130 adverse drug reaction (ADR) records, dividing them into sets for training an automated normalization model. The trained models, in conjunction with liver function laboratory tests, were used to thoroughly and efficiently identify liver injury cases. Ultimately, we applied a case-control study design to detect DDI signals increasing isoniazid’s liver injury risk. Results The Logistic Regression model demonstrated stable and superior performance in classification task. Based on laboratory criteria and NLP, we identified 128 liver injury cases among a cohort of 3,209 patients treated with isoniazid. Preliminary screening of 113 drug combinations with isoniazid highlighted 20 potential signal drugs, with antibacterials constituting 25%. Sensitivity analysis confirmed the robustness of signal drugs, especially in cardiac therapy and antibacterials. Conclusion Our NLP and machine learning approach effectively identifies isoniazid-related DDIs that increase the risk of liver injury, identifying 20 signal drugs, mainly antibacterials. Further research is required to validate these DDI signals.

Background A recent systematic review revealed issues in regard to performing and reporting agreement and reliability studies for ordinal scales, especially in the presence of more than two observers. This paper therefore aims to provide all necessary information in regard to the choice among the most meaningful and most used measures and the planning of agreement and reliability studies for ordinal outcomes. Methods This paper considers the generalisation of the proportion of (dis)agreement, the mean absolute deviation, the mean squared deviation and weighted kappa coefficients to more than two observers in the presence of an ordinal outcome. Results After highlighting the difference between the concepts of agreement and reliability, a clear and simple interpretation of the agreement and reliability coefficients is provided. The large sample variance of the various coefficients with the delta method is presented or derived if not available in the literature to construct Wald confidence intervals. Finally, a procedure to determine the minimum number of raters and patients needed to limit the uncertainty associated with the sampling process is provided. All the methods are available in an R package and a Shiny application to circumvent the limitations of current software. Conclusions The present paper completes existing guidelines, such as the Guidelines for Reporting Reliability and Agreement Studies (GRRAS), to improve the quality of reliability and agreement studies of clinical tests. Furthermore, we provide open source software to researchers with minimum programming skills.

Background The aim of the study was to investigate the development of evidence-based monitoring strategies in a population with progressive or recurrent disease. A simulation study of monitoring strategies using a new biomarker (ELF) for the detection of liver cirrhosis in people with known liver fibrosis was undertaken alongside a randomised controlled trial (ELUCIDATE). Methods Existing data and expert opinion were used to estimate the progression of disease and the performance of repeat testing with ELF. Knowledge of the true disease status in addition to the observed test results for a cohort of simulated patients allowed various monitoring strategies to be implemented, evaluated and validated against trial data. Results Several monitoring strategies ranging in complexity were successfully modelled and compared regarding the timing of detection of disease, the duration of monitoring, and the predictive value of a positive test result. The results of sensitivity analysis showed the importance of accurate data to inform the simulation. Results of the simulation were similar to those from the trial. Conclusion Monitoring data can be simulated and strategies compared given adequate knowledge of disease progression and test performance. Such exercises should be carried out to ensure optimal strategies are evaluated in trials thus reducing research waste. Monitoring data can be generated and monitoring strategies can be assessed if data is available on the monitoring test performance and the test variability. This work highlights the data necessary and the general method for evaluating the performance of monitoring strategies, allowing appropriate strategies to be selected for evaluation. Modelling work should be conducted prior to full scale investigation of monitoring strategies, allowing optimal monitoring strategies to be assessed.