No full-text available
To read the full-text of this research,
you can request a copy directly from the authors.
Data quality: “Garbage in – garbage out”
... However, due to limitations in information processing capacity, making optimal decisions can be challenging, if not impossible, for most people (Lachman et al., 2015). This challenge is exacerbated by the "garbage in, garbage out" effect, where low information quality further constrains decision-making capabilities (Kilkenny and Robinson, 2018). ...
... Consequently, there is a pressing need for high-quality information, necessitating an appropriate measurement of information quality. Since the 1980s, numerous researchers have endeavored to explore information quality and have developed various information quality models (Bailey and Pearson, 1983;DeLone and McLean, 1992;Jarke and Vassiliou, 1997;Kilkenny and Robinson, 2018;Miller, 1996;Wang and Strong, 1996). For example, Bailey and Pearson (1983) assessed information quality using 39 indicators, including timeliness, completeness, conciseness, format, and relevance. ...
Purpose
This study aims to investigate the role of information normalization in online healthcare consultation, a typical complex human-to-human communication requiring both effectiveness and efficiency. The globalization and digitization trend calls for high-quality information, and normalization is considered an effective method for improving information quality. Meanwhile, some researchers argued that excessive normalization (standardized answers) may be perceived as impersonal, repetitive, and cold. Thus, it is not appreciated for human-to-human communication, for instance, when patients are anxious about their health condition (e.g. with high-risk disease) in online healthcare consultation. Therefore, the role of information normalization in human communication is worthy to be explored.
Design/methodology/approach
Data were collected from one of the largest online healthcare consultation platforms (Dxy.com). This study expanded the existing information quality model by introducing information normalization as a new dimension. Information normalization was assessed using medical templates, extracted through natural language processing methods such as Bidirectional Encoder Representations from Transformers (BERT) and Latent Dirichlet Allocation (LDA). Patient decision-making behaviors, namely, consultant selection and satisfaction, were chosen to evaluate communication performance.
Findings
The results confirmed the positive impact of information normalization on communication performance. Additionally, a negative moderating effect of disease risk on the relationship between information normalization and patient decision-making was identified. Furthermore, the study demonstrated that information normalization can be enhanced through experiential learning.
Originality/value
These findings highlighted the significance of information normalization in online healthcare communication and extended the existing information quality model. It also facilitated patient decision-making on online healthcare platforms by providing a comprehensive information quality measurement. In addition, the moderating effects indicated the contradiction between informational support and emotional support, enriching the social support theory.
... The effectiveness of these algorithms depends heavily on the quality of the underlying data (Kilkenny and Robinson, 2018), and many organisations struggle to identify the most effective predictors and data sources for their turnover models. Over the past 15 years, employee performance has emerged as a particularly prominent focus in turnover prediction research (Alferaih, 2017;Yücel, 2021;Bui et al., 2024); this has prompted scholars to incorporate performance metrics into their models (Goodwin et al., 2011;Speer, 2021). ...
Purpose
Predicting employee turnover is a major challenge for organisations. While this topic has been studied for over a century, most research relies on specially collected data that may not reflect the data companies can access. As a result, many business applications fail due to data limitations or legal restrictions. This study aims to explore how including employee performance data can improve the accuracy of turnover predictions.
Design/methodology/approach
The authors analysed data from 1,518 sales employees in Germany, Switzerland and Austria, including human resource (HR) records, employee satisfaction surveys and performance data. A machine learning model was used to predict employee turnover.
Findings
The results show that turnover prediction is most accurate when performance data is included, with an accuracy score of 0.8998. Models without performance data perform significantly worse, which highlights the strong impact of performance data on predicting employee turnover.
Originality/value
This study contributes to turnover research by demonstrating and quantifying how employee performance metrics improve prediction accuracy. Unlike many studies that rely on artificial data sets, the authors use real-world company data and can thus offer insights that are relevant to HR professionals and business leaders.
... Overall, this findings is already know and likely stems from fragmented legacy systems and a lack of standardization in data formats within manufacturing environments (Meyer et al., 2018). The implication here is that companies may need to prioritize foundational investments in data management infrastructure before pursuing advanced AIS initiatives that possibly lead to cost-intensive issues, i.e., being aware of the "garbage-in garbage-out" principle (Kilkenny and Robinson, 2018). We argue, that automated, strictly verified tools for data preprocessing, cleaning, and enrichment could become increasingly important to address this bottleneck (Kovalenko et al., 2023), as also stated by one participant: "[. . . ...
Artificial-Intelligence Systems (AIS) are reshaping manufacturing by optimizing processes, enhancing efficiency , and reducing costs. Despite this potential, their adoption in practice remains challenging due to limited understanding of technological complexities and practical hurdles. In this study, we present findings of a survey involving 26 manufacturing AIS practitioners, highlighting key challenges, strategies for implementing AIS more effectively, and perceived added value. Data preparation, deployment, operation, and change management were identified as the most critical phases, emphasizing the need for robust data management and scalable, modular (i.e., configurable) solutions. Predictive maintenance, driven by supervised learning, dominates current AIS, aligning with industry goals to reduce downtime and improve productivity. Despite the benefits, broader applications, such as real-time optimization and advanced quality control, seem to remain underutilized. Overall, the study aims to provide insights for both practitioners and researchers, emphasizing the importance of overcoming these barriers to facilitate the adoption of AIS in advanced manufacturing.
... Practitioners integrating GenAI into collaborative humancentered innovation must invest significant effort in data management. Indeed, the old adage "garbage in, garbage out" resonates strongly with the use of GenAI in design sprints (Kilkenny and Robinson 2018). High-quality data inputs are essential to generating better outputs that enhance the problemsolving process. ...
Organisations across various industries are still exploring the potential of Generative Artificial Intelligence (GenAI) to automate a variety of knowledge work processes, including managing innovation. While innovation is often viewed as a product of individual creativity, it more commonly unfolds through a collaborative process where creativity intertwines with knowledge. However, the extent and effectiveness of GenAI in supporting this process remain open questions. Our study investigates this issue using a collaborative practice research approach focused on three GenAI‐enabled innovation projects conducted within different organisations. We explored how, why, and when GenAI could effectively be integrated into design sprints—a highly structured, collaborative process enabling human‐centred innovation. Our research identified challenges and opportunities in synchronising AI capabilities with human intelligence and creativity. To translate these insights into practical strategies, we propose four recommendations for organisations eager to leverage GenAI to both streamline and bring more value to their innovation processes: (1) establish a collaborative intelligence value loop with GenAI; (2) build trust in GenAI; (3) develop robust data collection and curation workflows; and (4) embrace a craftsman's discipline.
... Further limitations arise from the fidelity of input data and the physical plausibility of resampled scenarios (Kilkenny and Robinson, 2018). ML-WN assumes meteorological variables can be independently perturbed, yet real-world weather systems exhibit tightly coupled dynamics (e.g., temperature-humidity relationships, land-sea breeze cycles). ...
Air pollution causes millions of premature deaths annually, driving widespread implementation of clean air interventions. Quantitative evaluation of the efficacy of such interventions is critical in air quality management. Machine learning-based weather normalization (ML-WN) has been employed to isolate meteorological influences from emission-drive changes; however, it has its own limitations, particularly when abrupt emission shifts occur, e.g., after an intervention. Here we developed a logical evaluation framework, based on paired observational datasets and a test of ‘ML algebra’ (i.e., the ‘commutation’ of a normalisation step), to show that ML-WN significantly underestimates the immediate effects of short-term interventions on NOX, with discrepancies reaching up to 42 % for one-week interventions. This finding challenges assumptions about the robustness of ML-WN for evaluating short-term policies, such as emergency traffic controls or episodic pollution events. We propose a refined approach (MacLeWN) that explicitly accounts for intervention timing, reducing underestimation biases by >90 % in idealised but plausible cases studies. We applied both approaches to evaluate the impact of COVID-19 lockdown on NOX as measured at Marylebone Road, London. For the one-week period after the lockdown, ML-WN estimates approximately 17 % smaller NOX reductions compared to MacLeWN, and such underestimation diminishes as policy duration extends, decreasing to ~10 % for one-month and becoming insignificant after three months. Our findings indicate the importance of carefully selecting evaluation methodologies for air quality interventions, suggesting that ML-WN should be complemented or adjusted when assessing short-term policies. Increasing model interpretability is also crucial for generating trustworthy assessments and improving policy evaluations.
... Data quality is closely linked to radiomic features repeatability and reproducibility ("Garbage In, Garbage Out") [24]. These features may be influenced, for example, by the quality of the input images determined by multiple factors related to image acquisition. ...
Radiomics and artificial intelligence (AI) are rapidly evolving, significantly transforming the field of medical imaging. Despite their growing adoption, these technologies remain challenging to approach due to their technical complexity. This review serves as a practical guide for early-career radiologists and researchers seeking to integrate radiomics into their studies. It provides practical insights for clinical and research applications, addressing common challenges, limitations, and future directions in the field. This work offers a structured overview of the essential steps in the radiomics workflow, focusing on concrete aspects of each step, including indicative and practical examples. It covers the main steps such as dataset definition, image acquisition and preprocessing, segmentation, feature extraction and selection, and AI model training and validation. Different methods to be considered are discussed, accompanied by summary diagrams. This review equips readers with the knowledge necessary to approach radiomics and AI in medical imaging from a hands-on research perspective.
... Even though it was possible to apply a meta-regression, due to the small number of studies and effects, the results must be interpreted with caution. "Garbage in-garbage out" [107] fits very well for meta-analytical effect size pooling, so there is a paramount requirement for high-quality stretching research on RE to improve the current scientific evidence. ...
Background
Running economy (RE) determines the performance of endurance athletes. While stretching has been practised for decades, and is still one common integral component of warm-up routine, muscle stretching is also associated with decreased stiffness. For RE energy storage in the tendons which is accompanied with stiffness is of crucial importance. In turn, avoidance of pre-running stretching was frequently recommended. Although some studies supported this recommendation, the evidence is controversial. Nevertheless, yet, no systematic review on the effects of stretching on RE with effect size (ES) quantification was performed. Consequently, with this systematic review with meta-analysis, we aim to provide the first overview on this topic.
Methods
In adherence to PRISMA 2020 guidelines, we meta-analyzed effect sizes from three databases using PICOS guidelines on stretching effects on RE in healthy participants using robust variance estimation. Heterogeneity was reduced using subgroup analyses while meta-regression evaluated whether running velocity potentially moderates results. Risk of Bias was assessed using the PEDro scale, certainty of evidence was classified via GRADE working group criteria. The study protocol was registered in Open Science Framework https://doi.org/10.17605/OSF.IO/MA8D4).
Results
Overall, low certainty of evidence pooled from 15 studies with a total of 181 participants indicated that stretching did not significantly moderate RE acutely (p = 0.21–0.65), neither in general, nor were there any stretching types (dynamic, static and proprioceptive neuromuscular facilitation) that affected this result. Due to the limited number of chronic studies found in the literature, long-term stretching effects were exclusively evaluated qualitatively. Meaningful heterogeneity and reduced methodological quality (PEDro Score: 4.88, fair) contributed to certainty of evidence downgrading.
Conclusions
In contrast to common beliefs that stretching decreased stiffness parameters and would therefore hamper RE, current evidence does not support any effect of stretching on RE in running athletes. However, several flaws such as no investigation of the underlying mechanisms (e.g., stiffness), small sample sizes, determining RE at different velocities, and the implementation of unreasonable stretching durations strongly biased conclusions. Especially on chronic effects there is a large demand for improved evidence, including underlying mechanisms investigation. Yet, it seems unreasonable to avoid pre-running stretching to prevent RE decreases.
... Tumour documentation data are derived from routinely collected information. The utilisation of routinely collected data carries a risk highlighted by the "garbage-in, garbageout" phenomenon, which posits that the quality of output is contingent upon the quality of input within any given system (Kilkenny and Robinson, 2018). Additionally, transcription errors represent another potential source of inaccuracies in data entries, manifesting as errors that occur during the manual transcribing of information from unstructured sources into structured tumour documentation systems (Feng et al., 2020). ...
Background
Accurate documentation of tumours presents significant opportunities for advancing cancer research and improving patient care, yet it also poses challenges for healthcare management.
Objective
This study aimed to assess the effectiveness and resource implications of source data verification (SDV) in enhancing the quality of tumour documentation data, focusing on accuracy, completeness and correctness.
Method
Using tumour documentation data from a large German University Hospital, an SDV was conducted by an external audit group (group RE), comparing the data initially documented by the centre’s tumour documentalists (group TD) to available source documents for the years 2016–2020. The analysis set comprised 240 cases, with exemplary data fields strategically selected across various organ entities and other tumour features. Identified errors were cross-validated by a third group (group CO).
Results
Visualisations depicted error frequencies by diagnosis year and organ entity. Potential errors were identified, providing feedback to the tumour documentation unit. However, uncertainties in error identification raised questions about the efficacy of SDV.
Conclusion
While effective in identifying errors, SDV faced challenges due to ambiguous source data and potential bias from external auditors, as well as being deemed uneconomical. The study suggests SDVs suitability for small sample validation but questions its scalability for large datasets.
Implications for health information management
Alternative methods, such as data exchange interfaces to subsystems or plausibility checks, are recommended for enhancing data quality. This study emphasises the need to explore alternatives for improving data quality in tumour documentation.
... Notwithstanding the multiple important stages of a Machine Learning model, data collection and processing remain the cornerstone of any Data science project. Kilkenny and Robinson point out that the quality of input data directly influences output results (Kilkenny & Robinson, 2018). To this end, we start with a data collection strategy and the creation of a dataset. ...
Nowadays, the need for a powerful model in the task of accurate image classification is increasing. This paper focuses on evaluating and comparing the effectiveness of three main neural network models namely the multilayer perceptron model, convolutional neural networks and the transfer learning model for the classification of mineral and rock images in the mining region of Katanga. The main problem addressed is the identification of the most efficient and accurate machine learning techniques in the specific classification of mineral images, with a particular focus on the constitution of the dataset, the analysis of mineral image data and their associated labels. Parameters such as the burial lot, the number of cycles, the size of convolution filters, the precision and the loss have been taken into account. The results show that the transfer learning-based model significantly outperforms the multilayer perceptron models and convolutional neural networks in terms of accuracy and robustness, achieving a classification accuracy of 97.8% compared to 75% for the multilayer perceptron and 96% for the convolutional neural networks designed from scratch. These remarkable results demonstrate the importance of deep learning in processing complex images and open new perspectives for the use of these techniques in the mining sector of the Greater Katanga mining region in the identification of mineral resources. The broader implications of this study include an innovation in mining exploration strategies through faster and more accurate classification of minerals, thus influencing both economic decision-making and environmental policies associated with mining in the region.
... La qualité des modèles d'IA dépend directement de leurs données d'entraînement, comme l'illustre le proverbe anglo-saxon « garbage in, garbage out » : un algorithme entraîné sur des données de mauvaise qualité ne pourra produire que des prédictions médiocres [10]. ...
... However, linear regression should be applied with fundamental consideration-proper data quality. If basic requirement for data is not available, "garbage in garbage out" would appear (Kilkenny and Robinson 2018), leading wrong explanation for dynamics of IRSS. ...
Recently, the transition to the circular economy has become environmentally and economically urgent for every single nation in the world. Closing the loops of material is one of the key ideas behind the foundation of a circular economy (CE). The informal recyclable stations (IRSs) within the solid waste management (SWM) system play an important role as the reversed logistic system, being in charge of collecting and trading recyclable solid waste. This study aimed to comprehend the spatial nature of the system of IRSs in Danang city, Hue city, and Hoi An city as representative sites for the whole of central Vietnam and the nation. The integration of geographic information system (GIS), remote sensing, and statistical learning was performed to clarify spatial characteristics and dynamics of the system of IRSs as well as combat status of limitations of available data in developing countries. Results denoted that the system of IRSs was distributed in close proximity to transportation systems and residential areas with low vegetation coverage. Coverage ratios of the system of IRSs did not strongly fluctuate in case the number of IRSs decreased by 80% regarding the 3500 m distance covered. Negative binomial regression proved to be the most congruent model for understanding the prevalence of IRSs in central Vietnam. Population and normalized difference vegetation index were statistically related to prevalence of IRS. While linear regression depicted balance between variance and bias, support vector machine would be applied if prioritized aim is model performance. The results of this study are a scientific base for the management of the IRS system and the integration of this system into a formal SWM system as well as the transition to a CE.
... Precise input data appear to be critical here. In a different area, Poksinska et al. (2002), Kilkenny andRobinson (2018), Trinh et al. (2017), Bittner and Farajnia (2022) and Teno (2023) explore the significance of data quality and the impact of the GIGO principle from a data collection and analysis perspective, while the author highlights the severe effects of poor data quality (Munyisia et al., 2017). High-quality data are to be preferred, and the authors offer several suggestions. ...
In the age of smart or intelligent cities, the use of Artificial Intelligence (AI) presents a spectrum of new opportunities and challenges for both the research and policy community. The present study explores the intricate interplay between AI-generated content and actual choice spectra in urban planning. It focuses on the concept of 'city intelligence' and related AI concepts, underscoring the pivotal role of AI in addressing and understanding the quality of life in contemporary urban environments. As AI continues its transformative impact on communication and information systems in the realm of urban planning, this study brings to the forefront key insights into the challenges of validating AI-based information. Given the inherently subjective nature of AI-generated content, and its influential role in shaping user-perceived value, AI will most likely be a game changer catalyzing enhancements in the urban quality of life and inducing favorable urban developments. Additionally, the study also addresses the significance of the so-called 'Garbage-in Garbage-out' (GiGo) principle and 'Bullshit-in Bullshit out' (BiBo) principle in validating AI-generated content, and seeks to enhance our understanding of the spatial information landscape in urban planning by introducing the notion of an urban 'XXQ' performance production function.
... Modifying the data set involves selecting and enriching the available data [74], not only by choosing the most appropriate data but also by designing experiments to collect this data if necessary [75], ensuring it accurately represents the phenomena under study [76]. Techniques for data fusion and data integration are a key in this context [77]. ...
Machine learning has become a powerful tool for computational analysis in the biomedical sciences, with its effectiveness significantly enhanced by integrating domain-specific knowledge. This integration has give rise to informed machine learning, in contrast to studies that lack domain knowledge and treat all variables equally (uninformed machine learning). While the application of informed machine learning to bioinformatics and health informatics datasets has become more seamless, the likelihood of errors has also increased. To address this drawback, we present eight guidelines outlining best practices for employing informed machine learning methods in biomedical sciences. These quick tips offer recommendations on various aspects of informed machine learning analysis, aiming to assist researchers in generating more robust, explainable, and dependable results. Even if we originally crafted these eight simple suggestions for novices, we believe they are deemed relevant for expert computational researchers as well.
... To enhance data completeness, meticulous recording of all clinical parameters and outcomes in prospective registry databases is essential [12]. Adhering to the longstanding principle of 'garbage in, garbage out', key elements for meaningful analyses include data accuracy, reliability, and completeness [13]. Previous research has explored the impact of datadriven dashboards on various aspects, including clinical decision making, task completion time, satisfaction, research, and adherence to clinical guidelines, thereby strengthening quality assurance programs. ...
Background: Quality assurance in data collection is essential as data quality directly impacts the accuracy and reliability of outcomes. In the context of early detection of prostate cancer, improving data completeness is a key focus for enhancing patient care. This study aimed to evaluate the effectiveness of a data-driven feedback tool, visualized through a dashboard, in improving the completeness of data collection by healthcare professionals. Methods: A cohort of eight healthcare professionals were provided with a dashboard displaying weekly feedback on the completeness of 86 essential data items, including patient demographics, laboratory results, and imaging findings. A comparative analysis of data completeness was conducted for 577 patients enrolled in the prostate cancer early detection pathway, with 211 patients assessed before and 366 patients after the introduction of the dashboard. Statistical analysis was performed using the Mann–Whitney rank-sum test and Chi-square tests. Results: The implementation of the dashboard significantly improved data completeness across all healthcare professionals. The average completeness score increased from 0.70 (95% CI 0.67–0.76) before the dashboard’s introduction to 0.88 (95% CI 0.86–0.92) after its implementation, with a p-value of <0.001. Conclusions: The introduction of a data-driven feedback dashboard significantly enhanced data completeness within the prostate cancer early detection pathway. This improvement has the potential to positively impact the quality of care and to support the generation of high-quality data for future research.
... Also, the performance of the LSTM models is highly dependent on the features being used. Using more features doesn't mean better performance (garbage in, garbage out) (Kilkenny & Robinson, 2018). Researchers have also shown that feature selection significantly influences the performance of LSTM models (Li & Becker, 2021). ...
These past few years, technology has simplified the process of gathering and arranging time series data. This paves the way for tremendous opportunities to gain helpful insights by analysing these data. Historically, statistical models have been used for time series analysis. These models work well for linear or univariate data but struggle to accurately capture complex nonlinear trends or when unpredictable external factors impact the data (such as stock or trading prices). Long Short-Term Memory (LSTM) has emerged as one of the most popular options for analysing time series data efficiently. However, there are still a few challenges associated with it. The main reason for this study is to discuss these shortcomings and propose an intelligent system to deal with these flaws. One of the challenges is the choice of hyperparameters, which are handled successfully using the Genetic Algorithm (GA) to optimise the LSTM method’s hyperparameters and its variants. In this case, the GA solves the combinatorial optimization problem of finding the optimal hyper-parameters for the LSTM model and selecting the appropriate features from the data set. The proposed model has experimented on the National Stock Exchange-Fifty (NIFTY-50) dataset and found that the accuracy of all LSTM variants was improved. On average, the GA improved the performance of LSTM models by 60.11%. The Bidirectional LSTM model performs best with a root mean square error of an average of 66.39% and mean absolute error of an average of 49.43% after optimization, followed by the classic LSTM and the stacked LSTM models. The combination of LSTM with GA holds promise for enhancing the predictive power of time series analysis in various fields.
... The short statement "garbage in, garbage out" recognizes that poor quality of data would lead to unreliable outputs, emphasizing the utmost importance of improving and maintaining data quality [9]. In healthcare research, the collection of high-quality, reliable data is essential in the field of statistics and artificial intelligence (AI)-models, especially machine learning (ML) methods, which also include deep learning techniques for generating new clinically relevant algorithms for advancing medical knowledge, illness detection and personalized treatments [10]. ...
In the last decades, clinical laboratories have significantly advanced their technological capabilities, through the use of interconnected systems and advanced software. Laboratory Information Systems (LIS), introduced in the 1970s, have transformed into sophisticated information technology (IT) components that integrate with various digital tools, enhancing data retrieval and exchange. However, the current capabilities of LIS are not sufficient to rapidly save the extensive data, generated during the total testing process (TTP), beyond just test results. This opinion paper discusses qualitative types of TTP data, proposing how to divide laboratory-generated information into two categories, namely metadata and peridata. Being both metadata and peridata information derived from the testing process, it is proposed that the first is useful to describe the characteristics of data, while the second is for interpretation of test results. Together with standardizing preanalytical coding, the subdivision of laboratory-generated information into metadata or peridata might enhance ML studies, also by facilitating the adherence of laboratory-derived data to the Findability, Accessibility, Interoperability, and Reusability (FAIR) principles. Finally, integrating metadata and peridata into LIS can improve data usability, support clinical utility, and advance AI model development in healthcare, emphasizing the need for standardized data management practices.
... Given that sensitive attributes or their proxies 7 are often used as input variables in AI algorithms, how these variables affect the system is a general concern among the AI community. Perhaps it is in issues involving fairness that the old "Garbage in, Garbage out" motto becomes more pronounced [104,208], something that also points to the fact that algorithmic discrimination is but a reflection of social inequalities which, unfortunately, have no easy technical solution. When algorithms are trained on data that reflects societal biases or historical injustices, they may inadvertently learn and perpetuate those biases, leading to unfair or discriminatory outcomes, especially for specific demographic groups associated with sensitive attributes tied to those unfair cases. ...
This whitepaper offers normative and practical guidance for developers of artificial intelligence (AI) systems to achieve "Trustworthy AI". In it, we present overall ethical requirements and six ethical principles with value-specific recommendations for tools to implement these principles into technology. Our value-specific recommendations address the principles of fairness, privacy and data protection, safety and robustness, sustainability, transparency and explainability and truthfulness. For each principle, we also present examples of criteria for risk assessment and categorization of AI systems and applications in line with the categories of the European Union (EU) AI Act. Our work is aimed at stakeholders who can take it as a potential blueprint to fulfill minimum ethical requirements for trustworthy AI and AI Certification.
... Given that sensitive attributes or their proxies 7 are often used as input variables in AI algorithms, how these variables affect the system is a general concern among the AI community. Perhaps it is in issues involving fairness that the old "Garbage in, Garbage out" motto becomes more pronounced [104,208], something that also points to the fact that algorithmic discrimination is but a reflection of social inequalities which, unfortunately, have no easy technical solution. When algorithms are trained on data that reflects societal biases or historical injustices, they may inadvertently learn and perpetuate those biases, leading to unfair or discriminatory outcomes, especially for specific demographic groups associated with sensitive attributes tied to those unfair cases. ...
This whitepaper offers normative and practical guidance for developers of artificial intelligence (AI) systems to achieve "Trustworthy AI". In it, we present overall ethical requirements and six ethical principles with value-specific recommendations for tools to implement these principles into technology. Our value-specific recommendations address the principles of fairness, privacy and data protection, safety and robustness, sustainability, transparency and explainability and truthfulness. For each principle, we also present examples of criteria for risk assessment and categorization of AI systems and applications in line with the categories of the European Union (EU) AI Act. Our work is aimed at stakeholders who can take it as a potential blueprint to fulfill minimum ethical requirements for trustworthy AI and AI Certification.
... This strict approach allowed the review results to remain accurate and reliable, ensuring that the selected articles were in line with the research goals and enhanced the overall strength of the study. The quality of the review results depends on the sources or databases from which the articles are generated-"garbage-in, garbage-out" (Kilkenny & Robinson, 2018;Xiao & Watson, 2019) Table 3 displays the criteria for including and excluding the sources used in the SLR. The inclusion criteria were articles that were directly related to artificial intelligence (AI) in the context of L2 writing, as well as those that were relevant to writing-assisted tools, writing robots, writing evaluation, and automatic writing feedback in L2 writing. ...
The utilization of artificial intelligence (AI)-powered tools in second language (L2) writing has evolved over the last decade. This attracted second-language writers to evaluate and improve their writing. This study aims to contribute to the understanding of the current state of AI-powered software in L2 writing, identify gaps in the literature, and investigate areas for future research. In this systematic literature review (SLR), we categorize the typology of AI-powered tools and their impact on L2 writing performance, discuss L2 writers' perceptions, and provide an overview of how they mitigate challenges and limitations in utilizing writing-assisted tools. The results of this SRL may have implications for writing teachers, L2 researchers, and developers of AI-powered writing tools in the field of second language writing.
... 11 Generally, these types of problems in AI are known as "garbage in, garbage out" problems. When the input data is of poor quality, there will be problems in the outcome variables (Kilkenny and Robinson 2018). However, where cleaning the data is often the suggested solution (getting rid of noise, biases etc.) in the current situation, this will not work. ...
Machine learning (ML) has emerged as a promising tool in psychiatry, revolutionising diagnostic processes and patient outcomes. In this paper, I argue that while ML studies show promising initial results, their application in mimicking clinician-based judgements presents inherent limitations (Shatte et al. in Psychol Med 49:1426–1448. https://doi.org/10.1017/S0033291719000151, 2019). Most models still rely on DSM (the Diagnostic and Statistical Manual of Mental Disorders) categories, known for their heterogeneity and low predictive value. DSM's descriptive nature limits the validity of psychiatric diagnoses, which leads to overdiagnosis, comorbidity, and low remission rates. The application in psychiatry highlights the limitations of supervised ML techniques. Supervised ML models inherit the validity issues of their training data set. When the model's outcome is a DSM classification, this can never be more valid or predictive than the clinician’s judgement. Therefore, I argue that these models have little added value to the patient. Moreover, the lack of known underlying causal pathways in psychiatric disorders prevents validating ML models based on such classifications. As such, I argue that high accuracy in these models is misleading when it is understood as validating the classification. In conclusion, these models will not will not offer any real benefit to patient outcomes. I propose a shift in focus, advocating for ML models to prioritise improving the predictability of prognosis, treatment selection, and prevention. Therefore, data selection and outcome variables should be geared towards this transdiagnostic goal. This way, ML can be leveraged to better support clinicians in personalised treatment strategies for mental health patients.
... Working with unreliable data will only lead to unreliable results. This is commonly expressed in colloquial terms as: "Garbage in-garbage out" [13]. This means that the validity of all experimental results must be carefully checked before proceeding to the analysis stage. ...
Experimentation is the core of scientific research. Performing an experiment can be considered equivalent to asking a question to Nature and waiting for an answer. Understanding a natural phenomenon usually requires doing many experiments until a satisfactory model of such phenomenon is obtained. There are infinite possible ways to plan a set of experiments for researching a certain phenomenon, and some are more efficient than others. Experimental Design, also known as Design of Experiments (DoE), provides a systematic approach to obtain efficient experimental arrangements for different research problems. Experimental Design emerged almost a Century ago based on statistical analysis. Some decades after the development of DoE methods, they became widely used in all fields of Science and Engineering. Unfortunately, these valuable tools have been presently employed without a proper knowledge resulting in potentially erroneous conclusions. The purpose of this essay is discussing several mistakes that may occur due to the incorrect use of DoE methods.
... Practitioners integrating GenAI into collaborative human-centered innovation must invest significant effort in data management. Indeed, the old adage "garbage in, garbage out" resonates strongly with the use of GenAI in design sprints (Kilkenny & Robinson, 2018). High-quality data inputs are essential to generating better outputs that enhance the problem-solving process. ...
Organizations across various industries are still exploring the potential of Generative Artificial Intelligence (GenAI) to enhance knowledge work. While innovation is often viewed as a product of individual creativity, it more commonly unfolds through a highly structured, collaborative process where creativity intertwines with knowledge work. However, the extent and effectiveness of GenAI in supporting this process remain open questions. Our study investigates this issue using a collaborative practice research approach focused on three GenAI-enabled innovation projects conducted over a year within three different organizations. We explored how, why, and when GenAI could be integrated into design sprints, a highly structured, collaborative, and human-centered innovation method. Our research identified challenges and opportunities in synchronizing AI capabilities with human intelligence and creativity. To translate these insights into practical strategies, we propose four recommendations for organizations eager to leverage GenAI to both streamline and bring more value to their innovation processes: (1) establish a collaborative intelligence value loop with GenAI; (2) build trust in GenAI, (3) develop robust data collection and curation workflows, and (4) cultivate a craftsmanship mindset.
... The machine learning model developed through this study is expected to be applicable for evaluating and predicting the various intangible values (e.g., brand, technology, human resources). If this understanding is applied to open data openness and utilization in practical, real-world scenarios and the approach we propose for pre-evaluating and diagnosing open data utilization is implemented, it may help address the ongoing garbage data issues [89,90] related to open data. These findings were expected to serve as a catalyst for accelerating the process of unveiling the details of open data utilization. ...
As the digital transformation accelerates in our society, open data are being increasingly recognized as a key resource for digital innovation in the public sector. This study explores the following two research questions: (1) Can a machine learning approach be appropriately used for measuring and evaluating open data utilization? (2) Should different machine learning models be applied for measuring open data utilization depending on open data attributes (field and usage type)? This study used single-model (random forest, XGBoost, LightGBM, CatBoost) and multi-model (stacking ensemble) machine learning methods. A key finding is that the best-performing models differed depending on open data attributes (field and type of use). The applicability of the machine learning approach for measuring and evaluating open data utilization in advance was also confirmed. This study contributes to open data utilization and to the application of its intrinsic value to society.
... Ensuring established processes for high-quality data collection in health information management is crucial to prevent inaccuracies with significant implications, affecting funding and clinical documentation integrity. (Kilkenny & Robinson, 2018). ...
Purpose: This study proposes a conceptual framework for aligning performance management systems (PMS) with corporate culture development to cultivate organizational excellence. Method: The study employs an exploration approach. It involves analyzing and observing existing performance management systems, identifying corporate culture development and intervention alignment in the resource capabilities area, and offering detailed benchmarks for improvement. Results and Conclusion: By emphasizing strategic alignment, the framework facilitates nuanced benchmarking, enabling organizations to prioritize and implement decisions that harmonize the PMS with corporate culture development. The study contributes to effective management practices in the dynamic mining sector, providing a tailored solution for organizations striving for operational excellence through the integration of KBPMS and corporate culture development. Implications of the Research: The proposed conceptual framework offers practical implications for mining enterprises. It guides the enhancement of organizational performance by aligning KBPMS with corporate culture. Originality/Value: This study adds value by introducing a comprehensive conceptual framework tailored to the mining industry, addressing critical aspects of PMS and corporate culture alignment for achieving operational excellence.
... Dengan kata lain input data sangat berpengaruh terhadap output yang dihasilkan, atau biasa dikenal dengan istiah GARBAGE IN GARBAGE OUT (GIGO). Kualitas entri data buruk menyebabkan keluaran data tidak dapat diandalkan(Kilkenny, 2018) Penyusunan unsur manajemen dapat mencerminkan keberhasilan PAKSI, melalui pelatihan maupun sosialisasi terhadap pelaku PAKSI agar pemahaman dalam pengisian PAKSI dan output yang dihasilkan dapat seragam, mendalami JUKLAK PAKSI, serta evaluasi dan pembaharuan website dan aplikasi (e-PAKSI). ...
Rencana Pembangunan Jangka Menengah Nasional (RPJMN) 2020-2024 menjadi sasaran Visi Indonesia Maju 2045, menempatkan ketahanan pangan menjadi bagian dalam memperkuat ketahanan ekonomi, salah satunya melalui rehabilitasi jaringan irigasi. Selain itu pemerintah juga merumuskan Modernisasi Irigasi Strategis dan Program Rehabilitasi Mendesak yang terdiri dari peningkatan infrastruktur melalui kegiatan rehabilitasi (hard component) dan penguatan Sumber Daya Manusia yang terlibat dalam pengelolaan irigasi (soft component). Cara mengidentifikasi bangunan atau jaringan yang memerlukan rehabilitasi irigasi adalah dengan melakukan kegiatan PAKSI. PAKSI merupakan gabungan antara PAI (Pengelolaan Aset Irigasi) dan IKSI (pengukuran Indeks Kinerja Sistem Irigasi). Nilai IKSI Gabungan pada Daerah Irigasi Permukaan (Kewenangan Pusat dan Kewenangan TP-OP) Provinsi Kalimantan Selatan periode tahun 2021 – 2023. Penelitian ini bersifat deskriptif kuantitatif dengan menggunakan data primer dan sekunder hasil website PAKSI. Kajian ini menunjukkan pola turun-naik sejumlah 4 (empat) Daerah Irigasi, pola turun-turun sebanyak 1 (satu) Daerah Irigasi, pola naik-naik sejumlah 3 Daerah Irigasi, dan pola naik-turun sebanyak 2 (dua) Daerah Irigasi. Berdasarkan analisis menunjukkan bahwa, kenaikan/penurunan variabel nilai IKSI dikarenakan subjektivitas surveyor. Hal ini terlihat dari variabel produktivitas tanam bernilai 0 (nol) atau tidak diisi. Selain itu dokumentasi dan Petugas Pembagi Air nilainya menurun dan juga tidak diisi. Laporan akhir juga tidak menjelaskan secara lengkap akibat kenaikan/penurunan pada 6 (enam) variabel, serta bukti tidak lengkap. Untuk meningkatkan hasil monev PAKSI diperlukan unsur manajemen (5M, yaitu Man, Money, Method, Machine, Material). Penyusunan unsur manajemen bertujuan untuk membuat kriteria 5M, selanjutnya digunakan untuk evaluasi hasil PAKSI pada 10 (sepuluh) Daerah Irigasi Permukaan. Unsur money dan machine sudah sangat baik dan perlu ditingkatkan terutama performa website dan aplikasi PAKSI. Unsur man berbanding lurus dengan method dan material. Ketiga unsur ini masih kurang, sehingga perlu pelatihan, bimbingan teknis dan sosialisasi terutama terhadap surveyor untuk menyamakan presepsi dan keseragaman output yang dihasilkan. Keberhasilan PAKSI dapat mendukung ketahanan pangan dan program Irrigation Service Agreement (ISA).
... 25 It is particularly important to validate minimum database quality in healthcare contexts. 26 Our team completed error checks and validations to ensure data integrity. ...
Objective:
Family health history (FHx) is an important tool in assessing one's risk towards specific health conditions. However, user experience of FHx collection tools is rarely studied. ItRunsInMyFamily.com (ItRuns) was developed to assess FHx and hereditary cancer risk. This study reports a quantitative user experience analysis of ItRuns.
Methods:
We conducted a public health campaign in November 2019 to promote FHx collection using ItRuns. We used software telemetry to quantify abandonment and time spent on ItRuns to identify user behaviors and potential areas of improvement.
Results:
Of 11,065 users who started the ItRuns assessment, 4305 (38.91%) reached the final step to receive recommendations about hereditary cancer risk. Highest abandonment rates were during Introduction (32.82%), Invite Friends (29.03%), and Family Cancer History (12.03%) subflows. Median time to complete the assessment was 636 s. Users spent the highest median time on Proband Cancer History (124.00 s) and Family Cancer History (119.00 s) subflows. Search list questions took the longest to complete (median 19.50 s), followed by free text email input (15.00 s).
Conclusion:
Knowledge of objective user behaviors at a large scale and factors impacting optimal user experience will help enhance the ItRuns workflow and improve future FHx collection.
... The reliability of data-centric models depends on the quality of the dataset. Lowquality data produces inaccurate results [24]. It is of utmost importance to carefully craft and preprocess datasets. ...
Cybersecurity has become a major concern in the modern world due to our heavy reliance on cyber systems. Advanced automated systems utilize many sensors for intelligent decision-making, and any malicious activity of these sensors could potentially lead to a system-wide collapse. To ensure safety and security, it is essential to have a reliable system that can automatically detect and prevent any malicious activity, and modern detection systems are created based on machine learning (ML) models. Most often, the dataset generated from the sensor node for detecting malicious activity is highly imbalanced because the Malicious class is significantly fewer than the Non-Malicious class. To address these issues, we proposed a hybrid data balancing technique in combination with a Cluster-based Under Sampling and Synthetic Minority Oversampling Technique (SMOTE). We have also proposed an ensemble machine learning model that outperforms other standard ML models, achieving 99.7% accuracy. Additionally, we have identified the critical features that pose security risks to the sensor nodes with extensive explainability analysis of our proposed machine learning model. In brief, we have explored a hybrid data balancing method, developed a robust ensemble machine learning model for detecting malicious sensor nodes, and conducted a thorough analysis of the model’s explainability.
... Step 4: However, these raw data usually have some problems such as locked variables, missing values, and random noise, so preliminary data preprocessing is an essential step. Because there is a classic saying, " garbage in, garbage out" in the data mining field, 51 meaning that data quality can be a limiting factor for soft sensing modeling, a series of popular data preprocessing methods are adopted to enhance data quality before being fed into machine learning models (e.g., data cleaning, data reduction, data transformation, and data visualization). ...
Data-driven soft sensing modeling is becoming a powerful tool in the ironmaking process due to the rapid development of machine learning and data mining. Although various soft sensing techniques have been successfully used in both the sintering process and blast furnace, they have not been comprehensively reviewed. In this work, we provide an overview of recent advances on soft sensing in the ironmaking process, with a special focus on data-driven techniques. First, we present a general soft sensing development framework of the ironmaking process based on the mechanism analysis and process characteristics. Second, we provide a detailed taxonomy of current soft sensing methods categorized by their predictive tasks (i.e., quality indicators prediction, state parameters prediction, etc.). Finally, we outline several insightful and promising directions, such as self-supervised learning and digital twins in the ironmaking process, for future research.
... Causes of systematic bias in ML are rooted throughout all components of the ML life cycle [10]. In the data production and management phase, the use of biased data may lead to biased predictions, analogous to the "garbage in, garbage out" concept, where poor-quality data produce poor-quality output [10,23]. Specifically, bias may be introduced during the data collection process, resulting in training data that do not reflect the characteristics of the intended population (i.e., population bias) [10]. ...
Machine learning (ML) has become integral in educational decision-making through technologies such as learning analytics and educational data mining. However, the adoption of machine learning-driven tools without scrutiny risks perpetuating biases. Despite ongoing efforts to tackle fairness issues, their application to educational datasets remains limited. To address the mentioned gap in the literature, this research evaluates the effectiveness of four bias mitigation techniques in an educational dataset aiming at predicting students’ dropout rate. The overarching research question is: “How effective are the techniques of reweighting, resampling, and Reject Option-based Classification (ROC) pivoting in mitigating the predictive bias associated with high school dropout rates in the HSLS:09 dataset?" The effectiveness of these techniques was assessed based on performance metrics including false positive rate (FPR), accuracy, and F1 score. The study focused on the biological sex of students as the protected attribute. The reweighting technique was found to be ineffective, showing results identical to the baseline condition. Both uniform and preferential resampling techniques significantly reduced predictive bias, especially in the FPR metric but at the cost of reduced accuracy and F1 scores. The ROC pivot technique marginally reduced predictive bias while maintaining the original performance of the classifier, emerging as the optimal method for the HSLS:09 dataset. This research extends the understanding of bias mitigation in educational contexts, demonstrating practical applications of various techniques and providing insights for educators and policymakers. By focusing on an educational dataset, it contributes novel insights beyond the commonly studied datasets, highlighting the importance of context-specific approaches in bias mitigation.
... These datasets should be the result of a multidisciplinary team of professionals working towards a predefined goal. As a notorious quote says: "garbage in-garbage out", meaning input datasets are crucial in determining the final outcome [131]. Ensuring data quality requires a dedicated infrastructure, i.e., patient health records, diagnostic images, and realtime data monitoring. ...
The dawn of Artificial intelligence (AI) in healthcare stands as a milestone in medical innovation. Different medical fields are heavily involved, and pediatric emergency medicine is no exception. We conducted a narrative review structured in two parts. The first part explores the theoretical principles of AI, providing all the necessary background to feel confident with these new state-of-the-art tools. The second part presents an informative analysis of AI models in pediatric emergencies. We examined PubMed and Cochrane Library from inception up to April 2024. Key applications include triage optimization, predictive models for traumatic brain injury assessment, and computerized sepsis prediction systems. In each of these domains, AI models outperformed standard methods. The main barriers to a widespread adoption include technological challenges, but also ethical issues, age-related differences in data interpretation, and the paucity of comprehensive datasets in the pediatric context. Future feasible research directions should address the validation of models through prospective datasets with more numerous sample sizes of patients. Furthermore, our analysis shows that it is essential to tailor AI algorithms to specific medical needs. This requires a close partnership between clinicians and developers. Building a shared knowledge platform is therefore a key step.
... To achieve optimal performance, a high-quality input dataset proper for the nature of the problem is expected to be inputted by the learning models. This situation can be supported by an old proverb "Garbage in-garbage out", implying the poor quality of data can lead to unreliable outputs (Kilkenny and Robinson 2018). Therefore, the analyzed, collected, or acquired geospatial data should be of high accuracy, otherwise, can cast serious doubts on the reliability of the specific tasks. ...
Since the 1970s, the scientific community has dedicated significant efforts to the development of landslide susceptibility models through various approaches, with the current spotlight firmly on artificial intelligence techniques. Despite their unique advantages, these cutting-edge tools have introduced significant challenges, the solution of which hinges on critical user decisions. These decisions chiefly revolve around selecting landslide conditioning factors and designing the optimal configuration of internal mechanisms of susceptibility modeling approaches—both critical determinants influencing model predictive accuracy. To address origin of these issues, a systematic review of literature spanning seven years, from 2015 to 2021, was conducted. The results revealed the utilization of 151 various landslide conditioning factors, highlighting a clear dearth of consensus on the selection of geospatial covariates in the literature. Nonetheless, only about one-third of the reviewed articles considered the feature selection techniques to seek the optimal factor subset. The review also showed that 54 distinct machine learning algorithms were used, with logistic regression being the most commonly applied susceptibility modeling approach, featured in 70 articles. Notably, deep learning algorithms were marginally employed, appearing in a mere 7.08% of the reviewed articles since 2018. However, a significant proportion (64.32%) of the articles used non-optimized predictive models with default settings, while a trial-and-error approach was adopted in 10.81% of the reviewed literature. Beyond the comprehensive literature review, this chapter delves into a series of ill-explored open questions and reveals opportunities that can serve as potential research roadmaps, potentially guiding the trajectory of future studies in landslide susceptibility mapping.
Workforce Automation and Artificial Intelligence (AI) are revolutionizing how business operates in almost every type of industry. On the one hand, it makes doing our jobs more efficient and productive; on the other hand, it has serious ethical concerns. This paper analyzes the major moral problems of using AI to automate the workforce, including job replacement, prejudice, privacy risks, responsibility and regulatory responsibilities. This study seeks to proffer views about reconciling the progress of technology with the moral aspect to enable responsible AI deployment at workplaces by means of such analysis. KEYWORDS: AI ethics, workforce automation, job displacement, bias, privacy, accountability, transparency, regulation.
With rapid technological advancements, we are witnessing a global rush to develop and implement AI models across sectors. This chapter will focus on India, a developing country as diverse as its 1.42 billion citizens, and will discuss the emerging best practices to be adopted by the vibrant startup economy in India. With a focus on four core elements of responsible and ethical AI, this chapter will provide solutions on how Indian businesses can balance the tussle between innovation and regulation, while upholding ethical standards.
Global policies such as the OECD Principles and the UNESCO Guidelines on the use of AI are already prompting sectoral guidelines in India. Furthermore, the recent intervention by Indian authorities, mandating transparency and explainability in AI marks a significant regulatory step. Discussing regulatory principles, this chapter will throw light on potential regulatory models which may prove to be successful in the Indian economy. This chapter will also discuss the role of various stakeholders which will impact the use of AI in India. In conclusion, this chapter will highlight the importance of scalable AI models, along with a focus on sustainable development and use of AI models in India.
This research proposes a machine learning (ML) model that estimates the probability of emerald mineralization in rocks of the Western Emerald Belt (CEOC). Element concentrations, lithologies and coordinates were used as input variables and productivity as the target variable (176 samples). The variables were transformed to be integrated into the model. (1) Variable selection was performed using the Boruta method and backward elimination. (2) A logistic regression, a neural network, and a support vector machine were trained. (3) Calibration was achieved with the Platt method. (4) Calibration assessment was conducted by using the Brier score and calibration curves. The model selected was a calibrated support vector machine (C = 0.19 and λ = 0.1) that included 17 geochemical variables and the coordinates. The results were presented in a 3D plot. Assigning a probability value to each sample allows the mining targets to be ranked.
This chapter explores the role of Generative Artificial Intelligence (GAI) in supporting research design and methodology in educational leadership. Since the introduction of ChatGPT in 2022, the use of GAI in education has expanded rapidly, being leveraged for tasks in K-20 institutions (Chen et al., 2020; Crompton & Burke, 2023). GAI has the potential to streamline research processes and enhances the depth of analysis (Kooli, 2023; Paek & Kim, 2021). As an educational leadership professor, I focus on how GAI improves research in this field, though many points apply to other settings. While GAI technologies offer powerful capabilities to boost efficiency, they also raise ethical considerations, particularly around bias, data privacy, and transparency (Lund et al., 2023). This chapter addresses the opportunities and challenges of GAI integration, emphasizing the need for responsible, fair, and inclusive practices. The Human in the Loop (HITL) approach and the Action, Specificity, Knowledge (ASK) framework are strategies presented here to ensure ethical AI use.
INTRODUCTION: Many mistakes in clinical practice arise from confusing the probability of a positive test in those with the disease and the probability of having the disease in those who test positive. This misunderstanding leads to overestimating disease probability, diagnosing diseases in healthy individuals, ordering invasive diagnostic tests, and prescribing unnecessary treatments, resulting in unjustified adverse effect, psychological stress, and increased cost. Probabilistic reasoning is an essential skill to mitigate this confusion, and Bayes theorem is an important tool to accomplish this goal. OBJECTIVE: To present a step-by-step demonstration of Bayes' formula for positive and negative predictive values, fostering understanding and enabling its adoption in evidence-based medicine education and clinical practice as a supporting tool in the decision-making process. METHODS: In this article, we explain the difference between deductive and inductive thinking and how diagnostic reasoning is predominantly inductive, where evidence (the test result) is used to predict the cause (the presence of disease), a path that involves reverse probability, for which our reasoning is hazier. Through a clinical example involving the diagnosis of systemic lupus erythematosus, we use the Bayesian framework as a tool to help understand the difference between sensitivity/specificity (forward probability; deductive) and positive/negative predictive values (reverse probability: inductive). CONCLUSIONS: Excellent doctors are masters at applying Bayesian reasoning without using any formulas: they understand that the most important component of the diagnostic process is the reasoning that originates it and the resulting clinical decision depends on interpreting results considering their interaction with the context, not in isolation. Bad clinical reasoning results in bad clinical decisions, despite how accurate the diagnostic test: garbage in, garbage out. We hope our step-by-step approach to Bayes' rule can help demystify this powerful statistical tool and strengthen the idea that the value of a diagnostic test is directly proportional to the quality of clinical reasoning that led to its request.
Machine learning (ML) and prediction modelling have become increasingly influential in healthcare, providing critical insights and supporting clinical decisions, particularly in the age of big data. This paper serves as an introductory guide for health researchers and readers interested in prediction modelling and explores how these technologies support clinical decisions, particularly with big data, and covers all aspects of the development, assessment and reporting of a model using ML. The paper starts with the importance of prediction modelling for precision medicine. It outlines different types of prediction and machine learning approaches, including supervised, unsupervised and semi-supervised learning, and provides an overview of popular algorithms for various outcomes and settings. It also introduces key theoretical ML concepts. The importance of data quality, preprocessing and unbiased model performance evaluation is highlighted. Concepts of apparent, internal and external validation will be introduced along with metrics for discrimination and calibration for different types of outcomes. Additionally, the paper addresses model interpretation, fairness and implementation in clinical practice. Finally, the paper provides recommendations for reporting and identifies common pitfalls in prediction modelling and machine learning. The aim of the paper is to help readers understand and critically evaluate research papers that present ML models and to serve as a first guide for developing, assessing and implementing their own.
Structural planes are one of the key factors controlling the stability of rock masses. A comprehensive understanding of the spatial distribution characteristics of structural planes is essential for accurately identifying key blocks, analyzing rock mass stability, and addressing various rock engineering challenges. This study compares the effectiveness of four stochastic structural plane generation methods—the Monte Carlo method, the Copula-based method, generative adversarial networks (GAN), and denoised diffusion models (DDPM)—in generating stochastic structural planes and capturing potential correlations between structural plane parameters. The Monte Carlo method employs the mean and variance of three parameters of the measured factual structural planes to generate data that follow the same distributions. The other three methods take the entire set of measured factual structural planes as the overall input to generate structural planes that exhibit the same probability distributions. Five sets of structural planes on four rock slopes in Norway are examined as an example. The validation and analysis were performed using histogram comparison, data feature comparison, scatter plot comparison, and linear regression analysis. The results show that the Monte Carlo method fails to capture the potential correlation between the dip direction and dip angle despite the best fit to the measured factual structural planes. The Copula-based method performs better with smaller datasets, and GAN and DDPM are better at capturing the correlation of measured factual structural planes in the case of large datasets. Therefore, in the case of a limited number of measured structural planes, it is advisable to employ the Copula-based method. In scenarios where the dataset is extensive, the deep generative model is recommended due to its ability to capture complex data structures. The results of this study can be utilized as a valuable point of reference for the accurate generation of stochastic structural planes within rock masses.
The current digital landscape is evolving at an increasingly rapid pace where “data is the new oil” of the global economy. Like oil, extracting value from raw data is a complex process as it is not useful in its raw state. As data continue to be generated and relied upon, data literacy skills are becoming increasingly critical. Since data must be ‘prepared’ prior to its analysis, this paper highlights three core competencies—cleaning, transforming, merging (i.e., data preparation)—that are required to build a sound marketing analytics foundation. Expanding data literacy to include the ability to transform data from its raw state into a usable form will enhance students’ overall level of proficiency and marketability in marketing analytics. It is imperative to include the teaching of data preparation as part of the analytics curriculum in marketing analytics courses to ensure that students attain the greater level of data literacy. Data preparation assignments that will help students enhance their marketing analytics skillset, and increase their overall knowledge and marketability are included.
In recent years we have gained insight into the impact of minimum unit pricing (MUP)—a legal floor price below which a given volume of alcohol cannot be sold—on population‐level reductions in alcohol sales, consumption and harm. However, several questions remain unanswered including how individual‐level purchasing changes impact the local economy (e.g., balance between on‐licence and off‐licence outlets), lead to long‐term population‐level trends (e.g., youth drinking) and social harms (e.g., violence). Agent‐based modelling captures heterogeneity, emergence, feedback loops and adaptive and dynamic features, which provides an opportunity to understand the nuanced effects of MUP. Agent‐based models (ABM) simulate heterogeneous agents (e.g., individuals, organisations) often situated in space and time that interact with other agents and/or with their environment, allowing us to identify the mechanisms underlying social phenomena. ABMs are particularly useful for theory development, and testing and simulating the impacts of policies and interventions. We illustrate how ABMs could be applied to generate novel insights and provide best estimates of social network effects, and changes in purchasing behaviour and social harms, due to the implementation of MUP. ABMs like other modelling approaches can simulate alternative implementations of MUP (e.g., policy intensity [£0.50, £0.60] or spatial scales [local, national]) but can also provide an understanding of the potential impact of MUP on different population groups (e.g., alcohol exposure of young people who are not yet drinking). Using ABMs to understand the impact of MUP would provide new insights to complement those from traditional epidemiological and other modelling methods.
Background and objective:
The incidence of cancer is rising in Iran, and hence it is important to assess the accuracy of the Iranian cancer registry dataset. In this study, the completeness of the cancer registry in the Kohgiluyeh and Boyer-Ahmad (K&B) province is evaluated.
Method:
The data of registered cases of cancer of people who were living in the K&B province at the time of diagnosis were obtained from the provincial cancer registry offices in K&B, Fars and all other neighbouring provinces. A capture-recapture method along with log-linear statistical modelling were used for analysis.
Results:
The results indicated that of 2029 known cases of cancer, only 1400 (31%) were registered by the K&B cancer registry office. Age-adjusted incidence rates for all common types of cancer rose from 307.0 per 100,000 (95% confidence interval (CI); 293.8, 320.3, based on observed cases) to 376.4 per 100,000 (95% CI; 361.7, 391.1, based on expected number of cases estimated by capture-recapture analysis) ( p < 0.01). The completeness of cancer registry data varied significantly for different types of cancer.
Conclusion:
Results suggest that the provincial cancer dataset, which is a part of the national cancer registry programme, is neither complete nor representative. A major improvement in case finding, registry procedures and effective data sharing by provincial cancer registry offices is needed in order to provide valid data for epidemiology of cancer in Iran.
Background:
Efficient information systems support the provision of multi-disciplinary aged care and a variety of organisational purposes, including quality, funding, communication and continuity of care. Agreed minimum data sets enable accurate communication across multiple care settings. However, in aged care multiple and poorly integrated data collection frameworks are commonly used for client assessment, government reporting and funding purposes.
Objective:
To determine key information needs in aged care settings to improve information quality, information transfer, safety, quality and continuity of care to meet the complex needs of aged care clients.
Method:
Modified Delphi methods involving five stages were employed by one aged care provider in Victoria, Australia, to establish stakeholder consensus for a derived minimum data set and address barriers to data quality.
Results:
Eleven different aged care programs were identified; with five related data dictionaries, three minimum data sets, five program standards or quality frameworks. The remaining data collection frameworks related to diseases classification, funding, service activity reporting, and statistical standards and classifications. A total of 170 different data items collected across seven internal information systems were consolidated to a derived set of 60 core data items and aligned with nationally consistent data collection frameworks. Barriers to data quality related to inconsistencies in data items, staff knowledge, workflow, system access and configuration.
Conclusion:
The development an internal aged care minimum data set highlighted the critical role of primary data quality in the upstream and downstream use of client information; and presents a platform to build national consistency across the sector.
Background:
Information management systems and processes have an impact on quality and safety of care in any setting and particularly in the complex care setting of aged care. Few studies have comprehensively examined information management in the Australian aged care setting.
Objective:
To (i) critically analyse and synthesize evidence related to information management in aged care, (ii) identify aged care data collection frameworks and (iii) identify factors impacting information management.
Methods:
An integrative review of Australian literature published between March 2008 and August 2014 and data collection frameworks concerning information management in aged care were carried out.
Results:
There is limited research investigating the information-rich setting of aged care in Australia. Electronic systems featured strongly in the review. Existing research focuses on residential settings with community aged care largely absent. Information systems and processes in the setting of aged care in Australia are underdeveloped and poorly integrated.
Conclusions:
Data quality and access are more problematic within community aged care than residential care settings. The results of this review represent an argument for a national approach to information management in aged care to address multiple stakeholder information needs and more effectively support client care.
It seems today to be an indisputable fact that ISO 9000 is a powerful instrument, which cannot be disregarded. It is, far and away, the most influential initiative that grew from the quality movement of the late 1980s. This paper contains an evaluation of results from a survey on ISO 9000 certified companies and aims to present some aspects of the current state of the standard in Swedish industry. This study is focused on motives for implementation, perceived benefits and key implementation factors. The predominant reasons identified for seeking certification were the desire to improve corporate image and quality. Like many previous studies this study underlines the need for management commitment and participation. The very important conclusion drawn from this survey is that the motivation for certification may influence the performance of ISO 9000. The overall benefits which the companies gain from the standard showed dependence on the motivation which initiated the drive for the certification.
Background:
Research has associated some chronic conditions with self-harm and suicide. Quantifying such a relationship in mortality data relies on accurate death records and adequate techniques for identifying these conditions.
Objective:
This study aimed to quantify the impact of identification methods for co-morbid conditions on suicides in individuals aged 30 years and older in Australia and examined differences by gender.
Method:
A retrospective examination of mortality records in the National Coronial Information System (NCIS) was conducted. Two different methods for identifying co-morbidities were compared: International Statistical Classification of Diseases and Related Health Problems, 10th Revision (ICD-10) coded data, which are provided to the NCIS by the Australian Bureau of Statistics, and free-text searches of Medical Cause of Death fields. Descriptive statistics and χ(2) tests were used to compare the methods for identifying co-morbidities and look at differences by gender.
Results:
Results showed inconsistencies between ICD-10 coded and coronial reports in the identification of suicide and chronic conditions, particularly by type (physical or mental). There were also significant differences in the proportion of co-morbid conditions by gender.
Conclusion:
While ICD-10 coded mortality data more comprehensively identified co-morbidities, discrepancies in the identification of suicide and co-morbid conditions in both systems require further investigation to determine their nature (linkage errors, human subjectivity) and address them. Furthermore, due to the prescriptive coding procedures, the extent to which medico-legal databases may be used to explore potential and previously unrecognised associations between chronic conditions and self-harm deaths remains limited.
Objective:
Public health data sets such as the Victorian Perinatal Data Collection (VPDC) provide an important source for health planning, monitoring, policy, research and reporting purposes. Data quality is paramount, requiring periodic assessment of data accuracy. This article describes the conduct and findings of a validation study of data on births in 2011 extracted from the VPDC.
Method:
Data from a random sample of one percent of births in Victoria in 2011 were extracted from original medical records at the birth hospital and compared with data held in the VPDC. Accuracy was determined for 93 variables. Sensitivity, specificity, positive predictive value and negative predictive value were calculated for dichotomous items.
Results:
Accuracy of 17 data items was 99% or more, the majority being neonatal and intrapartum items, and 95% or more for 46 items. Episodes of care with the highest proportion of items with accuracy of 95% or more were neonatal and postnatal items at 80 and 64%, respectively. Accuracy was below 80% for nine items introduced in 2009. Agreement between medical records and VPDC data ranged from 48% to 100%, the exception being two highly inaccurate smoking-related items. Reasons for discrepancies between VPDC data and medical records included miscoding, missing and inconsistent information.
Conclusion:
This study found high levels of accuracy for data reported to the VPDC for births in 2011; however, some data items introduced in 2009 and not previously validated were less accurate. Data may be used with confidence overall and with awareness of limitations for some new items.
Background:
Despite increasing research on activity-based funding (ABF), there is no empirical evidence on the accuracy of outpatient service data for payment.
Objective:
This study aimed to identify data entry errors affecting ABF in two drug and alcohol outpatient clinic services in Australia.
Methods:
An audit was carried out on healthcare workers' (doctors, nurses, psychologists, social workers, counsellors, and aboriginal health education officers) data entry errors in an outpatient electronic documentation system.
Results:
Of the 6919 data entries in the electronic documentation system, 7.5% (518) had errors, 68.7% of the errors were related to a wrong primary activity, 14.5% were due to a wrong activity category, 14.5% were as a result of a wrong combination of primary activity and modality of care, 1.9% were due to inaccurate information on a client's presence during service delivery and 0.4% were related to a wrong modality of care.
Conclusion:
Data entry errors may affect the amount of funding received by a healthcare organisation, which in turn may affect the quality of treatment provided to clients due to the possibility of underfunding the organisation. To reduce errors or achieve an error-free environment, there is a need to improve the naming convention of data elements, their descriptions and alignment with the national standard classification of outpatient services. It is also important to support healthcare workers in their data entry by embedding safeguards in the electronic documentation system such as flags for inaccurate data elements.
Objective:
To examine the quality of the two routinely collected sets of data, the Incident Information Management System (IIMS) and the health information exchange (HIE) in hospitals in New South Wales, Australia.
Method:
IIMS records indicating a fall and its location were examined. HIE data were examined using International Classification of Diseases (ICD)-10-AM codes W00-W19 and an indicator, 'onset of the condition' for falls in hospital. If onset of the condition was not recorded, ICD-10-AM code for place of occurrence (Y92.22 = Health service area) immediately following ICD-10-AM code for the fall was used. Comprehensive criteria were applied to exclude records of earlier documented falls. IIMS and HIE data were linked. Characteristics of falls that were recorded in one data set but not in the other were determined.
Results:
Between January 2010 and December 2014, 8647 falls in hospitals were recorded in IIMS, 2169 were recorded in HIE and 9338 were recorded in either data set (rate of 3.2 falls per 1000 bed days). IIMS captured 93% and HIE captured 23% of these falls. Of the falls recorded in HIE, 677 (31%) were not recorded in IIMS. These were more likely to be subsequent falls, by patients who were female, younger than 65 years, who underwent a non-allied health procedure or had length of stay less than 1 week.
Conclusions:
IIMS captured the vast majority of falls in hospitals but failed to report one-third of falls recorded in HIE.
Background:
Instrumental vaginal deliveries (IVDs) account for approximately 11% of births in Australia. Complications resulting from IVD can occasionally be the subject of litigation. The Royal College of Obstetricians and Gynaecologists suggests a standardised pro forma in their guidelines as an aid to accurate and complete IVD documentation. Many units, including ours, use less structured reporting, which is probably also less adequate.
Aim:
To assess whether the introduction of a dedicated IVD form improves the quality of IVD documentation.
Method:
Analysis of the quality of IVD documentation before and after the implementation of a new dedicated IVD form. A survey to evaluate clinicians' opinion on the new standardised form.
Results:
Significant improvement was found in documentation of key information including the documentation of caput (p < 0.05), type of instrument, number of ventouse cup detachments, moulding of specific sutures, abdominal palpation (number of fifths of foetal head palpable), liquor colour and total time of instrument application (p < 0.001). A majority of clinicians believed the form to be beneficial in terms of completeness and that it reduced the amount of time required for documentation.
Conclusions:
IVD documentation is enhanced by the use of a dedicated form. Clinical judgement may also be enhanced by the discipline involved in the formal assessment required by the form.
The healthcare system in Finland has begun routine collection of health-related quality of life (HRQoL) information for patients in hospitals to support more systematic cost-effectiveness analysis (CEA). This article describes the systematic collection of HRQoL survey data, and addresses challenges in the implementation of patient surveys and acquisition of cost data in the case hospital. Challenges include problems with incomplete data and undefined management processes. In order to support CEA of hospital treatments, improvements are sought from the process management literature and in the observation of healthcare professionals. The article has been written from an information system and process management perspective, concluding that process ownership, automation of data collection and better staff training are keys to generating more reliable data.
Objective:
This study described information management incidents and adverse event reporting choices of health professionals.
Methods:
Hospital adverse events reported in an anonymous electronic reporting system were analysed using directed content analysis and descriptive and inferential statistics. The data consisted of near miss and adverse event incident reports ( ITALIC! n = 3075) that occurred between January 2008 and the end of December 2009.
Results:
A total of 824 incidents were identified. The most common information management incident was failure in written information transfer and communication, when patient data were copied or documented incorrectly. Often patient data were transferred using paper even though an electronic patient record was in use. Reporting choices differed significantly among professional groups; in particular, registered nurses reported more events than other health professionals.
Conclusion:
A broad spectrum of information management incidents was identified, which indicates that preventing adverse events requires the development of safe practices, especially in documentation and information transfer.
Electronic health records and the Internet will continue to transform how information is accessed and shared. Users of health data such as health professionals, governments, policymakers, researchers and patients themselves need to be able to access the right information at the right time and be confident in the quality of that information, whether personal, aggregated or knowledge based. It is essential to evaluate information systems and applications that claim to improve information quality and access in order to provide evidence that they support healthcare delivery and improve patient outcomes.
Health data concepts and information governance
- M Teslow
Teslow M (2016) Health data concepts and information governance. In: Abdelhak M and Hanken MA (eds) Health Information: Management of a Strategic Resource, 5th ed, pp. 88-144.
St Louis, Missouri: Elsevier Saunders.
Information management in the Australian aged care setting
- J Davis
- A Morgans
- S Burgess
Davis J, Morgans A and Burgess S (2017) Information management in the Australian aged care setting. Health Information
Management Journal 46(1): 3-14.
Data accuracy in the Victorian Perinatal Data Collection: results of a validation study of 2011 data
- M M Flood
- Mcdonald
- Sj
- W E Pollock
The ABS Data Quality Framework. Cat No 1520.20. Canberra
Australian Bureau of Statistics (2009) The ABS Data Quality
Framework. Cat No 1520.20. Canberra, Australian Capital
Territory: Australian Bureau of Statistics.