Article

Ensuring data quality and maximizing efficiency in coding agricultural and forestry injuries: Lessons to improve occupational injury surveillance

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Introduction: Specialized occupational injury surveillance systems are filling the gap in the undercount of work-related injuries in industries such as agriculture and forestry. To ensure data quality and maximize efficiency in the operation of a regional occupational injury surveillance system, the need for continued dual coding of occupational injury records was assessed. Methods: Kappa scores and percent agreement were used to compare interrater reliability for assigned variables in 1,259 agricultural and forestry injuries identified in pre-hospital care reports. The variables used for the comparison included type of event, source of injury, nature of injury, part of body, injury location, intentionality, and farm and agriculture injury classification (FAIC). Results: Kappa (κ) ranged from 0.2605 for secondary source to 0.8494 for event and exposure. Individual coder accuracy ranged from medium to high levels of agreement. Agreement beyond the first digit of OIICS coding was measured in percent agreement, and type of event or exposure, body part, and primary source of injury continued to meet levels of accord reaching 70% or greater agreement between all coders and the final choice, even to the most detailed 4th digit of OIICS. Conclusions: This research supports evidence-based decision making in customizing an occupational injury surveillance system, ultimately making it less costly while maintaining data quality. We foresee these methods being applicable to any surveillance system where visual inspection and human decisions are levied. Practical applications: Assessing the rigor of occupational injury record coding provides critical information to tailor surveillance protocols, especially those targeted to make the system less costly. System administrators should consider evaluating the quality of coding, especially when dealing with free-text narratives before deciding on single coder protocols. Further, quality checks should remain a part of the system going forward.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

ResearchGate has not been able to resolve any citations for this publication.
Article
Full-text available
Purpose Current injury surveillance efforts in agriculture are considerably hampered by the limited quantity of occupation or industry data in current health records. This has impeded efforts to develop more accurate injury burden estimates and has negatively impacted the prioritization of workplace health and safety in state and federal public health efforts. This paper describes the development of a Naïve Bayes machine learning algorithm to identify occupational injuries in agriculture using existing administrative data, specifically in pre-hospital care reports (PCR). Methods A Naïve Bayes machine learning algorithm was trained on PCR datasets from 2008–2010 from Maine and New Hampshire and tested on newer data from those states between 2011 and 2016. Further analyses were devoted to establishing the generalizability of the model across various states and various years. Dual visual inspection was used to verify the records subset by the algorithm. Results The Naïve Bayes machine learning algorithm reduced the volume of cases that required visual inspection by 69.5 percent over a keyword search strategy alone. Coders identified 341 true agricultural injury records (Case class = 1) (Maine 2011–2016, New Hampshire 2011–2015). In addition, there were 581 (Case class = 2 or 3) that were suspected to be agricultural acute/traumatic events, but lacked the necessary detail to make a certain distinction. Conclusions The application of the trained algorithm on newer data reduced the volume of records requiring visual inspection by two thirds over the previous keyword search strategy, making it a sustainable and cost-effective way to understand injury trends in agriculture.
Article
Full-text available
Objectives: To determine coders' agreement level for the Occupational Injury and Illness Classification System (OIICS) source of injury and injury event codes, and the Farm and Agricultural Injury Classification (FAIC) code in the AgInjuryNews.org and to determine the effects of supplemental information and follow-up discussion in final code assignments. Methods: Two independent researchers initially coded 1304 injury cases from AgInjurynews.org using the OIICS and the FAIC coding schemes. Code agreement levels for injury source, event, and FAIC and the effect of supplemental information and follow-up discussions on final coding was assessed. Results: Coders' agreement levels were almost perfect for OIICS source and event categories at the 3-digit level, with lower agreement at the 4-digit level. By using supplemental information and follow-up discussion, coders improved the coding accuracy by an average 20% for FAIC. Supplemental information and follow-up discussions had helped finalize the disagreed codes 55% of the time for OIICS source coding assignments and 40% of time for OIICS event coding assignments for most detailed 4-digit levels. Five key themes emerged regarding accurate and consistent coding of the agricultural injuries: inclusion/exclusion based on industry classification system; inconsistent/discrepant reports; incomplete/nonspecific reports; effects of supplemental information on coding; and differing interpretations of code selection rules. Practical applications: Quantifying the level of agreement for agricultural injuries will lead to a better understanding of coding discrepancies and may uncover areas for improvement to coding scheme itself. High level of initial and final agreement with FAIC and OIICS codes suggest that these coding schemes are user-friendly and amenable to widespread use.
Article
Full-text available
Commercial fishing is one of the most dangerous jobs in the United States, with a 2016 work-related fatality rate (86.0 deaths per 100,000 full-time equivalent workers) 23 times higher than that for all U.S. workers (3.6) (1). Sinking vessels cause the most fatalities in the industry; however, falling from a fishing vessel is a serious hazard responsible for the second highest number of commercial fishing-associated fatalities (2,3). CDC's National Institute for Occupational Safety and Health (NIOSH) analyzed data on unintentional fatal falls overboard in the U.S. commercial fishing industry to identify gaps in the use of primary, secondary, and tertiary prevention strategies. During 2000-2016, a total of 204 commercial fishermen died after unintentionally falling overboard. The majority of falls (121; 59.3%) were not witnessed, and 108 (89.3%) of these victims were not found. Among 83 witnessed falls overboard, 56 rescue attempts were made; 22 victims were recovered but were not successfully resuscitated. The circumstances, rescue attempts, and limited use of lifesaving and recovery equipment indicate that efforts to reduce these preventable fatalities are needed during pre-event, event, and post-event sequences of falls overboard. Vessel owners could consider strategies to prevent future fatalities, including lifeline tethers, line management, personal flotation devices (PFDs), man-overboard alarms, recovery devices, and rescue training.
Article
Full-text available
Background: This study focused on risk factors for serious injuries in farm and ranch operators in the central United States. Methods: The Central States Center for Agricultural Safety and Health, in collaboration with the National Agricultural Statistics Service, sent mail surveys to 6953, 6912, and 6912 farms/ranches in 2011-2013, respectively, covering seven Midwestern states. Results: The average survey response rate was 35%. The average annual incidence rate (injuries/100 workers) was 6.91 for all injuries and 2.40 for serious injuries. Univariate analyses determined several demographic and farm production-related risk factors for serious injury. Adjusted analysis showed a greater risk of serious injury for operators of age 45-54 years (vs. 65 and higher), those who worked 75-99% of their time (vs. less time), and those who operated larger land areas (vs. smaller). Conclusion: The identified risk factors should be considered when targeting injury prevention programs.
Article
Full-text available
The BLS Survey of Occupational Injuries and Illnesses offers many advantages over other data systems, and BLS has been working on improvements to increase its accuracy and scope; nevertheless, there is a debate about whether the survey undercounts injuries and illnesses to any significant extent
Article
Full-text available
The kappa statistic is frequently used to test interrater reliability. The importance of rater reliability lies in the fact that it represents the extent to which the data collected in the study are correct representations of the variables measured. Measurement of the extent to which data collectors (raters) assign the same score to the same variable is called interrater reliability. While there have been a variety of methods to measure interrater reliability, traditionally it was measured as percent agreement, calculated as the number of agreement scores divided by the total number of scores. In 1960, Jacob Cohen critiqued use of percent agreement due to its inability to account for chance agreement. He introduced the Cohen's kappa, developed to account for the possibility that raters actually guess on at least some variables due to uncertainty. Like most correlation statistics, the kappa can range from -1 to +1. While the kappa is one of the most commonly used statistics to test interrater reliability, it has limitations. Judgments about what level of kappa should be acceptable for health research are questioned. Cohen's suggested interpretation may be too lenient for health related studies because it implies that a score as low as 0.41 might be acceptable. Kappa and percent agreement are compared, and levels for both kappa and percent agreement that should be demanded in healthcare studies are suggested.
Article
Introduction Paramedics are involved in examining, treating and diagnosing patients. The accuracy of these diagnoses is evaluated using diagnostic accuracy studies. We undertook a systematic review of published literature to provide an overview of how accurately paramedics diagnose patients compared with hospital doctors. A bivariate meta-analysis was incorporated to examine the range of diagnostic sensitivity and specificity. Methods We searched MEDLINE, CINAHL, Embase, AMED and the Cochrane Database from 1946 to 7 May 2016 for studies where patients had been given a diagnosis by paramedics and hospital doctors. Keywords focused on study type (‘diagnostic accuracy’), outcomes (sensitivity, specificity, likelihood ratio?, predictive value?) and setting (paramedic*, pre-hospital, ambulance, ‘emergency service?’, ‘emergency medical service?’, ‘emergency technician?’). Results 2941 references were screened by title and/or abstract. Eleven studies encompassing 384 985 patients were included after full-text review. The types of diagnoses in one of the studies encompassed all possible diagnoses and in the other studies focused on sepsis, stroke and myocardial infarction. Sensitivity estimates ranged from 32% to 100% and specificity estimates from 14% to 100%. Eight of the studies were deemed to have a low risk of bias and were incorporated into a meta-analysis which showed a pooled sensitivity of 0.74 (0.62 to 0.82) and a pooled specificity of 0.94 (0.87 to 0.97). Discussion Current published research suggests that diagnoses made by paramedics have high sensitivity and even higher specificity. However, the paucity and varying quality of studies indicates that further prehospital diagnostic accuracy studies are warranted especially in the field of non-life-threatening conditions. PROSPERO registration number CRD42016039306.
Article
Background The AgInjuryNews system and dataset are a news report repository and information source for agricultural safety professionals, policymakers, journalists, and law enforcement officials. Methods AgInjuryNews was designed as a primary storage and retrieval system that allows users to: identify agricultural injury/fatality events; identify injury agents and emerging issues; provide safety messages for media in anticipation of trends; and raise awareness and knowledge of agricultural injuries and prevention strategies. Data are primarily collected through Google Alerts and a digital media subscription service. Articles are screened, reviewed, coded, and entered into the system. Results As of January 1, 2018, the system contained 3028 unique incidents. Of those, 650 involved youth, and 1807 were fatalities. The system also had registered 329 users from 39 countries. Conclusions AgInjuryNews combines injury reports into one dataset and may be the most current and comprehensive publicly available collection of news reports on agricultural injuries and deaths.
Article
This paper presents a general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies. The procedure essentially involves the construction of functions of the observed proportions which are directed at the extent to which the observers agree among themselves and the construction of test statistics for hypotheses involving these functions. Tests for interobserver bias are presented in terms of first-order marginal homogeneity and measures of interobserver agreement are developed as generalized kappa-type statistics. These procedures are illustrated with a clinical diagnosis example from the epidemiological literature.
Article
Debate surrounds the size of the underestimate of nonfatal occupational injuries produced by the U.S. Bureau of Labor Statistics (BLS). We developed models that separated categories of injuries: BLS Annual Survey, federal government, agriculture, state and local government, self-employed outside agriculture, and all other. The models generated varying estimates depending on the assumptions for each category pertaining to job risks and amount of underreporting. We offered justification for the assumptions based on published studies as well as our own analyses of BLS data. The models suggested the Annual Survey missed from 0% to 70% of the number of injuries (from private firms, excluding the self-employed) it was designed to capture. However, when we included firms and governments the Annual Survey was not designed to capture, and considered reasonable assumptions regarding underreporting, we estimated the BLS missed between 33% and 69% of all injuries. We concluded that there was substantial undercapture in the BLS Annual Survey, some due to the excluded categories of government workers and the self-employed, as well as some due to underreporting.
Article
Items such as physical exam findings, radiographic interpretations, or other diagnostic tests often rely on some degree of subjective interpretation by observers. Studies that measure the agreement between two or more observers should include a statistic that takes into account the fact that observers will sometimes agree or disagree simply by chance. The kappa statistic (or kappa coefficient) is the most commonly used statistic for this purpose. A kappa of 1 indicates perfect agreement, whereas a kappa of 0 indicates agreement equivalent to chance. A limitation of kappa is that it is affected by the prevalence of the finding under observation. Methods to overcome this limitation have been described.
NORA National Agriculture, Forestry, and Fishing Agenda for Occupational Safety and Health in Research and Practice in the U.S. Agriculture, Forestry, and Fishing Sector
  • F Nora Agricultural
  • Fishing Sector Council