Aaron M Cohen

Publications of Aaron M Cohen

• Studying the potential impact of automated document classification on scheduling a systematic review update.

  Authors: Aaron M Cohen, Kyle Ambert, Marian McDonagh

  BMC medical informatics and decision making. 04/2012; 12(1):33.

  ABSTRACT: BACKGROUND: Systematic Reviews (SRs) are an essential part of evidence-based medicine, providing support for clinical practice and policy on a wide range of medical topics. However,ABSTRACT: BACKGROUND: Systematic Reviews (SRs) are an essential part of evidence-based medicine, providing support for clinical practice and policy on a wide range of medical topics. However, producing SRs is resource-intensive, and progress in the research they review leads to SRs becoming outdated, requiring updates. Although the question of how and when to update SRs has been studied, the best method for determining when to update is still unclear, necessitating further research. METHODS: In this work we study the potential impact of a machine learning-based automated system for providing alerts when new publications become available within an SR topic. Some of these new publications are especially important, as they report findings that are more likely to initiate a review update. To this end, we have designed a classification algorithm to identify articles that are likely to be included in an SR update, along with an annotation scheme designed to identify the most important publications in a topic area. Using an SR database containing over 70,000 articles, we annotated articles from 9 topics that had received an update during the study period. The algorithm was then evaluated in terms of the overall correct and incorrect alert rate for publications meeting the topic inclusion criteria, as well as in terms of its ability to identify important, update-motivating publications in a topic area. RESULTS: Our initial approach, based on our previous work in topic-specific SR publication classification, identifies over 70% of the most important new publications, while maintaining a low overall alert rate. CONCLUSIONS: We performed an initial analysis of the opportunities and challenges in aiding the SR update planning process with an informatics-based machine learning approach. Alerts could be a useful tool in the planning, scheduling, and allocation of resources for SR updates, providing an improvement in timeliness and coverage for the large number of medical topics needing SRs. While the performance of this initial method is not perfect, it could be a useful supplement to current approaches to scheduling an SR update. Approaches specifically targeting the types of important publications identified by this work are likely to improve results.

• k-Information Gain Scaled Nearest Neighbors: A Novel Approach to Classifying Protein-Protein Interaction-Related Documents.

  Authors: Kyle H Ambert, Aaron M Cohen

  IEEE/ACM transactions on computational biology and bioinformatics / IEEE, ACM. 02/2011;

  Although publicly-accessible databases containing protein-protein interaction (PPI)-related information are important resources to bench and in silico research scientists alike, the amount of timeAlthough publicly-accessible databases containing protein-protein interaction (PPI)-related information are important resources to bench and in silico research scientists alike, the amount of time and effort required to keep them up to date is often burdonsome. In an effort to help identify relevant PPI publications, text-mining tools, from the machine learning discipline, can be applied to help in this process. Here, we describe and evaluate two document classification algorithms we submitted to the BioCreative II.5 PPI Classification Challenge Task. This task asked participants to design classifiers for identifying documents containing PPI-related information in the primary literature, and evaluated them against one another. One of our systems was the overall best-performing system submitted to the challenge task. It utilizes a novel approach to k-nearest neighbor classification, which we describe here, and compare it's performance to those of two support vector machine-based classification systems, one of which was also evaluated in the challenge task.

• Understanding Inter-rater Disagreement: A Mixed Methods Approach.

  Authors: Emily M Campbell, Dean F Sittig, Wendy W Chapman, Brian L Hazlehurst, Aaron M Cohen

  AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium. 01/2010; 2010:81-5.

  In an experiment to investigate cognitive skill differences between clinicians and lay persons, eight individuals in each group were asked to determine if an explicit concept existed in an ambulatoryIn an experiment to investigate cognitive skill differences between clinicians and lay persons, eight individuals in each group were asked to determine if an explicit concept existed in an ambulatory encounter note (a simple task) or if the concept could be inferred from the same note (a complex task). Subjects answered questions, highlighted text used to answer each question, and commented on their reasoning for selecting specific text. Quantitative results were mixed for expert vs. non-expert task performance on simple vs. complex tasks. Qualitative analysis revealed that data ambiguity obscured quantifiable skill differences between groups. In addition, this analysis offered new insight into whether a concept identification task is simple or complex. We present this case study to demonstrate the value of mixed method approaches to task-based performance study design and evaluation. We discuss the results in terms of their implications for evaluating meaningful use of technologies.

• A Prospective Evaluation of an Automated Classification System to Support Evidence-based Medicine and Systematic Review.

  Authors: Aaron M Cohen, Kyle Ambert, Marian McDonagh

  AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium. 01/2010; 2010:121-5.

  Systematic reviews (SR) are an important and labor-intensive part of the Evidence-based Medicine process that could benefit from automated literature classification tools. We conducted a prospectiveSystematic reviews (SR) are an important and labor-intensive part of the Evidence-based Medicine process that could benefit from automated literature classification tools. We conducted a prospective study of a support vector machine-based classifier for supporting the SR literature triage process. Over 50,000 training data samples were collected for 18 topics prior to March 2008, and used to make predictions on 11,000 test data samples collected during the subsequent two years. Test performance (AUC) was comparable to that estimated by cross-validation on the training set, and ranging from 0.75 - 0.99. Mean AUC macro-averaged across all topics was 0.89, demonstrating that these methods can achieve accurate results in near-real world conditions and are promising tools for deployment to groups conducting SRs.

• Cross-topic Learning for Work Prioritization in Systematic Review Creation and Update.

  Authors: Aaron M Cohen, Kyle Ambert, Marian McDonagh

  Journal of the American Medical Informatics Association : JAMIA. 07/2009;

  OBJECTIVE Machine learning systems can be an aid to experts performing systematic reviews (SRs) by automatically ranking journal articles for work-prioritization. The present work investigatesOBJECTIVE Machine learning systems can be an aid to experts performing systematic reviews (SRs) by automatically ranking journal articles for work-prioritization. The present work investigates whether a topic-specific automated document ranking system for SRs can be improved by using a hybrid approach, combining topic-specific training data with data from other SR topics. DESIGN A test collection was built using annotated reference files from 24 systematic drug class reviews. A support vector machine learning algorithm was evaluated with cross-validation, using seven different fractions of topic-specific training data in combination with samples from the other 23 topics. This approach was compared to both a baseline system, which used only topic-specific training data, and to a system using only the non-topic data sampled from the remaining topics. MEASUREMENTS Mean area under the receiver-operating curve (AUC) was used as the measure of comparison. RESULTS On average, the hybrid system improved mean AUC over the baseline system by 20%, when topic-specific training data was scarce. The system performed significantly better than the baseline system at all levels of topic-specific training data. In addition, the system performed better than the non-topic system at all but the two smallest fractions of topic specific training data, and no worse than the non-topic system with these smallest amounts of topic specific training data. CONCLUSIONS Automated literature prioritization could be helpful in assisting experts to organize their time when performing systematic reviews. Future work will focus on extending the algorithm to use additional sources of topic-specific data, and on embedding the algorithm in an interactive system available to systematic reviewers during the literature review process.

• A System for Classifying Disease Co-morbidity Status from Medical Discharge Summaries Using Automated Hotspot and Negated Concept Detection.

  Authors: Kyle H Ambert, Aaron M Cohen

  Journal of the American Medical Informatics Association : JAMIA. 05/2009;

  OBJECTIVE Free-text clinical reports serve as an important part of patient care management and clinical documentation of patient disease and treatment status. Free-text notes are commonplace inOBJECTIVE Free-text clinical reports serve as an important part of patient care management and clinical documentation of patient disease and treatment status. Free-text notes are commonplace in medical practice, but remain an under-utilized source of information for clinical and epidemiological research, as well as personalized medicine. We explore the challenges associated with automatically extracting information from clinical reports using our submission to the Integrating Informatics with Biology and the Bedside (i2b2) 2008 Natural Language Processing Obesity Challenge Task. DESIGN A text mining system for classifying patient comorbidity status, based on the information contained in clinical reports. Our approach incorporates a variety of automated techniques, including hot-spot filtering, negated concept identification, zero-vector filtering, weighting by inverse class-frequency, and error-correcting output codes with linear support vector machines. MEASUREMENTS Performance was evaluated in terms of the macro-averaged F1 measure. RESULTS Our automated system performed well against manual expert rule-based systems, finishing 5th in the Challenge's intuitive task, and 13th in the textual task. CONCLUSION Our system demonstrates that effective comorbidity status classification by an automated system is possible.

• Research Paper: A System for Classifying Disease Comorbidity Status from Medical Discharge Summaries Using Automated Hotspot and Negated Concept Detection.

  Authors: Kyle H. Ambert, Aaron M. Cohen

  JAMIA. 01/2009; 16:590-595.

• Research Paper: Cross-Topic Learning for Work Prioritization in Systematic Review Creation and Update.

  Authors: Aaron M. Cohen, Kyle H. Ambert, Marian McDonagh

  JAMIA. 01/2009; 16:690-704.

• Tasks, topics and relevance judging for the TREC Genomics Track: five years of experience evaluating biomedical text information retrieval systems.

  Authors: Phoebe M. Roberts, Aaron M. Cohen, William R. Hersh

  Inf. Retr. 01/2009; 12:81-97.

• A comparative analysis of retrieval features used in the TREC 2006 Genomics Track passage retrieval task.

  Authors: Hari Krishna Rekapalli, Aaron M Cohen, William R Hersh

  AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium. 02/2007;

  OBJECTIVE: Identify the set of features that best explained the variation in the performance measure of TREC 2006 Genomics information extraction task, Mean Average Passage Precision (MAPP). METHODS:OBJECTIVE: Identify the set of features that best explained the variation in the performance measure of TREC 2006 Genomics information extraction task, Mean Average Passage Precision (MAPP). METHODS: A multivariate regression model was built using a backward-elimination approach as a function of certain generalized features that were common to all the algorithms used by TREC 2006 Genomics track participants. RESULTS: Our regression analysis found that the following four factors were collectively associated with variation in MAPP: (1) Normalization of keywords in the query (2) Use of Entrez gene thesaurus for synonymous terms look-up (3) Unit of text retrieved using respective IR algorithms and (4) The way a passage was defined. CONCLUSION: These reasonably likely hypotheses, generated by an exploratory data analysis, are informative in understanding results of the TREC 2006 Genomics passage extraction task. This approach has general value for analyzing the results of similar common challenge tasks.

• Automatic summarization of mouse gene information by clustering and sentence extraction from MEDLINE abstracts.

  Authors: Jianji Yang, Aaron M Cohen, William Hersh

  AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium. 02/2007;

  Tools to automatically summarize gene information from the literature have the potential to help genomics researchers better interpret gene expression data and investigate biological pathways. TheTools to automatically summarize gene information from the literature have the potential to help genomics researchers better interpret gene expression data and investigate biological pathways. The task of finding information on sets of genes is common for genomic researchers, and PubMed is still the first choice because the most recent and original information can only be found in the unstructured, free text biomedical literature. However, finding information on a set of genes by manually searching and scanning the literature is a time-consuming and daunting task for scientists. We built and evaluated a query-based automatic summarizer of information on mouse genes studied in microarray experiments. The system clusters a set of genes by MeSH, GO and free text features and presents summaries for each gene by ranked sentences extracted from MEDLINE abstracts. Evaluation showed that the system seems to provide meaningful clusters and informative sentences are ranked higher by the algorithm.

• TREC 2007 Genomics Track Overview.

  Authors: William R. Hersh, Aaron M. Cohen, Lynn Ruslen, Phoebe M. Roberts

  Proceedings of The Sixteenth Text REtrieval Conference, TREC 2007, Gaithersburg, Maryland, USA, November 5-9, 2007; 01/2007

• The OHSU Biomedical Question Answering System Framework.

  Authors: Aaron M. Cohen, Jianji Yang, Seeger Fisher, Brian Roark, William R. Hersh

  Proceedings of The Sixteenth Text REtrieval Conference, TREC 2007, Gaithersburg, Maryland, USA, November 5-9, 2007; 01/2007

• The TREC 2004 genomics track categorization task: classifying full text biomedical documents.

  Authors: Aaron M Cohen, William R Hersh

  Journal of biomedical discovery and collaboration. 02/2006; 1:4.

  BACKGROUND: The TREC 2004 Genomics Track focused on applying information retrieval and text mining techniques to improve the use of genomic information in biomedicine. The Genomics Track consisted ofBACKGROUND: The TREC 2004 Genomics Track focused on applying information retrieval and text mining techniques to improve the use of genomic information in biomedicine. The Genomics Track consisted of two main tasks, ad hoc retrieval and document categorization. In this paper, we describe the categorization task, which focused on the classification of full-text documents, simulating the task of curators of the Mouse Genome Informatics (MGI) system and consisting of three subtasks. One subtask of the categorization task required the triage of articles likely to have experimental evidence warranting the assignment of GO terms, while the other two subtasks were concerned with the assignment of the three top-level GO categories to each paper containing evidence for these categories. RESULTS: The track had 33 participating groups. The mean and maximum utility measure for the triage subtask was 0.3303, with a top score of 0.6512. No system was able to substantially improve results over simply using the MeSH term Mice. Analysis of significant feature overlap between the training and test sets was found to be less than expected. Sample coverage of GO terms assigned to papers in the collection was very sparse. Determining papers containing GO term evidence will likely need to be treated as separate tasks for each concept represented in GO, and therefore require much denser sampling than was available in the data sets. The annotation subtask had a mean F-measure of 0.3824, with a top score of 0.5611. The mean F-measure for the annotation plus evidence codes subtask was 0.3676, with a top score of 0.4224. Gene name recognition was found to be of benefit for this task. CONCLUSION: Automated classification of documents for GO annotation is a challenging task, as was the automated extraction of GO code hierarchies and evidence codes. However, automating these tasks would provide substantial benefit to biomedical curation, and therefore work in this area must continue. Additional experience will allow comparison and further analysis about which algorithmic features are most useful in biomedical document classification, and better understanding of the task characteristics that make automated classification feasible and useful for biomedical document curation. The TREC Genomics Track will be continuing in 2005 focusing on a wider range of triage tasks and improving results from 2004.

• Enhancing access to the Bibliome: the TREC 2004 Genomics Track.

  Authors: William R Hersh, Ravi Teja Bhupatiraju, Laura Ross, Phoebe Roberts, Aaron M Cohen, Dale F Kraemer

  Journal of biomedical discovery and collaboration. 02/2006; 1:3.

  BACKGROUND: The goal of the TREC Genomics Track is to improve information retrieval in the area of genomics by creating test collections that will allow researchers to improve and better understandBACKGROUND: The goal of the TREC Genomics Track is to improve information retrieval in the area of genomics by creating test collections that will allow researchers to improve and better understand failures of their systems. The 2004 track included an ad hoc retrieval task, simulating use of a search engine to obtain documents about biomedical topics. This paper describes the Genomics Track of the Text Retrieval Conference (TREC) 2004, a forum for evaluation of IR research systems, where retrieval in the genomics domain has recently begun to be assessed. RESULTS: A total of 27 research groups submitted 47 different runs. The most effective runs, as measured by the primary evaluation measure of mean average precision (MAP), used a combination of domain-specific and general techniques. The best MAP obtained by any run was 0.4075. Techniques that expanded queries with gene name lists as well as words from related articles had the best efficacy. However, many runs performed more poorly than a simple baseline run, indicating that careful selection of system features is essential. CONCLUSION: Various approaches to ad hoc retrieval provide a diversity of efficacy. The TREC Genomics Track and its test collection resources provide tools that allow improvement in information retrieval systems.

• Functional gene group summarization by clustering MEDLINE abstract sentences.

  Authors: Jianji Yang, Aaron M Cohen, William R Hersh

  AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium. 02/2006;

  Tools to automatically summarize functional gene group information from the biomedical literature will help genomics researchers both better interpret gene expression data and understand biologicalTools to automatically summarize functional gene group information from the biomedical literature will help genomics researchers both better interpret gene expression data and understand biological pathways. In this study, we built a system that takes in a set of genes and MEDLINE records and outputs clusters of genes along with summaries of each cluster by sentence extraction from MEDLINE abstracts. Our preliminary use-case evaluation shows that this approach can identify gene clusters similar to manually generated groupings.

• An effective general purpose approach for automated biomedical document classification.

  Authors: Aaron M Cohen

  AMIA ... Annual Symposium proceedings / AMIA Symposium. AMIA Symposium. 02/2006;

  Automated document classification can be a valuable tool for biomedical tasks that involve large amounts of text. However, in biomedicine, documents that have the desired properties are often rare,Automated document classification can be a valuable tool for biomedical tasks that involve large amounts of text. However, in biomedicine, documents that have the desired properties are often rare, and special methods are usually required to address this issue. We propose and evaluate a method of classifying biomedical text documents, optimizing for utility when misclassification costs are highly asymmetric between the positive and negative classes. The method uses chi-square feature selection and several iterations of cost proportionate rejection sampling followed by application of a support vector machine (SVM), combining the resulting classifier results with voting. It is straightforward, fast, and achieves competitive performance on a set of standardized biomedical text classification evaluation tasks. The method is a good general purpose approach for classifying biomedical text.

• Combining Lexicon Expansion, Information Retrieval, and Cluster-based Ranking for Biomedical Question Answering.

  Authors: Aaron M. Cohen, Jianji Yang, Seeger Fisher, Brian Roark, William R. Hersh

  Proceedings of the Fifteenth Text REtrieval Conference, TREC 2006, Gaithersburg, Maryland, November 14-17, 2006; 01/2006

• TREC 2006 Genomics Track Overview.

  Authors: William R. Hersh, Aaron M. Cohen, Phoebe M. Roberts, Hari Krishna Rekapalli

  Proceedings of the Fifteenth Text REtrieval Conference, TREC 2006, Gaithersburg, Maryland, November 14-17, 2006; 01/2006

• Research Paper: Reducing Workload in Systematic Review Preparation Using Automated Citation Classification.

  Authors: Aaron M. Cohen, William R. Hersh, K. Peterson, Po-Yin Yen

  JAMIA. 01/2006; 13:206-219.