Figure 2 - uploaded by Mahmoud Elbattah
Content may be subject to copyright.
Source publication
The advent of transformer models has allowed for tremendous progress in the Natural Language Processing (NLP) domain. Pretrained transformers could successfully deliver the state-of-the-art performance in a myriad of NLP tasks. This study presents an application of transformers to learn contextual embeddings from free-text triage notes, widely reco...
Context in source publication
Citations
... Additionally, we can include recent studies [25][26][27][28], the majority of which suggested models based on bidirectional encoder representations from transformers (BERT) to enable the identification of named entities. ...
Studies that use medical records are often impeded due to the information presented in narrative fields. However, recent studies have used artificial intelligence to extract and process secondary health data from electronic medical records. The aim of this study was to develop a neural network that uses data from unstructured medical records to capture information regarding symptoms, diagnoses, medications, conditions, exams, and treatment. Data from 30,000 medical records of patients hospitalized in the Clinical Hospital of the Botucatu Medical School (HCFMB), São Paulo, Brazil, were obtained, creating a corpus with 1200 clinical texts. A natural language algorithm for text extraction and convolutional neural networks for pattern recognition were used to evaluate the model with goodness-of-fit indices. The results showed good accuracy, considering the complexity of the model, with an F-score of 63.9% and a precision of 72.7%. The patient condition class reached a precision of 90.3% and the medication class reached 87.5%. The proposed neural network will facilitate the detection of relationships between diseases and symptoms and prevalence and incidence, in addition to detecting the identification of clinical conditions, disease evolution, and the effects of prescribed medications.
... The early classical word embedding algorithm, word2vec, was proposed by Mikolov et al. [36] and is used for feedforward neural network training to predict the next word and then the preceding word of a given word. Recent research introduced a new algorithm for calculating word embeddings: BERT, which is a milestone in unsupervised training language models based on transformers [37][38][39]. The transformer is represented by the bidirectional encoder and decoder and uses a sequence-to-sequence model built using an attention mechanism. ...
Knowledge extraction from rich text in online health communities can supplement and improve the existing knowledge base, supporting evidence-based medicine and clinical decision making. The extracted time series health management data of users can help users with similar conditions when managing their health. By annotating four relationships, this study constructed a deep learning model, BERT-BiGRU-ATT, to extract disease–medication relationships. A Chinese-pretrained BERT model was used to generate word embeddings for the question-and-answer data from online health communities in China. In addition, the bidirectional gated recurrent unit, combined with an attention mechanism, was employed to capture sequence context features and then to classify text related to diseases and drugs using a softmax classifier and to obtain the time series data provided by users. By using various word embedding training experiments and comparisons with classical models, the superiority of our model in relation to extraction was verified. Based on the knowledge extraction, the evolution of a user’s disease progression was analyzed according to the time series data provided by users to further analyze the evolution of the user’s disease progression. BERT word embedding, GRU, and attention mechanisms in our research play major roles in knowledge extraction. The knowledge extraction results obtained are expected to supplement and improve the existing knowledge base, assist doctors’ diagnosis, and help users with dynamic lifecycle health management, such as user disease treatment management. In future studies, a co-reference resolution can be introduced to further improve the effect of extracting the relationships among diseases, drugs, and drug effects.
... This BERT model learned contextual embeddings. In this research, the utilization of the BERT method in the medical care field was discussed [15]. Tan et al. developed a rule-based NLP system and a machine learningbased NLP system to identify lumbar spine imaging findings related to LBP and compared their performance [16]. ...
... Recently, NLP system research on clinical reports written using official language has been conducted globally. Emilien et al. developed the NLP system for learning contextual embeddings from free-text form clinical records in France [15]. Kim Y et al. developed an NLP system for identifying a Korean medical corpus using BERT models [37]. ...
A natural language processing (NLP) pipeline was developed to identify lumbar spine imaging findings associated with low back pain (LBP) in X-radiation (X-ray), computed tomography (CT), and magnetic resonance imaging (MRI) reports. A total of 18,640 report datasets were randomly sampled (stratified by imaging modality) to obtain a balanced sample of 300 X-ray, 300 CT, and 300 MRI reports. A total of 23 radiologic findings potentially related to LBP were defined, and their presence was extracted from radiologic reports. In developing NLP pipelines, section and sentence segmentation from the radiology reports was performed using a rule-based method, including regular expression with negation detection. Datasets were randomly split into 80% for development and 20% for testing to evaluate the model’s extraction performance. The performance of the NLP pipeline was evaluated by using recall, precision, accuracy, and the F1 score. In evaluating NLP model performances, four parameters—recall, precision, accuracy, and F1 score—were greater than 0.9 for all 23 radiologic findings. These four scores were 1.0 for 10 radiologic findings (listhesis, annular fissure, disc bulge, disc extrusion, disc protrusion, endplate edema or Type 1 Modic change, lateral recess stenosis, Schmorl’s node, osteophyte, and any stenosis). In the seven potentially clinically important radiologic findings, the F1 score ranged from 0.9882 to 1.0. In this study, a rule-based NLP system identifying 23 findings related to LBP from X-ray, CT, and MRI reports was developed, and it presented good performance in regards to the four scoring parameters.
... Song Wang et al. extracted information nodes from radiology reports and constructed knowledge maps to assist the generation of reports [25]. Arnaud et al. used BERT to learn text representation from French free text note and check the quality of the learned embedding based on a set of clustering models [26]. ...
To investigate the feasibility of automated follow-up recommendations based on findings in radiology reports, this paper proposed a Natural Language Processing model specific for Pulmonary Nodule Radiology Reports. Unstructured findings used to describe pulmonary nodules in 48,091 radiology reports were processed in this study. We established an NLP model to extract information entities from findings of radiology reports, using deep learning and conditional random-field algorithms. Subsequently, we constructed a knowledge graph comprising 168 entities and four relationships, based on the export recommendations of the internationally renowned Fleischner Society for pulmonary nodules. These were employed in combination with rule templates to automatically generate follow-up recommendations. The automatically generated recommendations were then compared to the impression part of the reports to evaluate the matching rate of proper follow ups in the current situation. The NLP model identified eight types of entities with a recognition accuracy of up to 94.22%. A total of 43,898 out of 48,091 clinical reports were judged to contain appropriate follow-up recommendations, corresponding to the matching rate of 91.28%. The results show that NLP can be used on Chinese radiology reports to extract structured information at the content level, thereby realizing the prompt and intelligent follow-up suggestion generation or post-quality management of follow-up recommendations.
... In order to achieve a competitive advantage, organizations need to be prepared for and regularly updated on future trends and applications with the help of business intelligence (BI) tools [2]. In this vein, the use of machine-learning techniques is becoming irreplaceable in knowledge management [3,4] from Industry 4.0 in general [5] to healthcare [6], spanning a wide range from predicting diseases [7][8][9] to high-level knowledge extraction [10,11]. ...
... For example, the Bidirectional Encoder Representations from Transformers (BERT) [23] model was used by Arnaud et al. to learn contextual embeddings from free-text triage notes of a French hospital's emergency department [10]. BERT models were also used by Harnoune et al. to extract knowledge from biomedical clinical notes [11]. ...
... Developing NLP-based frameworks for extracting knowledge and actionable insights from medical and clinical data is at the heart of knowledge management in the healthcare industry [4,[7][8][9][10][11]25,28]. Aligned with these research efforts, the main purpose of this work is to provide a framework to extract knowledge management from business intelligence for the scholars and professionals in the healthcare knowledge-management field. ...
This paper proposes a framework to extract knowledge-management elements from business systems in healthcare organizations. According to results of in-depth interviews with experts in the field, a framework is defined, and software was developed to generate log files. Following the application of the Bag of Words (BoW) method on log files of 455 days for feature extraction, the k-means algorithm was used to cluster the feature vectors. The framework was tested with queries for confirmation. The developed framework successfully clustered the generated reports at operational, tactical, and strategic levels to extract knowledge-management elements. This study provides evidence for the knowledge-management pyramid by finding that the generated reports are reviewed mostly at the operational level, then tactical, and then the least at the strategic level. Our framework has the potential to be used not only in the health sector, but also in banking, insurance, and other businesses using business intelligence, especially in accordance with the organization’s goals at operational, tactical, and strategic levels of the knowledge-management pyramid.
... Bidirectional encoder representations from transformers (BERT) is a contextualized embedding method that preserves the distance of meanings with multihead attention [17]. After pretrained on the relevant corpora and proper architecture modification, BERT extracts meaningful embeddings from clinical text [18,19]. ...
Background
Machine learning (ML) achieves better predictions of postoperative mortality than previous prediction tools. Free-text descriptions of the preoperative diagnosis and the planned procedure are available preoperatively. Because reading these descriptions helps anesthesiologists evaluate the risk of the surgery, we hypothesized that deep learning (DL) models with unstructured text could improve postoperative mortality prediction. However, it is challenging to extract meaningful concept embeddings from this unstructured clinical text.
Objective
This study aims to develop a fusion DL model containing structured and unstructured features to predict the in-hospital 30-day postoperative mortality before surgery. ML models for predicting postoperative mortality using preoperative data with or without free clinical text were assessed.
Methods
We retrospectively collected preoperative anesthesia assessments, surgical information, and discharge summaries of patients undergoing general and neuraxial anesthesia from electronic health records (EHRs) from 2016 to 2020. We first compared the deep neural network (DNN) with other models using the same input features to demonstrate effectiveness. Then, we combined the DNN model with bidirectional encoder representations from transformers (BERT) to extract information from clinical texts. The effects of adding text information on the model performance were compared using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC). Statistical significance was evaluated using P<.05.
Results
The final cohort contained 121,313 patients who underwent surgeries. A total of 1562 (1.29%) patients died within 30 days of surgery. Our BERT-DNN model achieved the highest AUROC (0.964, 95% CI 0.961-0.967) and AUPRC (0.336, 95% CI 0.276-0.402). The AUROC of the BERT-DNN was significantly higher compared to logistic regression (AUROC=0.952, 95% CI 0.949-0.955) and the American Society of Anesthesiologist Physical Status (ASAPS AUROC=0.892, 95% CI 0.887-0.896) but not significantly higher compared to the DNN (AUROC=0.959, 95% CI 0.956-0.962) and the random forest (AUROC=0.961, 95% CI 0.958-0.964). The AUPRC of the BERT-DNN was significantly higher compared to the DNN (AUPRC=0.319, 95% CI 0.260-0.384), the random forest (AUPRC=0.296, 95% CI 0.239-0.360), logistic regression (AUPRC=0.276, 95% CI 0.220-0.339), and the ASAPS (AUPRC=0.149, 95% CI 0.107-0.203).
Conclusions
Our BERT-DNN model has an AUPRC significantly higher compared to previously proposed models using no text and an AUROC significantly higher compared to logistic regression and the ASAPS. This technique helps identify patients with higher risk from the surgical description text in EHRs.
... The embeddings in this paper are word embeddings learned by Word2vec, which cannot solve the problem of polysemy, so it affects accuracies of this framework. The latest works, such as these studies [17,18], study methods of learning contextual embeddings in medical texts from the latest language models such as BERT. Next, the framework proposed in this paper can be naturally extended to contextual embeddings to improve accuracies. ...
The aim of Medical Knowledge Graph Completion is to automatically predict one of three parts (head entity, relationship, and tail entity) in RDF triples from medical data, mainly text data. Following their introduction, the use of pretrained language models, such as Word2vec, BERT, and XLNET, to complete Medical Knowledge Graphs has become a popular research topic. The existing work focuses mainly on relationship completion and has rarely solved entities and related triples. In this paper, a framework to predict RDF triples for Medical Knowledge Graphs based on word embeddings (named PTMKG-WE) is proposed, for the specific use for the completion of entities and triples. The framework first formalizes existing samples for a given relationship from the Medical Knowledge Graph as prior knowledge. Second, it trains word embeddings from big medical data according to prior knowledge through Word2vec. Third, it can acquire candidate triples from word embeddings based on analogies from existing samples. In this framework, the paper proposes two strategies to improve the relation features. One is used to refine the relational semantics by clustering existing triple samples. Another is used to accurately embed the expression of the relationship through means of existing samples. These two strategies can be used separately (called PTMKG-WE-C and PTMKG-WE-M, respectively), and can also be superimposed (called PTMKG-WE-C-M) in the framework. Finally, in the current study, PubMed data and the National Drug File-Reference Terminology (NDF-RT) were collected, and a series of experiments was conducted. The experimental results show that the framework proposed in this paper and the two improvement strategies can be used to predict new triples for Medical Knowledge Graphs, when medical data are sufficiently abundant and the Knowledge Graph has appropriate prior knowledge. The two strategies designed to improve the relation features have a significant effect on the lifting precision, and the superposition effect becomes more obvious. Another conclusion is that, under the same parameter setting, the semantic precision of word embedding can be improved by extending the breadth and depth of data, and the precision of the prediction framework in this paper can be further improved in most cases. Thus, collecting and training big medical data is a viable method to learn more useful knowledge.
