Article

DeepGestalt - Identifying Rare Genetic Syndromes Using Deep Learning

Authors:
To read the full-text of this research, you can request a copy directly from the authors.

Abstract

Facial analysis technologies have recently measured up to the capabilities of expert clinicians in syndrome identification. To date, these technologies could only identify phenotypes of a few diseases, limiting their role in clinical settings where hundreds of diagnoses must be considered. We developed a facial analysis framework, DeepGestalt, using computer vision and deep learning algorithms, that quantifies similarities to hundreds of genetic syndromes based on unconstrained 2D images. DeepGestalt is currently trained with over 26,000 patient cases from a rapidly growing phenotype-genotype database, consisting of tens of thousands of validated clinical cases, curated through a community-driven platform. DeepGestalt currently achieves 91% top-10-accuracy in identifying over 215 different genetic syndromes and has outperformed clinical experts in three separate experiments. We suggest that this form of artificial intelligence is ready to support medical genetics in clinical and laboratory practices and will play a key role in the future of precision medicine.

No full-text available

Request Full-text Paper PDF

To read the full-text of this research,
you can request a copy directly from the authors.

... Automated morphological assessments have become available in recent years and may promote resource savings, along with the ability to accurately and objectively assess morphological features. Research regarding the use of these types of technologies in the assessment of NDDs and other disabilities are emerging (Aldridge et al., 2011;Gurovich et al., 2017;Gurovich et al., 2019;Lumaka et al., 2017;Obafemi-Ajayi et al., 2015). One of the early studies on the use of automated assessments in NDDs was published by Aldridge et al. (2011) and involved the use of a digital system More recent articles have been published on the use of the automated facial dysmorphology analysis, such as Face2Gene (FDNA, Boston, MA), a proprietary product, and the technology behind it that has recently been described as DeepGestalt (Gurovich et al., 2017;Gurovich et al., 2019). ...
... Research regarding the use of these types of technologies in the assessment of NDDs and other disabilities are emerging (Aldridge et al., 2011;Gurovich et al., 2017;Gurovich et al., 2019;Lumaka et al., 2017;Obafemi-Ajayi et al., 2015). One of the early studies on the use of automated assessments in NDDs was published by Aldridge et al. (2011) and involved the use of a digital system More recent articles have been published on the use of the automated facial dysmorphology analysis, such as Face2Gene (FDNA, Boston, MA), a proprietary product, and the technology behind it that has recently been described as DeepGestalt (Gurovich et al., 2017;Gurovich et al., 2019). The DeepGestalt algorithm in Face2Gene analyzes two-dimensional (2D) photos to detect morphological patterns and relationships of these patterns to genetic syndromes (Gripp, Baker, Telegrafi, & Monaghan, 2016;Gurovich et al., 2017;Gurovich et al., 2019). ...
... One of the early studies on the use of automated assessments in NDDs was published by Aldridge et al. (2011) and involved the use of a digital system More recent articles have been published on the use of the automated facial dysmorphology analysis, such as Face2Gene (FDNA, Boston, MA), a proprietary product, and the technology behind it that has recently been described as DeepGestalt (Gurovich et al., 2017;Gurovich et al., 2019). The DeepGestalt algorithm in Face2Gene analyzes two-dimensional (2D) photos to detect morphological patterns and relationships of these patterns to genetic syndromes (Gripp, Baker, Telegrafi, & Monaghan, 2016;Gurovich et al., 2017;Gurovich et al., 2019). To use Face2Gene, a user submits a photo through the Face2Gene web platform that is available free-of-charge to medical professionals and researchers. ...
Article
Full-text available
Physical examinations are recommended as part of a comprehensive evaluation for individuals with neurodevelopmental disorders (NDDs), such as autism spectrum disorder (ASD) and attention–deficit/hyperactivity disorder. These examinations should include assessment for morphological variants. Previous studies have shown an increase in morphological variants in individuals with NDDs, particularly ASD, and that these variants may be present in greater amounts in individuals with genetic alterations. Unfortunately, assessment for morphological variants can be subjective and time‐consuming, and require a high degree of clinical expertise. Therefore, objective, automated methods of morphological assessment are desirable. This study compared the use of Face2Gene, an automated tool to explore facial morphological variants, to clinical consensus assessment, using a cohort of N = 290 twins enriched for NDDs (n = 135 with NDD diagnoses). Agreement between automated and clinical assessments were satisfactory to complete (78.3–100%). In our twin sample, individuals with NDDs did not have greater numbers of facial morphological variants when compared to those with typical development, nor when controlling for shared genetic and environmental factors within twin pairs. Common facial morphological variants in those with and without NDDs were similar and included thick upper lip vermilion, abnormality of the nasal tip, long face, and upslanted palpebral fissure. We conclude that although facial morphological variants can be assessed reliably in NDDs with automated tools like Face2Gene, clinical utility is limited when just exploring the facial region. Therefore, currently, automated assessments may best complement, rather than replace, in‐person clinical assessments.
... Beyond language, capturing indicative patterns through deep-learning approaches has recently gained attention in assessing facial dysmorphism. 4,5 Artificial neural networks measure the similarities of patient photos to hundreds of disease entities. We hypothesized that results of this nextgeneration phenotyping tool could be used similarly to deleteriousness scores on the molecular level. ...
... By removing disorders that are confirmed by tests other than exome sequencing, such as Down syndrome (Supplementary Table 2), we ended up with 260 of 329 cases from the DeepGestalt set. 5 The facial images were analyzed with DeepGestalt, a deep convolutional neural network trained on more than 17,000 patient images. 5 The results of this analysis are gestalt scores that quantify the similarity to 216 different rare phenotypes per individual. ...
... By removing disorders that are confirmed by tests other than exome sequencing, such as Down syndrome (Supplementary Table 2), we ended up with 260 of 329 cases from the DeepGestalt set. 5 The facial images were analyzed with DeepGestalt, a deep convolutional neural network trained on more than 17,000 patient images. 5 The results of this analysis are gestalt scores that quantify the similarity to 216 different rare phenotypes per individual. These vectors can also be used to identify duplicates in the DeepGestalt training set and test set without the need to access the original photos. ...
Article
Full-text available
Purpose: Phenotype information is crucial for the interpretation of genomic variants. So far it has only been accessible for bioinformatics workflows after encoding into clinical terms by expert dysmorphologists. Methods: Here, we introduce an approach driven by artificial intelligence that uses portrait photographs for the interpretation of clinical exome data. We measured the value added by computer-assisted image analysis to the diagnostic yield on a cohort consisting of 679 individuals with 105 different monogenic disorders. For each case in the cohort we compiled frontal photos, clinical features, and the disease-causing variants, and simulated multiple exomes of different ethnic backgrounds. Results: The additional use of similarity scores from computer-assisted analysis of frontal photos improved the top 1 accuracy rate by more than 20-89% and the top 10 accuracy rate by more than 5-99% for the disease-causing gene. Conclusion: Image analysis by deep-learning algorithms can be used to quantify the phenotypic similarity (PP4 criterion of the American College of Medical Genetics and Genomics guidelines) and to advance the performance of bioinformatics pipelines for exome analysis.
... Beyond language, capturing indicative patterns through deep-learning approaches has recently gained attention in assessing facial dysmorphism. 4,5 Artificial neural networks measure the similarities of patient photos to hundreds of disease entities. We hypothesized that results of this nextgeneration phenotyping tool could be used similarly to deleteriousness scores on the molecular level. ...
... By removing disorders that are confirmed by tests other than exome sequencing, such as Down syndrome (Supplementary Table 2), we ended up with 260 of 329 cases from the DeepGestalt set. 5 The facial images were analyzed with DeepGestalt, a deep convolutional neural network trained on more than 17,000 patient images. 5 The results of this analysis are gestalt scores that quantify the similarity to 216 different rare phenotypes per individual. ...
... By removing disorders that are confirmed by tests other than exome sequencing, such as Down syndrome (Supplementary Table 2), we ended up with 260 of 329 cases from the DeepGestalt set. 5 The facial images were analyzed with DeepGestalt, a deep convolutional neural network trained on more than 17,000 patient images. 5 The results of this analysis are gestalt scores that quantify the similarity to 216 different rare phenotypes per individual. These vectors can also be used to identify duplicates in the DeepGestalt training set and test set without the need to access the original photos. ...
... Finally, the decision is handled with the help of the relationship between the lowest entropy value and other entropy values which are performed continuously until training the entire feature. After training the feature [16], it has been stored in the database for further genetic disease identification process. ...
Article
Full-text available
Text mining or analytics is important for various applications such as market analysis and biomedical purposes , it enables an efficient retrieval of information from large datasets. During the analysis, the dimensionality of data reduces the performance of an entire system because it may be retrieving irrelevant text, which creates malfunction . Therefore, this paper introduces big data and data mining techniques for analyzing large volumes of information while mining texts, emails, blogs, online forums, news, and call center documents. Initially, data are collected from various sources which contain noise, which is removed by applying normalization techniques. Data mining techniques eliminate irrelevant information and noise, and the relevant features are selected using the rough set-based particle swarm optimization (PSO) algorithm. The selected features are formed as a cluster using a fuzzy set with the PSO algorithm, which improves the efficiency of the mining process. Then, the efficiency of the system is evaluated with the help of the UC Irvine Machine Learning Repository(UCI) knowledge process mining database using the sum of intra cluster distance, mean square error rate, and accuracy.
... Finally, the decision is handled with the help of the relationship between the lowest entropy value and other entropy values which are performed continuously until training the entire feature. After training the feature [16], it has been stored in the database for further genetic disease identification process. ...
Article
Full-text available
Genetic diseases are the most common next-generation diseases because of the improper mutation of the genes and DNA. These genetic diseases are failed to predict with an accurate manner in the beginning stage by using the particular genes and related information. So, the genetic diseases are identified in the medical systems by utilizing the hybridization of multimedia techniques such as big data and related soft computing techniques.Initially, the genetic disease-related medical images are collected from healthcare sectors, and from the genetic image, various genetic data are collected from the large amount of datasets in which the major challenge is too high dimensionality that increases the complexity of the genetic disease prediction system. So, in this paper the complexity of the system is reduced by using the associative decision tree-based learning and Hopfield dynamic neural networks (HDNN). After collecting the data from the various resources, the immune clonal selection algorithm approach is used to remove inconsistent data and minimize the dimensionality of data. The selected features are trained by the proposed associative decision tree approach which helps to compare with the testing features using the HDNN that successfully recognize the genetic disease-based features effectively. The excellence of the system is measured with the aid of the experimental outcomes that are corresponding to the forecasting methods such as greedy algorithm, rough set method and artificial bee colony, and the comparison is made with the avail of the accuracy, sensitivity and specificity.
... fließt erstmals neben den bereits genannten Scores auch ein Ähnlichkeitswert aus der computergestützten Gesichtsanalyse ein. Bislang wird hierfür das künstliche neuronale Netzwerk DeepGestalt eingesetzt, welches auch in Face2Gene Verwendung findet [12]. Eine weitere Initiative, die einen ähnlichen Ansatz des maschinellen Lernens verwendet, ist Minerva&Me (https://www.minervaandme.com/). ...
Article
Full-text available
Background and methods Using high-throughput sequencing technology, a molecular cause can now be found in the majority of patients with intellectual disability. For the integration of whole genome sequencing into standard care, however, the use of effective filtering and prioritization methods is indispensable to be able to efficiently view quantities of data. Communication between clinic and laboratory, which enables the combination of phenotypic and molecular information, is crucial. In the Prioritization of Exome Data by Image Analysis (PEDIA) study, deep learning was used for the first time in image analysis to quantify syndromic similarities and make them usable for further bioinformatic evaluation. Results and conclusions In many monogenic diseases, facial abnormalities occur that are suitable for computer-assisted image analysis. Currently, a gestalt score can be calculated for approximately 300 different phenotypes, many of them with ID, using a portrait photo. In these diseases, the accuracy of the prioritization (top 1 accuracy rate) increases by approximately 20% if such similarity values are also taken into account. Deep learning will also play an important role in the evaluation of other medical image data in the future. To be able to use this technology in clinical decision support, it is necessary to adapt the IT infrastructure accordingly.
... Besides our previous studies, another facial analysis software, Face2Gene (FDNA, Boston, MA) has also been used to report on for the diagnosis of genetic syndromes in specialized (genetic) clinics. Its reported accuracy ranged between 60 and 69% (Gurovich et al., 2018(Gurovich et al., , 2019. Lumaka et al. (2017) evaluated the performance of Face2Gene when trained with a dataset that incorporated photographs of African patients from the DRC, Rwanda and France, similarly concluding that patient ancestry influences the evaluation of facial morphology. ...
Article
Full-text available
Down syndrome is one of the most common chromosomal anomalies affecting the world's population, with an estimated frequency of 1 in 700 live births. Despite its relatively high prevalence, diagnostic rates based on clinical features have remained under 70% for most of the developed world and even lower in countries with limited resources. While genetic and cytogenetic confirmation greatly increases the diagnostic rate, such resources are often non-existent in many low- and middle-income countries, particularly in Sub-Saharan Africa. To address the needs of countries with limited resources, the implementation of mobile, user-friendly and affordable technologies that aid in diagnosis would greatly increase the odds of success for a child born with a genetic condition. Given that the Democratic Republic of the Congo is estimated to have one of the highest rates of birth defects in the world, our team sought to determine if smartphone-based facial analysis technology could accurately detect Down syndrome in individuals of Congolese descent. Prior to technology training, we confirmed the presence of trisomy 21 using low-cost genomic applications that do not need advanced expertise to utilize and are available in many low-resourced countries. Our software technology trained on 132 Congolese subjects had a significantly improved performance (91.67% accuracy, 95.45% sensitivity, 87.88% specificity) when compared to previous technology trained on individuals who are not of Congolese origin (p < 5%). In addition, we provide the list of most discriminative facial features of Down syndrome and their ranges in the Congolese population. Collectively, our technology provides low-cost and accurate diagnosis of Down syndrome in the local population.
... Recent works show that AI applications are especially appropriate to detect and classify specific diseases, for example breast or skin cancer, in X-ray images [48,49]. Moreover, another AI application can identify gene-related diseases in face images of patients [50]. Generally, people tend to have optimistic perceptions of the use of AI in medicine [10]. ...
Article
Full-text available
In recent years Artificial Intelligence (AI) has gained much popularity, with the scientific community as well as with the public. Often, AI is ascribed many positive impacts for different social domains such as medicine and the economy. On the other side, there is also growing concern about its precarious impact on society and individuals, respectively. Several opinion polls frequently query the public fear of autonomous robots and artificial intelligence, a phenomenon coming also into scholarly focus. As potential threat perceptions arguably vary with regard to the reach and consequences of AI functionalities and the domain of application, research still lacks necessary precision of a respective measurement that allows for wide-spread research applicability. We propose a fine-grained scale to measure threat perceptions of AI that accounts for four functional classes of AI systems and is applicable to various domains of AI applications. Using a standardized questionnaire in a survey study (N = 891), we evaluate the scale over three distinct AI domains (medical treatment, job recruitment, and loan origination). The data support the dimensional structure of the proposed Threats of AI (TAI) scale as well as the internal consistency and factoral validity of the indicators. Implications of the results and the empirical application of the scale are discussed in detail. Recommendations for further empirical use of the TAI scale are provided.
... The resemblance between the face of a patient and the representation of a syndrome class can be quantified by the gestalt score obtained from Face2Gene. Here we used the gestalt scores of the top 30 syndromes suggested by Face2Gene's DeepGestalt 30 system as vectors for an indirect representation of each case in a high-dimensional vector space (Supplemental Fig. S-1 31 ) and the five novel PIGT cases to create a 215-dimensional syndrome space. A linear principal component analysis (PCA) was performed on the vector matrix to reduce representation to two and three dimensions, respectively ( Fig. 2 and Supplementary Figure S-1E). ...
Article
Full-text available
Purpose: To provide a detailed electroclinical description and expand the phenotype of PIGT-CDG, to perform genotype-phenotype correlation, and to investigate the onset and severity of the epilepsy associated with the different genetic subtypes of this rare disorder. Furthermore, to use computer-assisted facial gestalt analysis in PIGT-CDG and to the compare findings with other glycosylphosphatidylinositol (GPI) anchor deficiencies. Methods: We evaluated 13 children from eight unrelated families with homozygous or compound heterozygous pathogenic variants in PIGT. Results: All patients had hypotonia, severe developmental delay, and epilepsy. Epilepsy onset ranged from first day of life to two years of age. Severity of the seizure disorder varied from treatable seizures to severe neonatal onset epileptic encephalopathies. The facial gestalt of patients resembled that of previously published PIGT patients as they were closest to the center of the PIGT cluster in the clinical face phenotype space and were distinguishable from other gene-specific phenotypes. Conclusion: We expand our knowledge of PIGT. Our cases reaffirm that the use of genetic testing is essential for diagnosis in this group of disorders. Finally, we show that computer-assisted facial gestalt analysis accurately assigned PIGT cases to the multiple congenital anomalies-hypotonia-seizures syndrome phenotypic series advocating the additional use of next-generation phenotyping technology.
... Recent works show that AI applications are especially appropriate to detect and classify specific diseases, for example breast or skin cancer, in X-ray images [26,55]. Moreover, another AI application can identify gene-related diseases in face images of patients [21]. Generally, people tend to have optimistic perceptions of the use of AI in medicine [29]. ...
Preprint
Full-text available
In recent years Artificial Intelligence (AI) has gained much popularity, with the scientific community as well as with the public. AI is often ascribed many positive impacts for different social domains such as medicine and the economy. On the other side, there is also growing concern about its precarious impact on society and individuals. Several opinion polls frequently query the public fear of autonomous robots and artificial intelligence (FARAI), a phenomenon coming also into scholarly focus. As potential threat perceptions arguably vary with regard to the reach and consequences of AI functionalities and the domain of application, research still lacks necessary precision of a respective measurement that allows for wide-spread research applicability. We propose a fine-grained scale to measure threat perceptions of AI that accounts for four functional classes of AI systems and is applicable to various domains of AI applications. Using a standardized questionnaire in a survey study (N=891), we evaluate the scale over three distinct AI domains (loan origination, job recruitment and medical treatment). The data support the dimensional structure of the proposed Threats of AI (TAI) scale as well as the internal consistency and factoral validity of the indicators. Implications of the results and the empirical application of the scale are discussed in detail. Recommendations for further empirical use of the TAI scale are provided.
... It has been shown that they can be trained to be highly robust for imaging variation, reducing the need for highly controlled subject poses (Xiangyu Zhu et al., 2015). There are a number of current research and commercial efforts to create fully automated analysis pipelines for clinical interpretation of dysmorphologies (Ansari et al., 2014;Ferry et al., 2014;Manousaki et al., 2015;Basel-Vanagaite et al., 2016;Gripp et al., 2016;Baynam et al., 2017;Bengani et al., 2017;Dudding-Byth et al., 2017;Deciphering Developmental Disorders Study, 2017;Gardner et al., 2017;Hadj-Rabia et al., 2017;Kruszka et al., 2017a;Kruszka et al., 2017b;Kruszka et al., 2017c;Lumaka et al., 2017;Shukla et al., 2017;Valentine et al., 2017;Reijnders et al., 2018b;Gurovich et al., 2018;Knaus et al., 2018;Kruszka et al., 2018;Liehr et al., 2018;Pantel et al., 2018;Reijnders et al., 2018a;Reijnders et al., 2018b;Zarate et al., 2018). However, all these efforts are meeting the same barriers to progression of the methods and prospects for clinical impact, challenges to do with data access, ethics, governance, and security. ...
Article
Full-text available
The clinical utility of computational phenotyping for both genetic and rare diseases is increasingly appreciated; however, its true potential is yet to be fully realized. Alongside the growing clinical and research availability of sequencing technologies, precise deep and scalable phenotyping is required to serve unmet need in genetic and rare diseases. To improve the lives of individuals affected with rare diseases through deep phenotyping, global big data interrogation is necessary to aid our understanding of disease biology, assist diagnosis, and develop targeted treatment strategies. This includes the application of cutting-edge machine learning methods to image data. As with most digital tools employed in health care, there are ethical and data governance challenges associated with using identifiable personal image data. There are also risks with failing to deliver on the patient benefits of these new technologies, the biggest of which is posed by data siloing. The Minerva Initiative has been designed to enable the public good of deep phenotyping while mitigating these ethical risks. Its open structure, enabling collaboration and data sharing between individuals, clinicians, researchers and private enterprise, is key for delivering precision public health.
... These databases allow the comparison between the facial gestalt of patients and normal facial features, but these findings are affected by external factors such as available lighting conditions, posture of patients, etc. [Hammond, 2007]. In recent years, DeepGestalt program has been developed using computer technology and deep learning algorithms that measure similarities to hundreds of genetic syndromes based on unconstrained 2D images [Gurovich et al., 2018]. In a previous study, DeepGestalt achieved 91% top-10 correctness in identifying the syndrome on 502 different patient photos [Gurovich et al., 2019]. ...
Article
Full-text available
The diagnosis of rare genetic diseases is one of the most difficult areas in medicine. Whole-exome sequencing (WES) technology makes it easier to diagnose these diseases. In addition, next-generation phenotyping can help to diagnose computer-based algorithms. Detailed dysmorphologic findings of 25 patients diagnosed by WES in our center were described. The success of this technology in diagnosing rare genetic diseases was investigated by scanning the photographs of 25 patients with Face2Gene application. The application listed possible preliminary diagnoses (30 disease suggestion). Of these, 12 (48%) cases were correctly matched. The most common disease group in the patients was neurological disease (96%). The most common mode of inheritance in the patients was autosomal recessive. The rate of consanguineous marriages was determined in 80% of the patients. Ten patients had microcephaly and 7 patients had corpus callosum anomaly. In our study, we found that the success of Face2Gene was lower than described in the literature. We think that the probable cause of this condition is that the cases are very rare, and there is not enough data about these diseases in the application. Therefore, it is recommended that applications should be used more frequently by pediatricians and clinical geneticists. The diagnosis of rare diseases still is quite difficult. Nowadays, WES is a successful method. However, applications such as Face2Gene help to make a clinical prediagnosis and create a larger database.
Article
Full-text available
BACKGROUND: Giant transformations are going on currently in health care, and the greatest force behind this phenomenon is data. CONTENT: Big data has arrived into medicine field, lead to potential enhancement in accountability, quality, efficiency, and innovation. Most updated, artificial intelligence (AI) and machine-learning (ML) techniques rapidly developed, bring forth the big data analysis into more useful applications, from resource allocation to complex disease diagnosis. To realize this, a very large set of health-care data is needed for algorithms training and evaluation, including patients’ treatment data, patients respond to treatment, and personal patient information, such as genetic data, family history, health behavior, and vital signs. SUMMARY: Precision Health involving preventive, predictive, personalized and precise. The arrival of AI and ML will enhance and facilitates the improvement of this relationship through better accuracy, productivity, and workflow, thus develop a health system that will go beyond just curing disease, but further into wellness that preventing disease before it strikes, thus the patient–doctor bond is expected to be reformed and not be eroded. KEYWORDS: artificial intelligence, machine learning, deep learning, electronic health records, big data
Article
Introduction Phosphomannomutase-2 deficiency (PMM2-CDG) is associated with a recognisable facial pattern. There are no early severity predictors for this disorder and no phenotype–genotype correlation. We performed a detailed dysmorphology evaluation to describe facial gestalt and its changes over time, to train digital recognition facial analysis tools and to identify early severity predictors. Methods Paediatric PMM2-CDG patients were evaluated and compared with controls. A computer-assisted recognition tool was trained. Through the evaluation of dysmorphic features (DFs), a simple categorisation was created and correlated with clinical and neurological scores, and neuroimaging. Results Dysmorphology analysis of 31 patients (4–19 years of age) identified eight major DFs (strabismus, upslanted eyes, long fingers, lipodystrophy, wide mouth, inverted nipples, long philtrum and joint laxity) with predictive value using receiver operating characteristic (ROC) curveanalysis (p<0.001). Dysmorphology categorisation using lipodystrophy and inverted nipples was employed to divide patients into three groups that are correlated with global clinical and neurological scores, and neuroimaging (p=0.005, 0.003 and 0.002, respectively). After Face2Gene training, PMM2-CDG patients were correctly identified at different ages. Conclusions PMM2-CDG patients’ DFs are consistent and inform about clinical severity when no clear phenotype–genotype correlation is known. We propose a classification of DFs into major and minor with diagnostic risk implications. At present, Face2Gene is useful to suggest PMM2-CDG. Regarding the prognostic value of DFs, we elaborated a simple severity dysmorphology categorisation with predictive value, and we identified five major DFs associated with clinical severity. Both dysmorphology and digital analysis may help physicians to diagnose PMM2-CDG sooner.
Article
Full-text available
BACKGROUND: Current cancer drugs and treatments are aiming at eradicating tumor cells, but often are more toxic then effective, killing also the normal cells and not selectively the tumor cells. There is good personalized cancer therapy that involves administration to the cancer-bearing host of immune cells with direct anticancer activity, which called adoptive cell therapy (ACT). A review of the unique biology of T cell therapy and of recent clinical experience compels a reassessment of target antigens that traditionally have been viewed from the perspective of weaker immunotherapeutic modalities. CONTENT: Chimeric antigen receptors (CAR) are recombinant receptors which provide both antigen-binding and T cell-activating functions. Many kind of CARs has been reported for the past few years, targeting an array of cell surface tumor antigens. Their biologic functions have extremely changed following the introduction of tripartite receptors comprising a costimulatory domain, termed second-generation CARs. The combination of CARs with costimulatory ligands, chimeric costimulatory receptors, or cytokines can be done to further enhance T cell potency, specificity and safety. CARs reflects a new class of drugs with exciting potential for cancer immunotherapy. SUMMARY: CAR-T cells have been arising as a new modality for cancer immunotherapy because of their potent efficacy against terminal cancers. They are known to exert higher efficacy than monoclonal antibodies and antibodydrug conjugates, and act via mechanisms distinct from T cell receptor-engineered T cells. These cells are constructed by transducing genes encoding fusion proteins of cancer antigen-recognizing single-chain Fv linked to intracellular signaling domains of T cell receptors. KEYWORDS: chimeric antigen receptor, CAR T cells, adoptive cell therapy, ACT, T cell receptor, TCR, cancer, immunotherapy
Article
The use of artificial intelligence, and the deep-learning subtype in particular, has been enabled by the use of labeled big data, along with markedly enhanced computing power and cloud storage, across all sectors. In medicine, this is beginning to have an impact at three levels: for clinicians, predominantly via rapid, accurate image interpretation; for health systems, by improving workflow and the potential for reducing medical errors; and for patients, by enabling them to process their own data to promote health. The current limitations, including bias, privacy and security, and lack of transparency, along with the future directions of these applications will be discussed in this article. Over time, marked improvements in accuracy, productivity, and workflow will likely be actualized, but whether that will be used to improve the patient–doctor relationship or facilitate its erosion remains to be seen.
Conference Paper
Full-text available
Screening patients with genetic diseases using automated facial image analysis is an urgent task. A method for recognizing genetic syndromes from a frontal image of a face has been developed and studied. The classification was made to 8 syndromes (Angelman, Apert, Cornelia de Lange, Down, Fragile X, Progeria, Treacher Collins, Williams). Various types of features were investigated: geometric and deep features. For facial points localization, 3D face reconstruction was used (using the Deep3DFaceReconstruction library). Sets of 68 and 35709 (all points of 3D reconstruction) points were investigated. The effect of reducing the dimension of the feature vector on the classification accuracy is also investigated. According to the results of 5-fold cross-validation, the best average recognition accuracy was 92.5 % (combined features, Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA) and logistic regression), which is comparable to the results in similar works.
Article
Full-text available
One of the most pertinent applications of image analysis is face recognition and one of the most common genetic disorders is Down syndrome (DS), which is caused by chromosome abnormalities in humans. It is currently a challenge in computer vision in the domain of DS face recognition to build an automated system that equals the human ability to recognize face as one of the symmetrical structures in the body. Consequently, the use of machine learning methods has facilitated the recognition of facial dysmorphic features associated with DS. This paper aims to present a concise review of DS face recognition using the currently published literature by following the generic face recognition pipeline (face detection, feature extraction, and classification) and to identify critical knowledge gaps and directions for future research. The technologies underlying facial analysis presented in recent studies have helped expert clinicians in general genetic disorders and DS prediction.
Article
Personalized medicine is among the most exciting innovations in recent clinical research, offering the opportunity for tailored screening and management at the individual level. Biomarker-enriched clinical trials have shown increased efficiency and informativeness in cancer research due to the selective exclusion of patients unlikely to benefit. In acute stress situations, clinically significant decisions are often made in time-sensitive manners and providers may be pressed to make decisions based on abbreviated clinical assessments. Up to 30% of trauma survivors admitted to the Emergency Department (ED) will develop long-lasting posttraumatic stress psychopathologies. The long-term impact of those survivors with posttraumatic stress sequelae are significant, impacting both long-term psychological and physiological recovery. An accurate prognostic model of who will develop posttraumatic stress symptoms does not exist yet. Additionally, no scalable and cost-effective method that can be easily integrated into routine care exists, even though especially the acute care setting provides a critical window of opportunity for prevention in the so-called golden hours when preventive measures are most effective. In this review, we aim to discuss emerging machine learning (ML) applications that are promising for precisely risk stratification and targeted treatments in the acute care setting. The aim of this narrative review is to present examples of digital health innovations and to discuss the potential of these new approaches for treatment selection and prevention of posttraumatic sequelae in the acute care setting. The application of artificial intelligence-based solutions have already had great success in other areas and are rapidly approaching the field of psychological care as well. New ways of algorithm-based risk predicting, and the use of digital phenotypes provide a high potential for predicting future risk of PTSD in acute care settings and to go new steps in precision psychiatry.
Article
Full-text available
Introduction: About 0.2-6.1% of newborns in the developed world have been conceived by assisted reproductive techniques (ART). Higher rate of major and minor malformations have been described in this population, but the multiple possible confounders associated, make it difficult to establish a direct causal relationship, and the specific factors involved. Material and methods: To determine the risk of these malformations in our population, a collaborative prospective controlled cohort study was designed. We collected the specific ART-data related to the clinical gestation of women treated in a period of 2 years in the Reproduction Unit from a Spanish public tertiary-level hospital. 231 out of 267 newborns of these gestation (88%) were exhausted assessed by a Clinical Geneticist expertise in Dysmorphology at 12-20 and 26-40 months of age. At the same time a controlled group of children naturally conceived (NC) was selected according to the following criteria: the next NC newborn belonging to the same group of maternal and gestational age, and type of gestation (single or multiple). 230 controls were chosen and 208 participated in the study (90%). Results: Major malformations were presented in 7.8% of the ART-children and 7.2% of the controls, without founding statistically differences between groups. However, differences were found in the risk of some minor malformations such as capillary malformations and pigmentary lesions, higher in the ART-group. A recurrent pattern of craneofacial anomalies was also unexpectedly detected. Conclusions: In spite of the high rate of major congenital malformations detected, there were no differences between groups. Thus, our results suggest that ART may affect the normal embryonic development but in a milder way than other confounding factors do. The facial phenotype identified has not previously, either the higher risk of capillary malformations and pigmentary lesions. More studies are needed to confirm this association.
Article
Full-text available
Methyl CpG binding protein 2 (MECP2) is located at Xq28 and is a multifunctional gene with ubiquitous expression. Loss-of-function mutations in MECP2 are associated with Rett syndrome (RTT), which is a well-characterized disorder that affects mainly females. In boys, however, mutations in MECP2 can generate a wide spectrum of clinical presentations that range from mild intellectual impairment to severe neonatal encephalopathy and premature death. Thus, males can be more difficult to classify and diagnose than classical RTT females. In addition, there are some variants of unknown significance in MECP2, which further complicate the diagnosis of these children. Conversely, the entire duplication of the MECP2 gene is related to MECP2 duplication syndrome (MDS). Unlike in RTT, in MDS, males are predominantly affected. Usually, the duplication is inherited from an apparently asymptomatic carrier mother. Both syndromes share some characteristics, but also differ in some aspects regarding the clinical picture and evolution. In the following review, we present a thorough description of the different types of MECP2 variants and alterations that can be found in males, and explore several genotype–phenotype correlations, although there is still a lot to understand.
Article
Background: The unprecedented development of Artificial Intelligence has revolutionised the healthcare industry. In the next generation of healthcare systems, self-diagnosis will be pivotal to personalised healthcare services. During the COVID-19 pandemic, new screening and diagnostic approaches like mobile health are well-positioned to reduce disease spread and overcome geographical barriers. This paper presents a non-invasive screening approach to predict the health of a person from visually observable features using machine learning techniques. Images like face and skin surface of the patients are acquired using camera or mobile devices and analysed to derive clinical reasoning and prediction of the person’s health. Methods: In specific, a two-level classification approach is presented. The proposed hierarchical model chooses a class by training a binary classifier at the node of the hierarchy. Prediction is then made using a set of class-specific reduced feature set. Results: Testing accuracies of 86.87% and 76.84% are reported for the first and second-level classification. Empirical results demonstrate that the proposed approach yields favourable prediction results while greatly reduces the computational time. Conclusions: The study suggests that it is possible to predict the health condition of a person based on his/her face appearance using cost-effective machine learning approaches.
Article
Full-text available
Introduction About 0.2–6.1% of newborns in the developed world have been conceived by assisted reproductive techniques (ART). Higher rate of major and minor malformations have been described in this population, but the multiple possible confounders associated make it difficult to establish a direct causal relationship and the specific factors involved. Material and methods To determine the risk of these malformations in our population, a collaborative prospective controlled cohort study was designed. We collected the specific ART-data related to the clinical gestation of women treated in a period of 2 years in the Reproduction Unit from a Spanish public tertiary-level hospital. 231 out of 267 newborns of these gestation (88%) participated in the study and were assessed by a pediatrician with expertise in Clinical Genetics and Dysmorphology at 12–20 and 26–40 months of age. At the same time a controlled group of children naturally conceived (NC) was selected according to the following criteria: the next NC newborn belonging to the same group of maternal and gestational age, and type of gestation (single or multiple). 230 controls were chosen and 208 participated in the study (90%). Results Major malformations were presented in 7.8% of the ART-children and 7.2% of the controls, without founding statistically differences between groups. However, differences were found in the risk of some minor malformations such as capillary malformations and pigmentary lesions, higher in the ART-group. A recurrent pattern of craneofacial anomalies was also unexpectedly detected. Conclusions In spite of the high rate of major congenital malformations detected, there were no differences between groups. Thus, our results suggest that ART may affect the normal embryonic development but in a milder way than other confounding factors do. The facial phenotype identified has not previously been described, either the higher risk of capillary malformations and pigmentary lesions. More studies are needed to confirm this association.
Article
Craniofacial dysmorphism recognition is the first step in diagnosing most genetic syndromes. However, the number of genetic syndromes is enormous, and the specific facial features are difficult to memorize. For clinical practice, recent advances in artificial intelligence can be of use. One such tool, Face2Gene (FDNA, Inc., Boston, MA), is an innovative free group of applications, that helps clinicians recognize possible genetic syndromes from patients' facial two-dimensional photos. The initial data set used to train this technology consisted primarily of Caucasian patients. Because ethnic differences affect patients' facial features, the recognition probability in Asian patients may be limited. Our aim was to test the technology's recognition probability on Thai children with Down Syndrome (DS) as compared to Thai children without DS (non-DS). Two separate control groups of Thai non-DS children, either unaffected or having other syndromes, were included. Frontal photographs were obtained from all the participants. All 30 children with DS were recognized as DS in the top 10 syndrome-matches (100% sensitivity), and 27 were in the first ranking of suggested syndromes. Eighteen non-DS were recognized as DS (87.2% specificity) with an accuracy of 89%. We present a scientific basis for this novel tool, useful in the clinic where patients are of a different ethnicity unfamiliar to the evaluator. However, Face2Gene cannot be considered a replacement for clinicians' knowledge of phenotypes. Further studies on other genetic syndromes/ethnicities being identified by software algorithms are needed.
Article
Full-text available
We develop a face recognition algorithm which is insensitive to large variation in lighting direction and facial expression. Taking a pattern classification approach, we consider each pixel in an image as a coordinate in a high-dimensional space. We take advantage of the observation that the images of a particular face, under varying illumination but fixed pose, lie in a 3D linear subspace of the high dimensional image space-if the face is a Lambertian surface without shadowing. However, since faces are not truly Lambertian surfaces and do indeed produce self-shadowing, images will deviate from this linear subspace. Rather than explicitly modeling this deviation, we linearly project the image into a subspace in a manner which discounts those regions of the face with large deviation. Our projection method is based on Fisher's linear discriminant and produces well separated classes in a low-dimensional subspace, even under severe variation in lighting and facial expressions. The eigenface technique, another method based on linearly projecting the image space to a low dimensional subspace, has similar computational requirements. Yet, extensive experimental results demonstrate that the proposed “Fisherface” method has error rates that are lower than those of the eigenface technique for tests on the Harvard and Yale face databases
Article
Full-text available
Down syndrome is the most common cause of cognitive impairment and presents clinically with universally recognizable signs and symptoms. In this study, we focus on exam findings and digital facial analysis technology in individuals with Down syndrome in diverse populations. Photos and clinical information were collected on 65 individuals from 13 countries, 56.9% were male and the average age was 6.6 years (range 1 month to 26 years; SD = 6.6 years). Subjective findings showed that clinical features were different across ethnicities (Africans, Asians, and Latin Americans), including brachycephaly, ear anomalies, clinodactyly, sandal gap, and abundant neck skin, which were all significantly less frequent in Africans (P < 0.001, P < 0.001, P < 0.001, P < 0.05, and P < 0.05, respectively). Evaluation using a digital facial analysis technology of a larger diverse cohort of newborns to adults (n = 129 cases; n = 132 controls) was able to diagnose Down syndrome with a sensitivity of 0.961, specificity of 0.924, and accuracy of 0.943. Only the angles at medial canthus and ala of the nose were common significant findings amongst different ethnicities (Caucasians, Africans, and Asians) when compared to ethnically matched controls. The Asian group had the least number of significant digital facial biometrics at 4, compared to Caucasians at 8 and Africans at 7. In conclusion, this study displays the wide variety of findings across different geographic populations in Down syndrome and demonstrates the accuracy and promise of digital facial analysis technology in the diagnosis of Down syndrome internationally.
Article
Full-text available
TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms. A computation expressed using TensorFlow can be executed with little or no change on a wide variety of heterogeneous systems, ranging from mobile devices such as phones and tablets up to large-scale distributed systems of hundreds of machines and thousands of computational devices such as GPU cards. The system is flexible and can be used to express a wide variety of algorithms, including training and inference algorithms for deep neural network models, and it has been used for conducting research and for deploying machine learning systems into production across more than a dozen areas of computer science and other fields, including speech recognition, computer vision, robotics, information retrieval, natural language processing, geographic information extraction, and computational drug discovery. This paper describes the TensorFlow interface and an implementation of that interface that we have built at Google. The TensorFlow API and a reference implementation were released as an open-source package under the Apache 2.0 license in November, 2015 and are available at www.tensorflow.org.
Article
Full-text available
Facial analysis systems are becoming available to healthcare providers to aid in the recognition of dysmorphic phenotypes associated with a multitude of genetic syndromes. These technologies automatically detect facial points and extract various measurements from images to recognise dysmorphic features and evaluate similarities to known facial patterns (gestalts). To evaluate such systems' usefulness for supporting the clinical practice of healthcare professionals, the recognition accuracy of the Cornelia de Lange Syndrome (CdLS) phenotype was examined with FDNA's automated Facial Dysmorphology Novel Analysis (FDNA) technology. In the first experiment, 2D facial images of CdLS patients with either an NIPBL or SMC1A gene mutation as well as non-CdLS patients which were assessed by dysmorphologists in a previous study were evaluated by the FDNA technology; the average detection rate of experts was 77% while the system's detection rate was 87%. In the second study, when a new set of NIPBL, SMC1A and non-CdLS patient photos was evaluated, the detection rate increased to 94%. The results from both studies indicated that the system's detection rate was comparable to that of dysmorphology experts. Therefore, utilising such technologies may be a useful tool in a clinical setting.
Article
Full-text available
The analysis of facial appearance is significant to an early diagnosis of medical genetic diseases. The fast development of image processing and machine learning techniques facilitates the detection of facial dysmorphic features. This paper is a survey of the recent studies developed for the screening of genetic abnormalities across the facial features obtained from two dimensional and three dimensional images.
Article
Full-text available
Down syndrome is the most common chromosomal condition that presents characteristic facial morphology and texture patterns. The early detection of Down syndrome through an automatic, non-invasive and simple way is desirable and critical to provide the best health management to newborns. In this study, we propose such a computer-aided diagnosis system for Down syndrome from photography based on facial analysis with ensemble learning. First, geometric and texture facial features are extracted based on automatically located facial landmarks, followed by feature fusion and selection. Then multiple classifiers (i.e. support vector machines, random forests and linear discriminant analysis) are adopted to identify patients with Down syndrome. An accurate and reliable decision is finally achieved by optimally combining the outputs of these individual classifiers via ensemble learning that captures both the shared and complementary information from different classifiers. The best performance was achieved by using the median ensemble rule with 0.967 accuracy, 0.977 precision and 0.933 recall.
Article
Full-text available
Pushing by big data and deep convolutional neural network (CNN), the performance of face recognition is becoming comparable to human. Using private large scale training datasets, several groups achieve very high performance on LFW, i.e., 97% to 99%. While there are many open source implementations of CNN, none of large scale face dataset is publicly available. The current situation in the field of face recognition is that data is more important than algorithm. To solve this problem, this paper proposes a semi-automatical way to collect face images from Internet and builds a large scale dataset containing about 10,000 subjects and 500,000 images, called CASIAWebFace. Based on the database, we use a 11-layer CNN to learn discriminative representation and obtain state-of-theart accuracy on LFW and YTF. The publication of CASIAWebFace will attract more research groups entering this field and accelerate the development of face recognition in the wild.
Article
Full-text available
Many deep neural networks trained on natural images exhibit a curious phenomenon in common: on the first layer they learn features similar to Gabor filters and color blobs. Such first-layer features appear not to be specific to a particular dataset or task, but general in that they are applicable to many datasets and tasks. Features must eventually transition from general to specific by the last layer of the network, but this transition has not been studied extensively. In this paper we experimentally quantify the generality versus specificity of neurons in each layer of a deep convolutional neural network and report a few surprising results. Transferability is negatively affected by two distinct issues: (1) the specialization of higher layer neurons to their original task at the expense of performance on the target task, which was expected, and (2) optimization difficulties related to splitting networks between co-adapted neurons, which was not expected. In an example network trained on ImageNet, we demonstrate that either of these two issues may dominate, depending on whether features are transferred from the bottom, middle, or top of the network. We also document that the transferability of features decreases as the distance between the base task and target task increases, but that transferring features even from distant tasks can be better than using random features. A final surprising result is that initializing a network with transferred features from almost any number of layers can produce a boost to generalization that lingers even after fine-tuning to the target dataset.
Article
Full-text available
The ImageNet Large Scale Visual Recognition Challenge is a benchmark in object category classification and detection on hundreds of object categories and millions of images. The challenge has been run annually from 2010 to present, attracting participation from more than fifty institutions. This paper describes the creation of this benchmark dataset and the advances in object recognition that have been possible as a result. We discuss the challenges of collecting large-scale ground truth annotation, highlight key breakthroughs in categorical object recognition, provide detailed a analysis of the current state of the field of large-scale image classification and object detection, and compare the state-of-the-art computer vision accuracy with human accuracy. We conclude with lessons learned in the five years of the challenge, and propose future directions and improvements.
Article
Full-text available
Down syndrome, the most common single cause of human birth defects, produces alterations in physical growth and mental retardation. If missed before birth, the early detection of Down syndrome is crucial for the management of patients and disease. However, the diagnostic accuracy for pediatricians prior to cytogenetic results is moderate and the access to specialists is limited in many social and low-economic areas. In this study, we propose a simple, non-invasive and automated framework for Down syndrome detection based on disease-specific facial patterns. Geometric and local texture features are extracted based on automatically detected anatomical landmarks to describe facial morphology and structure. To accurately locate the anatomical facial landmarks, a hierarchical constrained local model using independent component analysis (ICA) is proposed. We also introduce a data-driven ordering method for selecting dominant independent components in ICA. The hierarchical structure of the model increases the accuracy of landmark detection by fitting separate models to different groups. Then the most representative features are selected and we also demonstrate that they match clinical observations. Finally, a variety of classifiers are evaluated to discriminate between Down syndrome and healthy populations. The best performance achieved 0.967 accuracy and 0.956 F1 score using combined features and linear discriminant analysis. The method was also validated on a dataset with mixed genetic syndromes and high performance (0.970 accuracy and 0.930 F1 score) was also obtained. The promising results indicate that our method could assist in Down syndrome screening effectively in a simple, non-invasive way, and extensible to detection of other genetic syndromes.
Conference Paper
Full-text available
. We present a new framework for interpreting face images and image sequences using an Active Appearance Model (AAM). The AAM contains a statistical, photo-realistic model of the shape and grey-level appearance of faces. This paper demonstrates the use of the AAM's efficient iterative matching scheme for image interpretation. We use the AAM as a basis for face recognition, obtain good results for difficult images. We show how the AAM framework allows identity information to be decoupled from other variation, allowing evidence of identity to be integrated over a sequence. The AAM approach makes optimal use of the evidence from either a single image or image sequence. Since we derive a complete description of a given image our method can be used as the basis for a range of face image interpretation tasks. 1 Introduction There is currently a great deal of interest in model-based approaches to the interpretation of images [17] [9] [15] [14][8]. The attractions are two-fold: robust interpr...
Article
Full-text available
Down syndrome is a chromosomal condition caused by the presence of all or part of an extra 21st chromosome. It has different facial symptoms. These symptoms contain distinctive information for face recognition. In this study, a novel method is developed to distinguish Down Syndrome in a custom face database. Gabor Wavelet Transform (GWT) is used as a feature extraction method. Dimension reduction is performed with Principal Component Analysis (PCA). New dimension which has most valuable information is derived with Linear Discriminant Analysis (LDA). Classification process is implemented with k-nearest neighbor (kNN) and Support Vector Machine (SVM) methods. The classification accuracy is carried out 96% and 97,34% with kNN and SVM methods, respectively. Different from the studies related with the Down Sydrome, feature selection process is applied before PCA according to the correlation between components of feature vectors. Best results are achieved with euclidean distance metric for kNN and linear kernel type for SVM. In this way, we developed an efficient system to recognize Down syndrome.
Article
Full-text available
Cornelia de Lange syndrome (CdLS) is a dominant disorder with classic severe forms and milder atypical variants. Central to making the diagnosis is identification of diagnostic facial features. With the recognition that patients with SMC1A and SMC3 mutations have milder, atypical features, we surveyed 65 dysmorphologists using facial photographs from 32 CdLS patients with the goals of (1) Illustrating examples of milder patients with SMC1A mutations and (2) Obtaining objective data to determine which facial features were useful and misleading in making a diagnosis of CdLS. Clinicians were surveyed whether the patient had CdLS or another diagnosis, the certainty of response and the clinical features used to support each response. Using only facial photographs, an average of 24 cases (75%) were accurately diagnosed per clinician. Correct diagnoses were made in 90% of classic CdLS and 87% of non-CdLS cases, however, only 54% of mild or variant CdLS were correctly diagnosed by respondents. We confirmed that CdLS is most accurately diagnosed in childhood and the diagnosis becomes increasingly difficult with age. This survey demonstrated that emphasis is placed on the eyebrows, nasal features, prominent upper lip and micrognathia. In addition, the presence of fuller, atypical eyebrows, a prominent nasal bridge and significant prognathism with age dissuaded survey takers from arriving at a diagnosis of CdLS in individuals with mild NIPBL and SMC1A mutations. This work underscores the difficulty in diagnosing patients with mild and variant CdLS and serves to objectively classify both useful and misleading features in the diagnosis of CdLS.
Article
Full-text available
Genetic syndromes often involve craniofacial malformations. We have investigated whether a computer can recognize disease-specific facial patterns in unrelated individuals. For this, 55 photographs (256 x 256 pixel) of patients with mucopolysaccharidosis type III (n=6), Cornelia de Lange (n=12), fragile X (n=12), Prader-Willi (n=12), and Williams-Beuren (n=13) syndromes were preprocessed by a Gabor wavelet transformation. By comparing the feature vectors at 32 facial nodes, 42/55 (76%) of the patients were correctly classified. In another four patients (7%), the correct and an incorrect diagnosis scored equally well. Clinical geneticists who were shown the same photographs achieved a recognition rate of 62%. Our results prove that certain syndromes are associated with a specific facial pattern and that this pattern can be described in mathematical terms.
Article
Full-text available
Clinical evaluation of children with developmental delay continues to present a challenge to the clinicians. In many cases, the face provides important information to diagnose a condition. However, database support with respect to facial traits is limited at present. Computer-based analyses of 2D and 3D representations of faces have been developed, but it is unclear how well a larger number of conditions can be handled by such systems. We have therefore analysed 2D pictures of patients each being affected with one of 10 syndromes (fragile X syndrome; Cornelia de Lange syndrome; Williams-Beuren syndrome; Prader-Willi syndrome; Mucopolysaccharidosis type III; Cri-du-chat syndrome; Smith-Lemli-Opitz syndrome; Sotos syndrome; Microdeletion 22q11.2; Noonan syndrome). We can show that a classification accuracy of >75% can be achieved for a computer-based diagnosis among the 10 syndromes, which is about the same accuracy achieved for five syndromes in a previous study. Pairwise discrimination of syndromes ranges from 80 to 99%. Furthermore, we can demonstrate that the criteria used by the computer decisions match clinical observations in many cases. These findings indicate that computer-based picture analysis might be a helpful addition to existing database systems, which are meant to assist in syndrome diagnosis, especially as data acquisition is straightforward and involves off-the-shelf digital camera equipment.
Conference Paper
Rectified activation units (rectifiers) are essential for state-of-the-art neural networks. In this work, we study rectifier neural networks for image classification from two aspects. First, we propose a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit. PReLU improves model fitting with nearly zero extra computational cost and little overfitting risk. Second, we derive a robust initialization method that particularly considers the rectifier nonlinearities. This method enables us to train extremely deep rectified models directly from scratch and to investigate deeper or wider network architectures. Based on our PReLU networks (PReLU-nets), we achieve 4.94% top-5 test error on the ImageNet 2012 classification dataset. This is a 26% relative improvement over the ILSVRC 2014 winner (GoogLeNet, 6.66%). To our knowledge, our result is the first to surpass human-level performance (5.1%, Russakovsky et al.) on this visual recognition challenge.
Conference Paper
We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 dif- ferent classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implemen- tation of the convolution operation. To reduce overfitting in the fully-connected layers we employed a recently-developed regularization method called dropout that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry
Article
X-linked hypohidrotic ectodermal dysplasia (XLHED) is a genetic disorder that affects ectodermal structures and presents with a characteristic facial appearance. The ability of automated facial recognition technology to detect the phenotype from images was assessed . In Phase 1 of this study we examined if the age of male patients affected the technology's recognition. In Phase 2 we investigated how well the technology discriminated affected males cases from female carriers and from individuals with other ectodermal dysplasia syndromes. The system detected XLHED to be the most likely diagnosis in all genetically confirmed affected male patients of all ages, and in 55% of heterozygous females. Interestingly, patients with other ED syndromes were also detected by the XLHED-targeted analysis, consistent with shared developmental features. Thus the automated facial recognition system represents a promising non-invasive technology to screen patients at all ages for a possible diagnosis of ectodermal dysplasia, with greatest sensitivity and specificity for males affected with XLHED.
Article
High throughput approaches are continuously progressing and have become a major part of clinical diagnostics. Still, the critical process of detailed phenotyping and gathering clinical information has not changed much in the last decades. Forms of Next Generation Phenotyping (NGP) are needed to increase further the value of any kind of genetic approaches, including timely considering of (molecular) cytogenetics during the diagnostic quest. As NGP we used in this study the Facial Dysmorphology Novel Analysis (FDNA) technology to automatically identify facial phenotypes associated with Emanuel (ES) and Pallister-Killian Syndrome (PKS) from 2D facial photos. The comparison between ES or PKS and normal individuals expressed a full separation between the cohorts. Our results show that NPG is able to help in the clinic, and could reduce the time patients spend in diagnostic odyssey. It also helps to differentiate ES or PKS from each other and other patients with small supernumerary marker chromosomes, especially in countries with no access to more sophisticated genetic approaches apart from banding cytogenetics. Inclusion of more facial pictures of patient with sSMC, like isochromosome-18p-, cat-eye-syndrome or others may contribute to higher detection rates in future.
Article
This paper presents a method for extracting distinctive invariant features from images that can be used to perform reliable matching between different views of an object or scene. The features are invariant to image scale and rotation, and are shown to provide robust matching across a substantial range of affine distortion, change in 3D viewpoint, addition of noise, and change in illumination. The features are highly distinctive, in the sense that a single feature can be correctly matched with high probability against a large database of features from many images. This paper also describes an approach to using these features for object recognition. The recognition proceeds by matching individual features to a database of features from known objects using a fast nearest-neighbor algorithm, followed by a Hough transform to identify clusters belonging to a single object, and finally performing verification through least-squares solution for consistent pose parameters. This approach to recognition can robustly identify objects among clutter and occlusion while achieving near real-time performance.
Article
DS) is the most common microdeletion syndrome and is underdiagnosed in diverse populations. This syndrome has a variable phenotype and affects multiple systems, making early recognition imperative. In this study, individuals from diverse populations with 22q11.2 DS were evaluated clinically and by facial analysis technology. Clinical information from 106 individuals and images from 101 were collected from individuals with 22q11.2 DS from 11 countries; average age was 11.7 and 47% were male. Individuals were grouped into categories of African descent (African), Asian, and Latin American. We found that the phenotype of 22q11.2 DS varied across population groups. Only two findings, congenital heart disease and learning problems, were found in greater than 50% of participants. When comparing the clinical features of 22q11.2 DS in each population, the proportion of individuals within each clinical category was statistically different except for learning problems and ear anomalies (P < 0.05). However, when Africans were removed from analysis, six additional clinical features were found to be independent of ethnicity (P ≥ 0.05). Using facial analysis technology, we compared 156 Caucasians, Africans, Asians, and Latin American individuals with 22q11.2 DS with 156 age and gender matched controls and found that sensitivity and specificity were greater than 96% for all populations. In summary, we present the varied findings from global populations with 22q11.2 DS and demonstrate how facial analysis technology can assist clinicians in making accurate 22q11.2 DS diagnoses. This work will assist in earlier detection and in increasing recognition of 22q11.2 DS throughout the world.
Article
Relatively small data sets available for expression recognition research make the training of deep networks for expression recognition very challenging. Although fine-tuning can partially alleviate the issue, the performance is still below acceptable levels as the deep features probably contain redun- dant information from the pre-trained domain. In this paper, we present FaceNet2ExpNet, a novel idea to train an expression recognition network based on static images. We first propose a new distribution function to model the high-level neurons of the expression network. Based on this, a two-stage training algorithm is carefully designed. In the pre-training stage, we train the convolutional layers of the expression net, regularized by the face net; In the refining stage, we append fully- connected layers to the pre-trained convolutional layers and train the whole network jointly. Visualization shows that the model trained with our method captures improved high-level expression semantics. Evaluations on four public expression databases, CK+, Oulu-CASIA, TFD, and SFEW demonstrate that our method achieves better results than state-of-the-art.
Article
The genetic basis of numerous intellectual disability (ID) syndromes has recently been identified by applying exome analysis on a research or clinical basis. There is significant clinical overlap of biologically related syndromes, as exemplified by Nicolaides-Baraitser (NCBRS) and Coffin-Siris (CSS) syndrome. Both result from mutations affecting the BAF (mSWI/SNF) complex and belong to the growing category of BAFopathies. In addition to the notable clinical overlap between these BAFopathies, heterogeneity exists for patients clinically diagnosed with one of these conditions. We report two teenagers with ID whose molecular diagnosis of a SMARC2A or ARID1B mutation, respectively, was established through clinical exome analysis. Interestingly, using only the information provided in a single clinically obtained facial photograph from each patient, the facial dysmorphology analysis detected similarities to facial patterns associated with NCBRS as the first suggestion for both individuals, followed by CSS as the second highest ranked in the individual with the ARID1B mutation. Had this information been available to the laboratory performing the exome analysis, it could have been utilized during the variant analysis and reporting process, in conjunction with the written summary provided with each test requisition. While the available massive parallel sequencing technology, variant calling and variant interpretation are constantly evolving, clinical information remains critical for this diagnostic process. When trio analysis is not feasible, additional diagnostic tools may become particularly valuable. Facial dysmorphology analysis data may supplement the clinical phenotype summary and provide data independent of the clinician's personal experience and bias. © 2016 Wiley Periodicals, Inc.
Article
Purpose: Noonan syndrome (NS) is an autosomal-dominant disorder characterized by craniofacial dysmorphism, growth retardation, cardiac abnormalities, and learning difficulties. It belongs to the RASopathies, which are caused by germ-line mutations in genes encoding components of the RAS mitogen-activated protein kinase (MAPK) pathway. RIT1 was recently reported as a disease gene for NS, but the number of published cases is still limited. Methods: We sequenced RIT1 in 310 mutation-negative individuals with a suspected RASopathy and prospectively in individuals who underwent genetic testing for NS. Using a standardized form, we recorded clinical features of all RIT1 mutation-positive patients. Clinical and genotype data from 36 individuals with RIT1 mutation reported previously were reviewed. Results: Eleven different RIT1 missense mutations, three of which were novel, were identified in 33 subjects from 28 families; codons 57, 82, and 95 represent mutation hotspots. In relation to NS of other genetic etiologies, prenatal abnormalities, cardiovascular disease, and lymphatic abnormalities were common in individuals with RIT1 mutation, whereas short stature, intellectual problems, pectus anomalies, and ectodermal findings were less frequent. Conclusion: RIT1 is one of the major genes for NS. The RIT1-associated phenotype differs gradually from other NS subtypes, with a high prevalence of cardiovascular manifestations, especially hypertrophic cardiomyopathy, and lymphatic problems.Genet Med advance online publication 21 April 2016Genetics in Medicine (2016); doi:10.1038/gim.2016.32.
Article
Noonan syndrome is a heterogeneous autosomal dominant disorder caused by mutations in at least eight genes involved in the RAS/MAPK signaling pathway. Recently, RIT1 (Ras-like without CAAX 1) has been shown to be involved in the pathogenesis of some patients. We report a series of 44 patients from 30 pedigrees (including nine multiplex families) with mutations in RIT1. These patients display a typical Noonan gestalt and facial phenotype. Among the probands, 8.7% showed postnatal growth retardation, 90% had congenital heart defects, 36% had hypertrophic cardiomyopathy (a lower incidence compared with previous report), 50% displayed speech delay and 52% had learning difficulties, but only 22% required special education. None had major skin anomalies. One child died perinatally of juvenile myelomonocytic leukemia. Compared with the canonical Noonan phenotype linked to PTPN11 mutations, patients with RIT1 mutations appear to be less severely growth retarded and more frequently affected by cardiomyopathy. Based on our experience, we estimate that RIT1 could be the cause of 5% of Noonan syndrome patients. Because mutations found constitutionally in Noonan syndrome are also found in several tumors in adulthood, we evaluated the potential contribution of RIT1 to leukemogenesis in Noonan syndrome. We screened 192 pediatric cases of acute lymphoblastic leukemias (96 B-ALL and 96 T-ALL) and 110 cases of juvenile myelomonocytic leukemias (JMML), but detected no variation in these tumoral samples, suggesting that Noonan patients with germline RIT1 mutations are not at high risk to developing JMML or ALL, and that RIT1 has at most a marginal role in these sporadic malignancies.European Journal of Human Genetics advance online publication, 13 January 2016; doi:10.1038/ejhg.2015.273.
Article
Unsupervised joint alignment of images has been demonstrated to improve performance on recognition tasks such as face verification. Such alignment reduces undesired variability due to factors such as pose, while only requiring weak supervision in the form of poorly aligned examples. However, prior work on unsu-pervised alignment of complex, real-world images has required the careful selection of feature representation based on hand-crafted image descriptors, in order to achieve an appropriate, smooth optimization landscape. In this paper, we instead propose a novel combination of unsupervised joint alignment with unsupervised feature learning. Specifically, we incorporate deep learning into the congealing alignment framework. Through deep learning, we obtain features that can represent the image at differing resolutions based on network depth, and that are tuned to the statistics of the specific data being aligned. In addition, we modify the learning algorithm for the restricted Boltzmann machine by incorporating a group sparsity penalty, leading to a topographic organization of the learned filters and improving subsequent alignment results. We apply our method to the Labeled Faces in the Wild database (LFW). Using the aligned images produced by our proposed unsupervised algorithm, we achieve higher accuracy in face verification compared to prior work in both unsupervised and supervised alignment. We also match the accuracy for the best available commercial method.
Article
Face Recognition has been studied for many decades. As opposed to traditional hand-crafted features such as LBP and HOG, much more sophisticated features can be learned automatically by deep learning methods in a data-driven way. In this paper, we propose a two-stage approach that combines a multi-patch deep CNN and deep metric learning, which extracts low dimensional but very discriminative features for face verification and recognition. Experiments show that this method outperforms other state-of-the-art methods on LFW dataset, achieving 99.85% pair-wise verification accuracy and significantly better accuracy under other two more practical protocols. This paper also discusses the importance of data size and the number of patches, showing a clear path to practical high-performance face recognition systems in real world
Article
We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions. The method is straightforward to implement and is based an adaptive estimates of lower-order moments of the gradients. The method is computationally efficient, has little memory requirements and is well suited for problems that are large in terms of data and/or parameters. The method is also ap- propriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The method exhibits invariance to diagonal rescaling of the gradients by adapting to the geometry of the objective function. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. We demonstrate that Adam works well in practice when experimentally compared to other stochastic optimization methods.
Article
Background: In general, medical geneticists aim to pre-diagnose underlying syndromes based on facial features before performing cytological or molecular analyses where a genotype-phenotype interrelation is possible. However, determining correct genotype-phenotype interrelationships among many syndromes is tedious and labor-intensive, especially for extremely rare syndromes. Thus, a computer-aided system for pre-diagnosis can facilitate effective and efficient decision support, particularly when few similar cases are available, or in remote rural districts where diagnostic knowledge of syndromes is not readily available. Methods: The proposed methodology, visual diagnostic decision support system (visual diagnostic DSS), employs machine learning (ML) algorithms and digital image processing techniques in a hybrid approach for automated diagnosis in medical genetics. This approach uses facial features in reference images of disorders to identify visual genotype-phenotype interrelationships. Our statistical method describes facial image data as principal component features and diagnoses syndromes using these features. Results: The proposed system was trained using a real dataset of previously published face images of subjects with syndromes, which provided accurate diagnostic information. The method was tested using a leave-one-out cross-validation scheme with 15 different syndromes, each of comprised 5-9 cases, i.e., 92 cases in total. An accuracy rate of 83% was achieved using this automated diagnosis technique, which was statistically significant (p<0.01). Furthermore, the sensitivity and specificity values were 0.857 and 0.870, respectively. Conclusion: Our results show that the accurate classification of syndromes is feasible using ML techniques. Thus, a large number of syndromes with characteristic facial anomaly patterns could be diagnosed with similar diagnostic DSSs to that described in the present study, i.e., visual diagnostic DSS, thereby demonstrating the benefits of using hybrid image processing and ML-based computer-aided diagnostics for identifying facial phenotypes.
Conference Paper
In modern face recognition, the conventional pipeline consists of four stages: detect => align => represent => classify. We revisit both the alignment step and the representation step by employing explicit 3D face modeling in order to apply a piecewise affine transformation, and derive a face representation from a nine-layer deep neural network. This deep network involves more than 120 million parameters using several locally connected layers without weight sharing, rather than the standard convolutional layers. Thus we trained it on the largest facial dataset to-date, an identity labeled dataset of four million facial images belonging to more than 4,000 identities, where each identity has an average of over a thousand samples. The learned representations coupling the accurate model-based alignment with the large facial database generalize remarkably well to faces in unconstrained environments, even with a simple classifier. Our method reaches an accuracy of 97.25% on the Labeled Faces in the Wild (LFW) dataset, reducing the error of the current state of the art by more than 25%, closely approaching human-level performance.
Conference Paper
We present a new dataset with the goal of advancing the state-of-the-art in object recognition by placing the question of object recognition in the context of the broader question of scene understanding. This is achieved by gathering images of complex everyday scenes containing common objects in their natural context. Objects are labeled using per-instance segmentations to aid in understanding an object's precise 2D location. Our dataset contains photos of 91 objects types that would be easily recognizable by a 4 year old along with per-instance segmentation masks. With a total of 2.5 million labeled instances in 328k images, the creation of our dataset drew upon extensive crowd worker involvement via novel user interfaces for category detection, instance spotting and instance segmentation. We present a detailed statistical analysis of the dataset in comparison to PASCAL, ImageNet, and SUN. Finally, we provide baseline performance analysis for bounding box and segmentation detection results using a Deformable Parts Model.
Conference Paper
We present an approach to the detection of parts of highly deformable objects, such as the human body. Instead of using kinematic constraints on relative angles used by most existing approaches for modeling part-to-part relations, we learn and use special observed 'linking' features that support particular pairwise part configurations. In addition to modeling the appearance of individual parts, the current approach adds modeling of the appearance of part-linking, which is shown to provide useful information. For example, configurations of the lower and upper arms are supported by observing corresponding appearances of the elbow or other relevant features. The proposed model combines the support from all the linking features observed in a test image to infer the most likely joint configuration of all the parts of interest. The approach is trained using images with annotated parts, but no a-priori known part connections or connection parameters are assumed, and the linking features are discovered automatically during training. We evaluate the performance of the proposed approach on two challenging human body parts detection datasets, and obtain performance comparable, and in some cases superior, to the state-of-the-art. In addition, the approach generality is shown by applying it without modification to part detection on datasets of animal parts and of facial fiducial points.
We present a unified model for face detection, pose estimation, and landmark estimation in real-world, cluttered images. Our model is based on a mixtures of trees with a shared pool of parts; we model every facial landmark as a part and use global mixtures to capture topological changes due to viewpoint. We show that tree-structured models are surprisingly effective at capturing global elastic deformation, while being easy to optimize unlike dense graph structures. We present extensive results on standard face benchmarks, as well as a new “in the wild” annotated dataset, that suggests our system advances the state-of-the-art, sometimes considerably, for all three tasks. Though our model is modestly trained with hundreds of faces, it compares favorably to commercial systems trained with billions of examples (such as Google Picasa and face.com).
Article
We investigate the problem of automatically labelling appearances of characters in TV or film material with their names. This is tremendously challenging due to the huge variation in imaged appearance of each character and the weakness and ambiguity of available annotation. However, we demonstrate that high precision can be achieved by combining multiple sources of information, both visual and textual. The principal novelties that we introduce are: (i) automatic generation of time stamped character annotation by aligning subtitles and transcripts; (ii) strengthening the supervisory information by identifying when characters are speaking. In addition, we incorporate complementary cues of face matching and clothing matching to propose common annotations for face tracks, and consider choices of classifier which can potentially correct errors made in the automatic extraction of training data from the weak textual annotation. Results are presented on episodes of the TV series ‘‘Buffy the Vampire Slayer”.
Article
Thesupport-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data. High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
Article
Computer systems play an important role in clinical genetics and are a routine part of finding clinical diagnoses but make it difficult to fully exploit information derived from facial appearance. So far, automated syndrome diagnosis based on digital, facial photographs has been demonstrated under study conditions but has not been applied in clinical practice. We have therefore investigated how well statistical classifiers trained on study data comprising 202 individuals affected by one of 14 syndromes could classify a set of 91 patients for whom pictures were taken under regular, less controlled conditions in clinical practice. We found a classification accuracy of 21% percent in the clinical sample representing a ratio of 3.0 over a random choice. This contrasts with a 60% accuracy or 8.5 ratio in the training data. Producing average images in both groups from sets of pictures for each syndrome demonstrates that the groups exhibit large phenotypic differences explaining discrepancies in accuracy. A broadening of the data set is suggested in order to improve accuracy in clinical practice. In order to further this goal, a software package is made available that allows application of the procedures and contributions toward an improved data set.
Article
The facial photographs of 81 individuals with Noonan syndrome, from infancy to adulthood, have been evaluated by two dysmorphologists (JA and MZ), each of whom has considerable experience with disorders of the Ras/MAPK pathway. Thirty-two of this cohort have PTPN11 mutations, 21 SOS1 mutations, 11 RAF1 mutations, and 17 KRAS mutations. The facial appearance of each person was judged to be typical of Noonan syndrome or atypical. In each gene category both typical and unusual faces were found. We determined that some individuals with mutations in the most commonly affected gene, PTPN11, which is correlated with the cardinal physical features, may have a quite atypical face. Conversely, some individuals with KRAS mutations, which may be associated with a less characteristic intellectual phenotype and a resemblance to Costello and cardio-facio-cutaneous syndromes, can have a very typical face. Thus, the facial phenotype, alone, is insufficient to predict the genotype, but certain facial features may facilitate an educated guess in some cases.
Article
The objective of the study was to overview the role of genetic research in fostering translational studies of craniofacial diseases of dental interest. Background information is presented to illustrate influences affecting genetic research studies of Mendelian diseases. Genetic studies of amelogenesis imperfecta, dentinogenesis imperfecta, hereditary gingival fibromatosis and Papillon Lefèvre syndrome are reviewed. Findings are presented to illustrate how translational applications of clinical and basic research may improve clinical care. Clinical and basic science research has identified specific genes and mutations etiologically responsible for amelogenesis imperfecta, dentinogenesis imperfecta, hereditary gingival fibromatosis and Papillon Lefèvre syndrome. These findings are enabling researchers to understand how specific genetic alterations perturb normal growth and development of dental tissues. Identification of the genetic basis of these conditions is enabling clinicians and researchers to more fully understand the etiology and clinical consequences of these diseases of dental importance. Findings from genetic studies of dental diseases provide a basis for diagnostic genetic testing and development of therapeutic intervention strategies directed at the underlying disease etiology. These studies are advancing our understanding of the development of dental tissues in health and disease. The dental community must consider how to incorporate these developments into effective disease prevention paradigms to facilitate the diagnosis and treatment of individuals with genetic diseases.
Article
The data base of an ongoing population-based registry with multiple sources of ascertainment was used to estimate the present population load from genetic disease in more than 1 million consecutive live births. It was found that, before approximately age 25 years, greater than or equal to 53/1,000 live-born individuals can be expected to have diseases with an important genetic component. This total was composed of single-gene disorders (3.6/1,000), consisting of autosomal dominant (1.4/1,000), autosomal recessive (1.7/1,000), and X-linked recessive disorders (0.5/1,000). Chromosomal anomalies accounted for 1.8/1,000, multifactorial disorders (including those present at birth and those of onset before age 25 years) accounted for 46.4/1,000, and cases of genetic etiology in which the precise mechanism was not identified accounted for 1.2/1,000. Previous studies have usually considered all congenital anomalies (ICD 740-759) as part of the genetic load, but only those judged to fit into one of the above categories were included in the present study. Data for congenital anomalies are therefore also presented separately, to facilitate comparison with earlier studies. If all congenital anomalies are considered as part of the genetic load, then greater than or equal to 79/1,000 live-born individuals have been identified as having one or other genetic disorder before approximately age 25 years. These new data represent a better estimate of the genetic load in the population than do previous studies.
Article
This paper presents a novel and efficient facial image representation based on local binary pattern (LBP) texture features. The face image is divided into several regions from which the LBP feature distributions are extracted and concatenated into an enhanced feature vector to be used as a face descriptor. The performance of the proposed method is assessed in the face recognition problem under different challenges. Other applications and several extensions are also discussed.