Mark Hoogendoorn

Mark Hoogendoorn
Vrije Universiteit Amsterdam | VU · Department of Computer Science

About

219
Publications
28,688
Reads
How we measure 'reads'
A 'read' is counted each time someone views a publication summary (such as the title, abstract, and list of authors), clicks on a figure, or views or downloads the full-text. Learn more
2,196
Citations
Introduction

Publications

Publications (219)
Conference Paper
Full-text available
Reinforcement learning (RL) is a powerful framework for learning complex behaviors, but lacks adoption in many settings due to sample size requirements. We introduce a framework for increasing sample efficiency of RL algorithms. Our approach focuses on optimizing environment rewards with high-level instructions. These are modeled as a high-level co...
Article
Full-text available
Deep reinforcement learning (DRL) has shown remarkable success in artificial domains and in some real-world applications. However, substantial challenges remain such as learning efficiently under safety constraints. Adherence to safety constraints is a hard requirement in many high-impact application domains such as healthcare and finance. These co...
Preprint
Full-text available
Adapting a Reinforcement Learning (RL) agent to an unseen environment is a difficult task due to typical over-fitting on the training environment. RL agents are often capable of solving environments very close to the trained environment, but when environments become substantially different, their performance quickly drops. When agents are retrained...
Chapter
Finding well-defined clusters in data represents a fundamental challenge for many data-driven applications, and largely depends on good data representation. Drawing on literature regarding representation learning, studies suggest that one key characteristic of good latent representations is the ability to produce semantically mixed outputs when dec...
Article
Context There are many e-Health mobile apps on the apps store, from apps to improve a user’s lifestyle to mental coaching. Whilst these apps might consider user context when they give their interventions, prompts, and encouragements, they still tend to be rigid e.g., not using user context and experience to tailor themselves to the user. Objective...
Article
Full-text available
Introduction: Determining the optimal timing for extubation can be challenging in the intensive care. In this study, we aim to identify predictors for extubation failure in critically ill patients with COVID-19. Methods: We used highly granular data from 3464 adult critically ill COVID patients in the multicenter Dutch Data Warehouse, including...
Chapter
Full-text available
Within software engineering, social sustainability is the dimension of sustainability that focuses on the “support of current and future generations to have the same or greater access to social resources by pursuing social equity.” An important domain that strives to achieve social sustainability is e-Health, and more recently e-Health mobile apps....
Article
Full-text available
OBJECTIVES:. As coronavirus disease 2019 is a novel disease, treatment strategies continue to be debated. This provides the intensive care community with a unique opportunity as the population of coronavirus disease 2019 patients requiring invasive mechanical ventilation is relatively homogeneous compared with other ICU populations. We hypothesize...
Article
Full-text available
Objectives: Unexpected ICU readmission is associated with longer length of stay and increased mortality. To prevent ICU readmission and death after ICU discharge, our team of intensivists and data scientists aimed to use AmsterdamUMCdb to develop an explainable machine learning–based real-time bedside decision support tool. Derivation Cohort: Da...
Article
Full-text available
Background The Coronavirus disease 2019 (COVID-19) pandemic has underlined the urgent need for reliable, multicenter, and full-admission intensive care data to advance our understanding of the course of the disease and investigate potential treatment strategies. In this study, we present the Dutch Data Warehouse (DDW), the first multicenter electro...
Article
Full-text available
Background The identification of risk factors for adverse outcomes and prolonged intensive care unit (ICU) stay in COVID-19 patients is essential for prognostication, determining treatment intensity, and resource allocation. Previous studies have determined risk factors on admission only, and included a limited number of predictors. Therefore, usin...
Preprint
Full-text available
While reinforcement learning (RL) has proven to be the approach of choice for tackling many complex problems, it remains challenging to develop and deploy RL agents in real-life scenarios successfully. This paper presents pH-RL (personalization in e-Health with RL) a general RL architecture for personalization to bring RL to health practice. pH-RL...
Article
Introduction In recent years, reinforcement learning (RL) has gained traction in the healthcare domain. In particular, RL methods have been explored for haemodynamic optimization of septic patients in the Intensive Care Unit. Most hospitals however, lack the data and expertise for model development, necessitating transfer of models developed using...
Preprint
Full-text available
Finding well-defined clusters in data represents a fundamental challenge for many data-driven applications, and largely depends on good data representation. Drawing on literature regarding representation learning, studies suggest that one key characteristic of good latent representations is the ability to produce semantically mixed outputs when dec...
Article
Full-text available
This study proposes a framework for mining temporal patterns from Electronic Medical Records. A new scoring scheme based on the Wilson interval is provided to obtain frequent and predictive patterns, as well as to accelerate the mining process by reducing the number of patterns mined. This is combined with a case study using data from general pract...
Article
Full-text available
The major application areas of reinforcement learning (RL) have traditionally been game playing and continuous control. In recent years, however, RL has been increasingly applied in systems that interact with humans. RL can personalize digital systems to make them more relevant to individual users. Challenges in personalization settings may be diff...
Article
Background Digital health interventions (DHIs) are poised to reduce target symptoms in a scalable, affordable, and empirically supported way. DHIs that involve coaching or clinical support often collect text data from 2 sources: (1) open correspondence between users and the trained practitioners supporting them through a messaging system and (2) te...
Article
Full-text available
A variety of effective psychotherapies for depression are available, but patients who suffer from depression vary in their treatment response. Combining face-to-face therapies with internet-based elements in the sense of blended treatment is a new approach to treatment for depression. The goal of this study was to answer the following research ques...
Article
Background Self-reported client assessments during online treatments enable the development of statistical models for the prediction of client improvement and symptom development. Evaluation of these models is mandatory to ensure their validity. Methods For this purpose, we suggest besides a model evaluation based on study data the use of a simulat...
Preprint
Trace clustering has increasingly been applied to find homogenous process executions. However, current techniques have difficulties in finding a meaningful and insightful clustering of patients on the basis of healthcare data. The resulting clusters are often not in line with those of medical experts, nor do the clusters guarantee to help return me...
Chapter
Semi-supervised learning can be applied to datasets that contain both labeled and unlabeled instances and can result in more accurate predictions compared to fully supervised or unsupervised learning in case limited labeled data is available. A subclass of problems, called Positive-Unlabeled (PU) learning, focuses on cases in which the labeled inst...
Chapter
In many sequential decision making problems progress is predominantly based on artificial data sets. This can be attributed to insufficient access to real data. Here we propose to mitigate this by using generative adversarial networks (GANs) to generate representative data sets from real data. Specifically, we investigate how GANs can generate trai...
Conference Paper
We introduce an end-to-end reinforcement learning (RL) solution for the problem of sending personalized digital health interventions. Previous work has shown that personalized interventions can be obtained through RL using simple, discrete state information such as the recent activity performed. In reality however, such features are often not obser...
Conference Paper
The huge wealth of data in the health domain can be exploited to create models that predict development of health states over time. Temporal learning algorithms are well suited to learn relationships between health states and make predictions about their future developments. However, these algorithms: (1) either focus on learning one generic model...
Conference Paper
Full-text available
Clustering of users underlies many of the personalisation algorithms that are in use nowadays. Such clustering is mostly performed in an offline fashion. For a health and wellbeing setting, offline clustering might however not be suitable, as limited data is often available and patient states can also quickly evolve over time. Existing online clust...
Conference Paper
Language systems have been of great interest to the research community and have recently reached the mass market through various assistant platforms on the web. Reinforcement Learning methods that optimize dialogue policies have seen successes in past years and have recently been extended into methods that personalize the dialogue, e.g. take the pe...
Chapter
Trace clustering has increasingly been applied to find homogenous process executions. However, current techniques have difficulties in finding a meaningful and insightful clustering of patients on the basis of healthcare data. The resulting clusters are often not in line with those of medical experts, nor do the clusters guarantee to help return me...
Preprint
Full-text available
The huge wealth of data in the health domain can be exploited to create models that predict development of health states over time. Temporal learning algorithms are well suited to learn relationships between health states and make predictions about their future developments. However, these algorithms: (1) either focus on learning one generic model...
Preprint
Grouping patients meaningfully can give insights about the different types of patients, their needs, and the priorities. Finding groups that are meaningful is however very challenging as background knowledge is often required to determine what a useful grouping is. In this paper we propose an approach that is able to find groups of patients based o...
Chapter
Research has shown that personalization of health interventions can contribute to an improved effectiveness. Reinforcement learning algorithms can be used to perform such tailoring. In this paper, we present a cluster-based reinforcement learning approach which learns optimal policies for groups of users. Such an approach can speed up the learning...
Chapter
Simulation environments have proven to be very useful as testbeds for reinforcement learning (RL) algorithms. For settings where an actual human user is involved, these simulation environments allow one to test out the suitability of new RL approaches without having to include real users at first. It obviously does require the simulator to have a c...
Chapter
Personalization of support in health and wellbeing settings is challenging. While personalization has shown to be highly beneficial to maximize the success of interventions, often only very limited experiences are available to personalize support strategies. Because of its focus on finding suitable actions/interventions that lead to long term rewar...
Article
Full-text available
Background: Different treatment alternatives exist for psychological disorders. Both clinical and cost effectiveness of treatment are crucial aspects for policy makers, therapists, and patients and thus play major roles for healthcare decision-making. At the start of an intervention, it is often not clear which specific individuals benefit most fro...
Article
Full-text available
Research has shown that personalization of health interventions can contribute to an improved effectiveness. Reinforcement learning algorithms can be used to perform such tailoring using data that is collected about users. Learning is however very fragile for health interventions as only limited time is available to learn from the user before disen...
Article
Recent developments in mobile technology, sensor devices, and artificial intelligence have created new opportunities for mental health care research. Enabled by large datasets collected in e-mental health research and practice, clinical researchers and members of the data mining community increasingly join forces to build predictive models for heal...
Chapter
In this chapter the theoretical underpinning of supervised learning are discussed. The whole supervised machine learning process is explained from a more formal perspective as well as some underlying theories. The theories discussed include concepts such as PAC learnability and VC dimensions. The implications of these theories are discussed.
Chapter
In this chapter, approaches to remove noise, that is inherently present in sensory data, are introduced. This includes outlier detection algorithms, missing value imputation, as well as approaches to filter more subtle noise in the data including the low pass filter and principal component analysis. The Kalman filter is also explained to remove noi...
Chapter
This chapter discusses reinforcement learning, a technique that can be used to learn when to provide what kind of feedback or intervention to a user to better accomplish the set goals. The techniques discussed are SARSA and Q-learning. In addition, approaches to allow reinforcement learning to cope with detailed sensor information such as discretiz...
Chapter
Supervised learning approaches that do not explicitly take the time component into account are briefly discussed in this chapter. The approaches explained include feedforward neural networks, support vector machines, k-nearest neighbor, decision trees, naïve bayes and ensembles. Guidelines are provided on how to apply these algorithms to quantified...
Chapter
This chapter focuses on clustering of the data resulting from quantified selves. It introduces distance functions that can be used to compare individual data points, but also entire datasets of users. Among these are dynamic time warping and the cross-correlation coefficient. The chapter provides a brief discussion of popular clustering techniques....
Chapter
Approaches to automatically generate useful features from sensory data are introduced in this chapter. Most of the approaches introduced focus on datasets that have a temporal ordering. Features in the time domain are explained, thereby summarizing both numerical and categorical values in a certain historical window. The frequency domain is also di...
Chapter
This chapter focuses on supervised learning approaches that do take time into account explicitly. Times series approaches are explained as well as recurrent neural networks (including echo state networks). In addition, parameter optimization techniques that can be used to fine-tune more knowledge driven predictive temporal models (dynamical systems...
Chapter
A typical quantified self/sensory dataset is introduced in this chapter. Popular sensors are briefly highlighted and a procedure is provided to transform raw sensory datasets into a suitable format to enable the application of machine learning techniques. The dataset introduced in this chapter is used to illustrate the approaches explained in the r...
Book
Self tracking has become part of a modern lifestyle; wearables and smartphones support self tracking in an easy fashion and change our behavior such as in the health sphere. The amount of data generated by these devices is so overwhelming that it is difficult to get useful insight from it. Luckily, in the domain of artificial intelligence technique...
Article
Technology driven interventions provide us with an increasing amount of fine-grained data about the patient. This data includes regular ecological momentary assessments (EMA) but also response times to EMA questions by a user. When observing this data, we see a huge variation between the patterns exhibited by different patients. Some are more stabl...
Article
Full-text available
In this paper, we explore the potential of predicting therapy success for patients in mental health care. Such predictions can eventually improve the process of matching effective therapy types to individuals. In the EU project E-COMPARED, a variety of information is gathered about patients suffering from depression. We use this data, where 276 pat...
Article
Recent developments in the field of sensor devices provide new possibilities to measure a variety of health related aspects in a precise and fine-grained manner. Subsequently, more empirical data will be generated than ever before. While this greatly improves the opportunities for creating accurate predictive models, other types of models besides t...
Conference Paper
Full-text available
Predictive modeling of future health states can greatly contribute to more effective health care. Healthcare professionals can for example act in a more proactive way or predictions can drive more automated ways of therapy. However, the task is very challenging. Future developments likely depend on observations in the (recent) past, but how can we...
Article
Predicting therapeutic outcome in the mental health domain is of utmost importance to enable therapists to provide the most effective treatment to a patient. Using information from the writings of a patient can potentially be a valuable source of information, especially now that more and more treatments involve computer-based exercises or electroni...
Conference Paper
Full-text available
Information in Electronic Medical Records (EMRs) can be used to generate accurate predictions for the occurrence of a variety of health states, which can contribute to more pro-active interventions. The very nature of EMRs does make the application of off-the-shelf machine learning techniques difficult. In this paper, we study two approaches to mak...
Article
Information in Electronic Medical Records (EMRs) can be used to generate accurate predictions for the occurrence of a variety of health states, which can contribute to more pro-active interventions. The very nature of EMRs does make the application of off-the-shelf machine learning techniques difficult. In this paper, we study two approaches to mak...
Conference Paper
Full-text available
Mental health related problems are responsible for great sorrow for patients and social surrounding involved. The costs for society are estimated to be 2.5 trillion dollar worldwide. More detailed data about the mental states and behaviour is becoming available due to technological developments, e.g. using Ecological Momentary Assessments. Unfortun...
Article
Over the past years, research utilizing routine care data extracted from Electronic Medical Records (EMRs) has increased tremendously. Yet there are no straightforward, standardized strategies for pre-processing these data. We propose a dedicated medical pre-processing pipeline aimed at taking on many problems and opportunities contained within EMR...
Article
Objective: Machine learning techniques can be used to extract predictive models for diseases from electronic medical records (EMRs). However, the nature of EMRs makes it difficult to apply off-the-shelf machine learning techniques while still exploiting the rich content of the EMRs. In this paper, we explore the usage of a range of natural languag...
Conference Paper
Full-text available
Within the domain of health care, more and more fine-grained models are observed that predict the development of specific health (or disease-related) states over time. This is due to the increased use of sensors , allowing for continuous assessment, leading to a sharp increase of data. These specific models are often much more complex than high-lev...
Conference Paper
Electronic Medical Records (EMRs) provide a wealth of data that can be used to generate predictive models for diseases. Quite some studies have been performed that use EMRs to generate such models for specific diseases, but most of them are based on more traditional techniques used in medical domain, such as logistic regression. This paper studies...
Conference Paper
Full-text available
Automatically generating behavior for Non-Player Characters (NPCs) in serious games can be problematic as the specification of their behavior heavily relies on the availability of domain expertise. This expertise can be difficult and costly to extract, and the specified behavior usually does not allow for generalization to new scenarios or users. A...
Conference Paper
Parameter controllers for Evolutionary Algorithms (EAs) deal with adjusting parameter values during an evolutionary run. Many ad hoc approaches have been presented for parameter control, but few generic parameter controllers exist. Recently, successful parameter control methods based on Reinforcement Learning (RL) have been suggested for one-off ap...
Article
More than a decade after the first extensive overview on parameter control, we revisit the field and present a survey of the state-of-the-art. We briefly summarize the development of the field and discuss existing work related to each major parameter or component of an evolutionary algorithm. Based on this overview, we observe trends in the area, i...
Article
As users can have greatly different preferences, the personalization of ambient devices is of utmost importance. Several approaches have been proposed to establish such a personalization in the form of machine learning or more dedicated knowledge-driven learning approaches. Despite its huge successes in optimization, evolutionary algorithms (EAs) h...
Conference Paper
Parameter controllers for Evolutionary Algorithms (EAs) deal with adjusting parameter values during an evolutionary run. Many ad hoc approaches have been presented for parameter control, but few generic parameter controllers exist and, additionally, no comparisons or in depth analyses of these generic controllers are available in literature. This p...
Conference Paper
Colorectal cancer (CRC) is a relatively common cause of death around the globe. Predictive models for the development of CRC could be highly valuable and could facilitate an early diagnosis and increased survival rates. Currently available predictive models are improving, but do not fully utilize the wealth of data available about patients in routi...
Article
Parameter control in Evolutionary Computing stands for an approach to parameter setting that changes the parameters of an Evolutionary Algorithm (EA) on-the-fly during the run. In this paper we address the issue of a generic and parameter-independent controller that can be readily plugged into an existing EA and offer performance improvements by va...
Conference Paper
Gaining insight on the effect of policies upon the Agricultural domain is essential to effectively regulate such a sector. A variety of models have therefore been developed that enable a prediction of agricultural development under different policies. Most models do however not make predictions on a fine grained level, making it difficult to see th...
Conference Paper
Full-text available
In this paper, an approach is advocated to use a hybrid approach towards learning behavior for computer generated entities (CGEs) in a serious gaming setting. Hereby, an agent equipped with cognitive model is used but this agent is enhanced with Machine Learning (ML) capabilities. This facilitates the agent to exhibit human like behavior but avoid...