Content uploaded by Thomas Johnson
Author content
All content in this area was uploaded by Thomas Johnson on Jul 16, 2021
Content may be subject to copyright.
Sensor Fusion and The City: Visualisation and
Aggregation of Environmental & Wellbeing Data
1st Thomas Johnson
Smart Sensing Lab, Department of Computer Science
Nottingham Trent University, Nottingham, United Kingdom
thomas.johnson@ntu.ac.uk
2nd Eiman Kanjo
Smart Sensing Lab, Department of Computer Science
Nottingham Trent University,Nottingham, United Kingdom
eiman.kanjo@ntu.ac.uk
Abstract—The proliferation of miniaturized electronics has
fuelled a shift toward environmental sensing technologies rang-
ing from pollution to weather monitoring at higher granularity.
However, little consideration has been given around the rela-
tionship between environmental stressors (e.g. air pollution) and
mental wellbeing. In this paper, we aim at capturing fluctuations
in momentary wellbeing and behaviour in response to changes
in the ambient environment. This is achieved using a multi-
sensor fusion approach that simultaneously collected urban en-
vironmental factors (e.g. including PM10, PM2.5, PM1.0, Noise,
Reducing gases, NH3), body reactions (physiological reactions
including EDA, HR and HRV) and users perceived responses
(e.g. self-reported geo-tagged valence). Our approach leverages
an exploratory data visualisation along with geometrical and
spatial data analysis algorithms, allowing spatial and temporal
comparisons of data clusters in relation to people’s wellbeing.
The effectiveness of our approach is demonstrated through a
positive correlation between environmental factors and physi-
ology reactions. By implementing spatial visualisation with our
real-world data shows the potential opportunities to understand
how the environment can effect mental wellbeing.
Index Terms—Urban, Environment, Voronoi, Visualisation,
Wellbeing
I. INTRODUCTION
The short and long term exposure to environmental urban
factors (such as air pollution, gases, particulates and noise)
can significantly impact an individual’s wellbeing and mental
health [1], [2]. The World Health Organisation (WHO) [3]
find 91% of places in the world are above the natural allowed
limit for pollution, resulting in people breathing in dangerous
levels of pollution. In addition, wellbeing illnesses are dras-
tically increasing around the world, expecting to cost the UK
economy over £2.4 billion per year [4]. To seek relief, many
use outdoor recreation. However, some environments are busy,
noisy, hectic and polluted resulting in a negative impact to
wellbeing and mental health. Through recent technological
advances, several studies have shown new opportunities to
assess the effects of the urban environment on humans.
For example, Heart Rate (HR) monitors and Electrodermal
Activity (EDA) sensors have shown enhanced results when
following the exposure to restorative environments.
Many scientific studies [5], [6] show that there is a rela-
tionship between environmental noise and mental health and
human well-being. The research results proved that noise can
impair productivity and cause serious health problems such
as chronic stress and heart diseases [7]. Physiological signals
contain useful patterns that aim to help identify individual’s
mental state. For example, using physiological sensors such
as Electrocardiogram, Electromyogram, skin conductance and
respiration has found to determine a driver’s overall stress
levels [8]. Furthermore, recent studies have begun to use
physiological sensors to identify an individual’s mood and
emotions [9], [10].
There are many applications emerging in the area of
environmental monitoring and the impact the environment has
on health and well-being [6] [11]. For instance, wearables,
low power, air quality and environmental monitoring sensor
sampling a range of air pollutants (CO, NO2 and O3). The
sensors in the system using real-time information helped those
suffering from health related problems such as asthma which
in turn helped those individual’s to avoid highly-polluted
environments [12]. Furthermore, the application NoiseSpy is
a low-cost sound measurement device that monitors environ-
mental noise levels, allowing users to explore an environment
while visualising noise levels in real time [5].
In this paper, we investigate the potential of Sensor fusion
of environmental sensors, bio-metric, contextual, behavioural
and self-reported data in an urban context to assess the mental
reactivity to environmental stressors. The built environments
considered in this work consist of both green spaces and
urban built area allowing us to understand how the exposure
to natural green spaces may promote an improved mental
wellbeing. Voronoi polygons [13] and heat maps are exploited
to visualise the interplay between the different sources of
data. By visualising these viable sources of city data, it will
be possible to expound the interactions between environment
variables, and on-body changes in real-time, which may be
useful to decipher the hidden links between urban stressors
and mental well-being. Such knowledge is key to develop
interventions addressing these stressors or mitigating their
effects.
II. RE LATE D WOR KS
Continuous and repeated exposure to environmental factors
(such as air pollution, weather and noise) have been shown
to impact our health [14], mental health [15], and wellbeing
[1], [6]. Studies in recent years have begun focusing on
how pollution (such as gases, particulates) can impact an
individual’s wellbeing and mental health.
Personal mobile sensing has been proven a good method in
understanding how the environment is impacted by wellbeing
and mental health through their ability to collect data contin-
ually and in real-time [1], [6], [5]. Technological advances
in sensors allow users exposure to the environment to be
accurately assessed and calculated [16]. Furthermore, mobile
phones and physiological sensors present many opportunities
to observe human emotion in urban environments [17].
DigitalExposome [1] used a sensor fusion approach to
study the environmental factors (such as pollutants, noise and
gases) and how they can directly impact walking around a
specified route around a polluted urban environment. The
results demonstrated how high levels of particulate matter
noticeably impacted HRV and EDA.
Additionally, ExpoApp [11] used a sensor fusion approach
to understand the impact of short term exposure to high
polluted environments. Their analysis suggests a positive
correlation between the environment, body temperature, Elec-
troDermal Activity (EDA), motion and Heart Rate (HR). In
addition, ’Project HELIX’ studied the environmental impact
whereby increased levels of blood pressure, asthma, allergy
related illnesses and behaviour issues were found for those
living in urban environments.
Several studies have investigated how environmental noise
can be monitored in real-time [5], [1].
In 2021, [2] a 25-year study results in that air pollution,
particularly nitrogen oxide and particulate matter could be
a risk factor in developing mental health symptoms later
on in life. Neighbourhoods with worse physical, social and
economical conditions were found to have the highest levels
of air pollution.
Research has highlighted how a number of environmental
pollutants can be associated with mental health conditions
[15]. In particular the work shows that environments with
higher levels of particulate matter lead to increased anxiety
[18]. Furthermore, more recent studies have reported an
association between environmental pollution and anxiety for
those living with pre-existing health conditions [19], and
causing on-set depression for all ages [20].
III. EXP ER IM EN TAL SET UP / METHODOLOGY
The experiment was conducted around Nottingham Trent
University Clifton Campus which is located within a busy,
polluted, urban environment. The purpose of the study is
to observe the relational impact between our environment,
physiology, wellbeing and mental health. The study involved
a total of 25 adult participants who individually were shown a
set route to follow around an urban environment. The walking
route was made up of a quiet and busy polluted area. Due
to the on-going COVID-19 pandemic restrictions it has been
difficult to recruit any further participants to undertake the
study.
Participants’ used three devices, which included a portable
environmental monitoring kit (Enviro-IoT), Empatica E4
and a Samsung smartphone pre-loaded with the app ’Env-
BodySens’. The portable environmental sensing device col-
lected data every 20 seconds and sampled such as; Particulate
Matter (1.0, 2.5 and 10), Carbon Dioxide, Volatile Organic
Compound, Ammonia ’NH3’, reduced and oxidised gases.
The Empatica E4 device sensors’ data was sampled at dif-
ferent rates with HR at 1Hz and EDA, BVP, HRV and body
temp at 64Hz.
The Samsung smartphone was pre-installed with a custom
built app called ’EnvBodySens’ used to continually self-report
participant’s perceived wellbeing. In designing this custom
app we adopt the ’Personal Wellbeing Index for Adults’ which
asks users repeatedly how they are feeling [21]. This ’state
of the art’ method for calculating subjective wellbeing has
been shown to give high reliability [22]. We have adapted
this approach into the form of a five-point Likert SAM scale.
Within the app, the user is met with 5 well known emojis
from 1=negative/low to 5=positive/high. At the start of the
experiment, the participant is reminded to continually select
an emoji which best represents how they are feeling at a
specific time along the specified route.
Shortly after collection, the data was cleaned and pre-
processed. Linear interpolation [23] was used to sample the
data due to the varying rates across the data collection devices.
The physiological data was down-sampled to a rate of 1Hz
and the Enviro-IoT device was up-sampled in order to match
this rate. At the pre-processing stage, issues surrounding the
gas sensors for carbon dioxide and volatile organic compound
were discovered whereby they did not collect sufficient data
to analyse. In total there are 13,681 lines of data across all
participants that took part in the experiment.
IV. SPATIA L VISUALISATION
To help understand the impact of the environment on
our mental wellbeing and observe the correlations between
independent and dependent variables, we employ the use of
different spatial and data visualisation techniques such as
heat-maps, correlation matrix and Voronoi.
Fig. 1. Heatmaps showing environmental and physiological sensor data.
In visualising dynamic data trends, the experiment route
is divided into smaller areas which will highlight particular
correlations between variables. Mapping sensor data is a
proven method in visualising dynamic data [24].
Figure 1, presents several heat-maps with environmental
and physiological variables plotted with level of intensity as
the user moved within the specified urban route. Each heat-
map shows several hot-spots scattered along the busy road
(right of map) and not confined to one particular area. While
heat-maps show the intensity of each variable based on GPS
coordinates, our maps indicate the real distribution of all
sensor data. In particular, we can observe that at numerous
places around the route certain pollutants (PM2.5, Noise,
Reducing gases and NH3) also resulted in higher HRV and
EDA. This can been demonstrated by the temperature colour
scheme used whereby red signifies a more concentrated level.
In order to further understand the impact and the level to
which they correlate, we utilise a correlation matrix, depicted
at Figure 2.
Fig. 2. Correlation matrix between environmental and physiological data.
The correlation matrix shows how both environmental and
physiological collected sensor data was positively correlated
when compared. In particular, EDA is considered to correlate
with PM2.5 (0.23) and PM10 (0.20). In addition, HRV shows
a positive correlation between reducing gases (0.15), PM10
(0.25) and PM2.5 (0.25).
One of the biggest challenges in using this method is
within the metric layer visualisation that lies in scalability.
In particular, as with this experiment, there are large numbers
of records to map in a single city. Drawing all these records
at their accurate locations simultaneously will be impossible,
due to the large amount of rendering cost and the required
visual bandwidth.
To address these issues, we utilise a combinatorial com-
putational geometry algorithm called “Voronoi”, which is a
diagram partitioning a plane into regions based on distance
to points in a specific subset of the plane [13]. Diagrams are
divided into a set of regions called Voronoi cells, including
the space that is closest to the object (location). The size of
these cells give an indication of the density of the area a
certain object is in or the size of an object.
The cell structure also shows the Delaunay triangulation,
which enables the calculation of an object’s immediate set of
neighbours. The definition of a Voronoi cell is given by the
following Equation (1), where x is a planar metric space; p
is the set of generator points in the metric space; and d is
the distance between all points in x and a specific generator
point.
Vori ={x|d(x, pi)≤d(x, pj), j 6=i}(1)
The creation of a Voronoi tessellations is a dynamic pro-
cedure that is repeated until all of the points are represented
in adjacent polygons. If there are not a sufficient number of
particles to satisfy Equation (1), then the Voronoi diagram
is only partially filled and the data is redistributed. The
Voronoi diagram is composed of a collection of tessellations
as Equation (2) where Vor, where:
V or ={V or1, V or2. . . Vorn}(2)
By giving each polygon a value that corresponds to the
sensor value collected in a particular GPS coordinate, it is
possible to divide the space into adjacent polygons with
different sensor reading which are represented in colours. This
type of spatial analysis, helps in understanding the degree to
which a place is similar to other nearby places, the correlation
among different spatial objects needs to be defined. We have
mapped users to different regions based on their mobility and
form several user groups for further exploration.
V. R ES ULTS & DISCUSSION
Fig. 3. (Top) Voronoi overlay from one participant. Each polygon represents
one location trace tagged with a wellbeing label while collecting the data in
a specified route, (bottom, collected label data from start to end).
This paper presents a novel approach of Voronoi to demon-
strate the impact of urban environments on human physiology.
Understanding the interaction between personal characteris-
tics, human physiology and urban environment will be useful
to observe the impact of environmental stressors to mental
wellbeing.
Figure 3 shows the self-reported wellbeing data using the
app whilst walking in real-time around the urban specified
route. To represent the metric value of each record, we apply
a linear mapping from the value to the colour hue, which
forms a white to red palette. In particular, the red polygons
are indicating that there was negative sentiment by the user.
The yellow/white polygons aim to highlight where there was
a positive sentiment reported. On top of the polygons we have
plotted the reported label. Furthermore, we have added two
arrows between the label graph and map which highlight the
changes in wellbeing reported by the user. By observing these
changes, it is clear that when users travelled into a polluted
urban environment they reported negatively and when users
returned to green spaces where pollution was significantly low
they reported positively, indicated by yellow and white.
VI. CONCLUSION
In this paper, we have presented our sensor-fusion approach
to aid in quantifying the relationship between the environment
and wellbeing. Three indicators are proposed to evaluate
the relationship, which include heat-maps, correlation matrix
and Voronoi diagrams. As a result, the experiment is a
valuable example of how through spatial visualisations can
demonstrate the potential to visualise and explore a range of
environmental and physiological sensor data. In the results of
this paper, clear correlations are highlighted between changes
within the environment and physiological on-body responses.
The results from this study can be meaningful to city centre
planners which could utilise this data to add calming areas
such as green spaces in environments which are considered to
give a negative impact to wellbeing. In addition, keeping walk
ways a certain distance from busy roads could be considered
to help improve wellbeing. By understanding the environ-
ments which can induce mental wellbeing and those that aid
relaxation we hope future urban developments can use this
method of data visualisation to create restorative experiences
in polluted areas to help improve mental wellbeing.
As future works, the increase of participants and additional
environmental sensors (particulates, gases) will be advan-
tageous to help in understanding and evaluating the full
potential this form of spatial exploration creates.
REFERENCES
[1] T. Johnson, E. Kanjo, and K. Woodward, “DigitalExposome: Quantify-
ing the Urban Environment Influence on Wellbeing based on Real-Time
Multi-Sensor Fusion and Deep Belief Network,” jan 2021.
[2] A. Reuben, L. Arseneault, A. Beddows, S. D. Beevers, T. E. Moffitt,
A. Ambler, and R. M. Latham, “Association of Air Pollution Exposure
in Childhood and Adolescence With Psychopathology at the Transition
to Adulthood,” vol. 4, no. 4, pp. 1–14, 2021.
[3] W. H. Organisation, “Ambient air pollution,” 2018. [Online]. Available:
http://www.who.int/airpollution/ambient/en/
[4] Perkbox, “The 2018 UK Workplace Stress Survey — Perkbox,” 2018.
[5] E. Kanjo, “NoiseSPY: A real-time mobile phone platform for urban
noise monitoring and mapping,” Mobile Networks and Applications,
vol. 15, no. 4, pp. 562–574, 2010.
[6] E. Kanjo, E. M. Younis, and N. Sherkat, “Towards unravelling the
relationship between on-body, environmental and emotion data using
sensor information fusion approach,” Information Fusion, vol. 40, no.
May, pp. 18–31, 2018.
[7] S. A. Stansfeld and M. P. Matheson, “Noise pollution: Non-auditory
effects on health,” British Medical Bulletin, vol. 68, pp. 243–257, 2003.
[8] C. L. Lisetti and F. Nasoz, “Using noninvasive wearable computers to
recognize human emotions from physiological signals,” pp. 1672–1687,
sep 2004.
[9] K. Woodward, E. Kanjo, A. Oikonomou, and S. Burton, “Emoecho:
A tangible interface to convey and communicate emotions,” in Ubi-
Comp/ISWC 2018 - Adjunct Proceedings of the 2018 ACM Interna-
tional Joint Conference on Pervasive and Ubiquitous Computing and
Proceedings of the 2018 ACM International Symposium on Wearable
Computers. Association for Computing Machinery, Inc, oct 2018, pp.
746–749.
[10] K. Woodward, E. Kanjo, D. Brown, T. M. McGinnity, B. Inkster,
D. J. Macintyre, and A. Tsanas, “Beyond mobile apps: A survey of
technologies for mental well-being,” arXiv, 2019.
[11] D. Donaire-Gonzalez, A. Valent´
ın, E. van Nunen, A. Curto, A. Ro-
driguez, M. Fernandez-Nieto, A. Naccarati, S. Tarallo, M. Y. Tsai,
N. Probst-Hensch, R. Vermeulen, G. Hoek, P. Vineis, J. Gulliver, and
M. J. Nieuwenhuijsen, “ExpoApp: An integrated system to assess
multiple personal environmental exposures,” Environment International,
vol. 126, pp. 494–503, 2019.
[12] P. Zappi, E. Bales, J. H. Park, W. Griswold, and T. ˇ
Simuni, “The
CitiSense Air Quality Monitoring Mobile Sensor Node,” in Proceedings
of the 11th ACM/IEEE Conference on Information Processing in Sensor
Networks, 2012.
[13] Q. Dut and V. Fabert, “Centroidal Voronoi Tessellations: Applications
and Algorithms*,” Tech. Rep. 4, 1999.
[14] M. Kampa and E. Castanas, “Human health effects of air pollution,”
Environmental Pollution, vol. 151, no. 2, pp. 362–367, jan 2008.
[15] J. King, “Air pollution, mental health, and implications for urban
design: a review,” Journal of Urban Design and Mental Health, no.
March, p. 4:6, 2018.
[16] A. Stamatelopoulou, D. Chapizanis, S. Karakitsios, P. Kontoroupis,
D. N. Asimakopoulos, T. Maggos, and D. Sarigiannis, “Assessing
and enhancing the utility of low-cost activity and location sensors for
exposure studies,” Environmental Monitoring and Assessment, vol. 190,
no. 3, mar 2018.
[17] N. Alajmi, E. Kanjo, A. Chamberlain, and N. El Mawass, “Shopmobia:
An Emotion-Based Shop Rating System,” 2013.
[18] M. C. Power, M. A. Kioumourtzoglou, J. E. Hart, O. I. Okereke,
F. Laden, and M. G. Weisskopf, “The relation between past exposure
to fine particulate air pollution and prevalent anxiety: Observational
cohort study,” BMJ (Online), vol. 350, 2015.
[19] V. C. Pun, J. Manjourides, and H. Suh, “Association of ambient air
pollution with depressive and anxiety symptoms in older adults: Results
from the NSHAP study,” Environmental Health Perspectives, vol. 125,
no. 3, pp. 342–348, 2017.
[20] N. A. Ali and A. Khoja, “Growing evidence for the impact of air
pollution on depression,” Ochsner Journal, vol. 19, no. 1, p. 4, 2019.
[21] R. A. Cummins and F. A. S. Ps, Personal Wellbeing Index-Adult (PWI-
A) (English) 5 th Edition The International Wellbeing Group MANUAL
2013 Personal Wellbeing Index-Adult.
[22] A. A. Yousefi, A. Alipour, and N. Sharif, “Reliability and validity of
the ”personal well-being index - adult” in mothers of mentally retarded
students in north of Tehran-Iran,” Iranian Journal of Psychiatry and
Behavioral Sciences, vol. 5, no. 2, pp. 106–113, 2011.
[23] C. R. Boxer, “Science and Civilisation in China. Vol. III. Mathematics
and the Sciences of the Heavens and the Earth,” International Affairs,
vol. 36, no. 2, pp. 270–270, apr 1960.
[24] D. Mashima, S. Kobourov, and Y. Hu, “Visualizing dynamic data with
maps,” IEEE Transactions on Visualization and Computer Graphics,
vol. 18, no. 9, pp. 1424–1437, 2012.