A Generic Architecture for Sensor Data Integration with the Grid

Abstract

This paper describes the design and implementation of a model of how to integrate sensors and devices into a Grid infrastructure.
We describe its proxy-based approach, the port-type requirements and the set of tools implemented to facilitate configuration
of experimental scenarios. Two real world devices, a wearable medical jacket and an Antarctic lake probe, deployed out in
the field using this architecture are described, along with their relevance in scientific research.

Full text

Page 1

A Generic Architecture for Sensor Data Integration with the GRID

Jan Humble1 Chris Greenhalgh1
Henk L Muller2
1 University of Nottingham, Department of Computer Science
2 University of Bristol, Department of Computer Science
Alastair Hamsphire1
Stefan Rennick Egglestone1


Abstract

This paper describes the design and implementation
of a model of how to integrate sensors and devices into
a GRID infrastructure. We describe its proxy-based
approach, the port-type requirements and the set of
tools implemented to facilitate configuration of
experimental scenarios. Two real world devices, a
wearable medical jacket and an Antarctic lake probe,
deployed out in the field using this architecture are
described, along with their relevance in scientific
research.

1. Introduction

The GRID platform provides great potential for
delivering tools for computation, management of
resources, and data storage for scientists [1, 2]. A
substantial amount of funding has concentrated on the
requirements for post-processing of gathered scientific
data. Our interest, however, lies beyond the lab and its
suite of software tools. We wish to accommodate the
integration of data collected from remotely located
hardware sensors and devices into the GRID
infrastructure and the Open Grid Services Architecture
(OGSA) [3] as we know it today.
Experimental scenarios require for the most part a
contextual analysis of the gathered data, usually from a
range of sensors attached to a particular device.
Correlation of the data from different sources and
sensors (e.g. time, location and activity of a medical
subject) is useful in explaining patterns and a
requirement of both clinicians and “in the field”
scientists. We wish to facilitate the management of
such experimental scenarios and provide easy access to
historical or real-time data for further correlation
analysis. In some cases it becomes necessary not only
to make GRID facilities available to remote users, but
also to allow users of GRID services to directly
interact with remote
configuration).
devices (e.g. remote
Our architecture, constructed using the Globus
Toolkit 3 [4], and a set of tools facilitate the publishing
and processing of data by scientists without the need of
extensive knowledge of GRID-based development.
The devices and sensors that we are dealing with
typically have limited computational power, limited
memory and limited network connectivity (e.g.
intermittent or occasional). As such, they are not
suitable for directly hosting Grid Services given
current technologies. However, in some cases they are
capable of hosting Grid Services directly, while in
others we might anticipate improvements in underlying
technologies that would make this possible. Our
proxy-based approach will nonetheless guarantee
communication with a device representative at all
times, providing notification on current device and
connection state.
Two real world sensor devices were already
available as part of focused research by collaborating
institutions: the CyberJacket [5, 6] from Bristol
University, which collects data from patient life signals
and ambient data, and the eScience Antarctic lake
probe [7] from the University of Nottingham, which
records data from a lake in the Antarctic. In detailing
these, we will demonstrate how they were integrated
into the sensor architecture and their role in future
research together with GRID technologies.

2. Background

A common and generic architecture for monitoring
devices is motivated by the prolific and long-standing
research programmes in both health care and
environmental research.
In the UK, there has been a rising trend in
Telemedicine and Telecare clinical trials such as those
carried out by the Oxford centre for e-health [8], the
Biomedical Informatics
University [9] and the Glasgow Royal Infirmary and
Glasgow University [10]. These required custom built
sensors and infrastructures despite being of relatively
small scale.
group at Nottingham

Page 2

Our research partners have a long-standing interest
in the recording of physiological data for clinical
diagnosis [11, 12, 13, 14, 15] through sophisticated
bio-signal processing strategies [16, 17]. We also have
extensive experience in wearable devices and means of
extracting, archiving and manipulating context driven
data. Combining the various interests from the Science
community and wireless networking is a UK
government funded project entitled “Grid Based
Medical Devices for Everyday Health” [13]. It aims to
create tools and exercise strategies for unobtrusive
medical monitoring.
Professor Laybourn-Parry and her colleagues have
been studying the ecology of freshwater Antarctic
lakes for 12 years, and in particular the cycling of
carbon through the ecosystem [7]. It is a well
preserved and unique of environment and an
immensely valuable resource of information, but it’s
isolated location and harsh environment make it very
difficult to study. A monitoring device with a range of
sensors has taken over most of the labour-intensive
manual work of the past; however work still needs to
be done on exposing the data to the emerging GRID
scientific tools for the scientists “back home”.

3. Architecture

Our approach is, wherever possible, to make
devices and sensors available on the Grid as if they
were first class Grid Services. To this end we have
devised a proxy-based approach and defined two new
application-independent port types: one for a generic
sensor, and one for a generic device (which is assumed
to host a number of sensors). These port types can be
supported directly by sensors and devices of sufficient
capability and reliability of communication. However,
all of the devices and sensors that we are working with
at the moment depend on proxy Grid Services to
implement these interfaces on their behalf. The – low
capability and/or intermittently connected – devices
then communicate with their respective proxies using
whatever protocol is appropriate, as and when
communication channels are available.

3. 1 Device and Sensor Proxies

In our architecture, device proxies are used to
expose sensing devices as grid services and sensor
proxies are used to expose individual sensors as grid
services. A physical sensing device may have several
individual sensors attached to it. To model this, a
device proxy is therefore a service group, with a
number of sensor proxies as service group entries. This
allows client applications to perform operations on the
device as a whole (e.g. to configure the device) or to
interact with individual sensors (e.g. to collect the
latest data).
To expose a physical device and its sensors as grid
services, a client
DeviceProxyFacoryService, passing it an XML
configuration file. The file specifies the initial
configuration of the device and its sensors. The
DeviceProxyFactoryService
DeviceProxyService instance, which in turn creates a
SensorProxyService instance for each physical sensor
specified in the configuration file.
A client application is able to contact the device
proxy to retrieve, among other things, a list of
available sensors. The client is then able to contact
some or all of the sensors to, for example, retrieve the
latest readings. Sensor proxies can also provide other
methods to interact with or reconfigure the physical
sensor (such as changing the sampling rate of the
sensor or monitoring battery life).
Sensor data produced by a physical sensor is sent to
an instance of a sensor proxy service, where it is stored
until downloaded or discarded. The duration for which
the data is stored at the proxy is determined by a
measurement discard policy dependent on the sensor.
For example, because one of our portable devices in
the wearable medical devices project has limited
memory, stored data is discarded regularly. Storage of
data at the proxy is only intended as temporary storage.

Physical Sensing Device
Device Proxy Service
(Service Group)
interacts with a
creates a
Retrieve
sensor data
Configure
sensor
Configure
device


Physical
sensors

Sensor Proxy
Service (Service
Group Entry)
Configure
sensor
Deliver data
Client

Figure 1 Representing sensing devices as GRID
devices
The sensor proxy stores an internal list of all un-
discarded sensor data alongside a sequence number.
Client applications are able to request a subset of the
currently held sensor data (bound by min and max
sequence number values),
MeasurementByCountQueryMinMax
element. The sensor proxy service is able to expose
dynamic data as grid service data by using a service
data callback, which generates an array of data items
between the specified min and max values when a
service data query is made. Measurements returned
are represented in XSIL format [19] in compliance
with standardized databases,
by requesting
service
the
data
but with some

Page 3

straightforward modifications. A typical output might
look like this …

<Table>
<ColumnName=”wind speed” Type=”float”
Unit=”ms-1”/>
<Column Name=”wind direction”
Type=”float”/>
<Stream Type=”Local” Delimeter=”,”>
10, 23
12, 45
</Stream>
</Table>

Different kinds of sensors may take measurements
at widely varying rates, and each measurement may
represent differing quantities of data. A measurement
publishing policy allows the way in which the sensor
reveals (publishes) new measurements to be controlled.
For example, a very rapidly sampling sensor might be
configured to announce new measurements no more
than 5 times per second (even if it is taking and
making available 5000 measurements during this
period).

3.2 Managing Trial Scenarios

We use an application called a trial manager to
pump data from the proxies into a persistent database.
The user is presented with a list of devices with
available sensors, and selects sensors to store data
from. The trial manager uses the database grid
interface to configure the database to store data from
the sensors, and a sensor data pump is then
instantiated to deliver data produced by the sensor to
the database.
The sensor data pump registers for notification of
changes in the MeasurementCounter service data
element in the proxy of the sensor for which it has
been instantiated. When notified, it fetches new data in
the proxy’s measurement queue and stores it in the
database. For sensors with a high sampling rate, it can
register for a notification after a number of new items
of data, and can then retrieve the last posted items of
data using the MeasurementByCountQueryMinMax
service data element.

Sensor Proxy Service
Store
data
Retrieve data
(min=retrieved,
max=counter)
Deliver
notification
(counter)
Register for
notifications
Database
Service
Sensor Data
Pump
MeasurementCounter SDE
MeasurementByCountQueryMinMax SDE
Other SDE ...
Measurements
retrieved
counter

Figure 2 Transferring sensor data to a database
Once data is stored in the database, methods on its
grid service can be used to access data and metadata.
These include:
• getListOfDevices() : returns a list of device Ids
stored in the database
• getListOfSensorsPerDevice(deviceID) : returns a
list of sensor Ids for sensors attached to a device
specified by a given device Id.
• queryBySensorIDandExpression(sensorID, where)
– this method returns an XSIL string containing
sensor readings for the sensor specified by
sensorID and conforming to the given where
clause. For example, the where clause might
indicate a date range, between which readings
should be returned.

Other convenience methods are available to, for
example, return data from several sensors of the same
type. Further, the interface could easily be extended to
provide more advanced querying mechanisms.

3.3 Tools

Since devices are self-describing they provide
information about their own data output format. We
take advantage of this property to automatically plot
views of live data. A modifiable template plot window
is easily summoned for quick monitoring, and is used
as a base library for more complex monitoring
applications.
Users can also specify a dataflow computation by
creating a dataflow graph. They then perform the
computation by invoking a scheduling algorithm on
the graph. The scheduling algorithm examines
processing units in the graph, and executes a
processing unit only when it has sufficient new input
data and when it has an output that is clear onto which
data can be placed.
The dataflow methodology was chosen as a
potential way of offering easy access to data on an
instance of the grid database, because all the
complexity of accessing the database can be hidden in
the simple visual representation of a processing unit.

4. Wearable Medical Devices

The wearable medical devices project aims to allow
patients to be monitored remotely, allowing short-term
clinical exercises without restricting the patient’s
movements. This would, for example, allow patients
to walk about the hospital, clinical office or home
whilst being monitored, subject to them being within
range of total or partial wireless network connectivity.

Page 4

Patients wear a jacket containing a small computer
(bitsy), attached to which are several sensing devices
through a 9-wire bus [6]. Each sensor takes a reading
at a predefined interval and sends the reading to the
wearable computer.

Bitsy
Strong
ARM
LiIon
GPS
SpO2
ECG
Electrodes
SpO2Sensor

Figure 4 Bristol Lifejacket and modular design
A Wavelan card attached to the bitsy allows the
sensor readings to be transferred to network services
for analysis and storage. Should the analysis suggest
the patient is in immediate physiological danger, a
clinician could be alerted. Additionally, data
collection from a patient over a long period of time
could be further analysed to reveal long term health
problems.
In a Wavelan setting readings are produced
regularly and continuously, so it was decided that
individual readings would be reported. The bitsy has a
lightweight implementation of SOAP called GSOAP
[20], and this was used to add readings to individual
sensor proxy measurements queues as they were
produced. This data was used either through a live
monitoring application or by pumping it into a
database.
Typically, information is gathered from multiple
sensor types in order to accurately track and diagnose a
given disease state. The combination of electro-
physiological and other medical parameters that must
be monitored for a specific medical condition varies
widely; subsequently the sensors used in this study are
designed to ease their introduction and withdrawal
from the monitoring exercise as necessary. At the time
of writing, the sensors integrated include:

Blood oxygen saturation sensor (SpO2): derives
the level of oxygen present in arterial blood via a
calculation on the ratio of light absorption resulting
from oxygenated and reduced haemoglobin through a
well-perfused body part (e. g. the finger).
Electrocardiogram (ECG): signal produced on the
surface of the skin by the electrical activity of the
JavaPhone
Blood sugar
meter
Data logger
PAR sensor
PAR sensor
Other sensors
Cyberjacket
(Bitsy)
S/w module
ECG sensor
accelerometer
GPS receiver
9 - wire bus
( pluggable )
JavaPhone proxyJavaPhone proxy
Blood sugar
meter proxymeter proxy
Data logger
proxyproxy
PAR sensor
proxyproxy
Other sensor
proxyproxy
Cyberjacket
proxyproxy
ECG sensor
proxyproxy
Other sensor
proxyproxy
Generic device proxy
factory(s) factory(s)
GPRS
Iridium
802.11
Multicast
beacon
DF DF
DD
DD
DD
SS
SS
SS
SS
SS
Device Proxy
Management ClientManagement Client
Register new
device device
New device
configurationconfiguration
Sensor
data-pumpdata-pump
RDBMSRDBMS
Sensor
Database
ServiceService
Sensor
data-pumpdata-pump
Sensor
data-pumpdata-pump
Trial
managermanager
Add sensor to
trial database
Live monitoring display
Sensor and
device status
displaydisplay
GPS live mapGPS live map
‘elipse of
normality’
visualisation
Scrolling
sensor charts
‘elipse of
normality’
visualisation
SS
Data
chooser/
fetcher
Table views Table views
Graph viewsGraph views
Dataflow user interface
Data
chooser/
fetcher
DF
D
S
GRID Port Types:
DF
D
S
Sensor
DeviceProxyFactory
Device
Sensor
JavaPhone
Blood sugar
meter
Data logger
PAR sensor
PAR sensor
Other sensors
Cyberjacket
(Bitsy)
S/w module
ECG sensor
accelerometer
GPS receiver
9 - wire bus
( pluggable )
Blood sugar
Data logger
PAR sensor
Other sensor
Cyberjacket
ECG sensor
Other sensor
Generic device proxy
GPRS
Iridium
802.11
Multicast
beacon
Device Proxy
Register new
New device
Sensor
Sensor
Database
Sensor
Sensor
Trial
Add sensor to
trial database
Live monitoring display
Sensor and
device status
Scrolling
sensor charts
Dataflow user interface
GRID Port Types:
DeviceProxyFactory
Device
Figure 3 Overview of architecture diagram

Page 5

heart.
Skin temperature: monitored using clinical grade
thermistors.
GPS: Satellite positioning while outdoor.

Live monitoring of a patient’s life signals can be
accomplished by posting data through a notification or
a request-post. The origin of such post events can be
directly from the devices (via the proxy), after a
midway post-process, via notification from database
updates, or from computation services. In this manner,
clinicians are not only capable of accessing the raw
patient data, but the data after a predetermined filtering
mechanism or data as output from real-time
computations making use of a window of historical
data (e. g. to detect immediate health risk warnings).

5. The Antarctic Device

The body of the Antarctic device contains a
commercial scientific data logger, facilities to
communicate via Iridium satellite modem, a battery,
and solar panels to augment the power available from
the battery. It is only deployed when the ice is thick
enough to support its weight. Various sensors are
attached to the data logger, including light (PAR), air
temperature (both above the ice and at various depths
in the water column), and wind speed and direction.
This logged data is collected every few days, either
via the satellite link or in person using a serial cable
connected to the device. Each data file collected is
post-processed to add units, and to generate correctly
calibrated values. This file is then emailed to scientists
in Nottingham for analysis.
Scientists in Nottingham are using data collected
from the device to develop a physical model of the
lake environment, which is used to help develop
models of carbon cycling in the lake.


Figure 5 the Antarctic device and connected
sensors, which is currently deployed on Crooked
Lake in the Vestfold Hills.
It was decided not to maintain a continuous
connection to the Antarctic device over Iridium
satellite connection due to expense. Therefore, a table
of data containing hundreds of sets of readings is
downloaded from the device and is then processed by
the device proxy. For each sensor attached to the
device, the device proxy extracts individual readings
from the table and appends them to the sensor proxy’s
measurement queue. Data can then be pumped into an
instance of the grid database to be made available.

6. Summary

We have described an architecture around Globus
Toolkit 3 to integrate devices and sensors into the
GRID. A proxy model intermediates between
remotely connected devices and the local network and
acts as an always-available first class GRID service.
This facilitates as seamless as possible integration of
collected data for immediate post-processing by
available resources. We presented two examples with
different methods and issues of connectivity utilizing
an implementation of the architecture.

10. References

[1] I. Foster, C. Kesselman, J. Nick, S. Tuecke, “The
Physiology of the Grid: An Open Grid Services Architecture
for Distributed Systems Integration”, Open Grid Service
Infrastructure WG, Global Grid Forum, June 22, 2002.
[2] de Roure, D., Jennings, N., Shadbolt, N “Research
Agenda for the Semantic Grid: A future e-Science
infrastructure”
http://dcs.gla.ac.uk/Nesc/general/technical_papers/DavidDeR
oure.etal.SemanticGrid.pdf
[3] The Open Grid Services Architecture (OGSA)
http://www.globus.org/ogsa
[4] GLOBUS. The
http://www.globus.org/toolkit/, April 2003.
[5] Barratt, C, et al, "Extending the Grid to Support Remote
Medical Monitoring", Proceedings of the 2nd UK e-Science
All Hands Meeting 2003
[6] J. Crowe, B. Hayes-Gill, M. Sumner, C. Barratt, B.
Palethorpe, C. Greenhalgh, O. Storz, A. Friday, J. Humble,
C. Setchell, C. Randell, H. L Muller, "Modular Sensor
Architecture for Unobtrusive Routine Clinical Diagnosis",
24th International Conference on Distributed Computing
Systems Workshops - W3: IWSAWC (ICDCSW'04), March
23 - 24, 2004, Hachioji, Tokyo, Japan.
[7] S. Benford, N. Crout, J. Crowe, S. Egglestone, M. Foster,
C. Greenhalgh, A. Hampshire, B. Hayes-Gill, J. Humble,
A. Irune, J. Laybourn-Parry, B. Palethorpe, T. Reid, and
globus toolkit.

Page 6

M. Sumner. e-Science from the antarctic to the GRID.
Escience All Hands Meeting 2003.
[8] The Oxford
http://www.medicine.ox.ac.uk/ndog/tmr/
[9] The Biomedical Informatics Group at Nottingham
University http://www.eee.nott.ac.uk/medical/
[10] Computer Assisted Reporting of Electrocardiograms,
Glasgow
http://www.gla.ac.uk/departments/medicalcardiology/researc
h/care.html
[11] C. Barratt, B. Hayes-Gill, H. Vyas, and J. Crowe.
Selection of pulse oximetry equipment for ambulatory
monitoring. Journal Medical Eng & Tech, 21(1):17–24,
2001.
[12] J. Crowe, A. Harrison, and B. Hayes-Gill. The
feasibility of long-term fatal heart rate monitoring in the
home environment using maternal abdominal electrodes.
Physiol. Meas, 16:195–202, 1995.
[13] J. Crowe, B. Hayes-Gill, B. Francon, L. Hardebecke, D.
Rogers, Y. Thong, P. Dimmock, K.Wyatt, and P. O’Brien.
Customisation of a personal digital assistant for logging
premenstrual syndrome symptoms. British Journal of
Healthcare Computing &
17(4):33–35, 2000.
[14] A. Harrison, B. Hayes-Gill, J. Crowe, and S. Chang.
The application of an Actel field programmable gate array in
the design of an ecg rr interval recorder. Journal Medical
Eng. and Tech, 19(6):198–204, 1995.
[15] J. Pieri, J. Crowe, B. Hayes-Gill, C. Spencer, K. Bhogal,
and D. James. Compact long-term recorder of the
transabdominal foetal and maternal electrocardiogram.
Med.Biol.Eng. & Comp., 39(1):118–125, 2001.
[16] J. Hall and J. Crowe. Ambulatory electrocardiogram
compression using wavelet packets to approximate the
Karhunen-Loeve transform. Applied Signal Processing,
3:25–36, 1996.
[17] A. Harrison, B. Hayes-Gill, J. Crowe, and S. Chang.
The application of an Actel field programmable gate array in
the design of an ecg rr interval recorder. Journal Medical
Eng. and Tech, 19(6):198–204, 1995.
[18] GRID based medical devices for everyday health.
http://www.gridoutreach.org.uk/docs/pilots/meddev.htm,
2002.
[19] XSIL
http://www.cacr.caltech.edu/projects/xsil
[20] gSOAP: Generator Tools for Coding SOAP/XML Web
Services in C
http://www.cs.fsu.edu/~engelen/soap.html


Centre for e-health
University
Information Management,
Specification,
and C++.