Content uploaded by Abdulmotaleb El Saddik
Author content
All content in this area was uploaded by Abdulmotaleb El Saddik on Jan 14, 2014
Content may be subject to copyright.
Ambient Intelligence Multimedia
Management Framework
M. Anwar Hossain, M. Shamim Hossain and Abdulmotaleb El Saddik
Multimedia Communications Research Laboratory (MCRLab)
School of Information Technology and Engineering (SITE)
University of Ottawa, Canada
{anwar, shamim, abed}@mcrlab.uottawa.ca
Abstract – Ambient intelligence has been gaining a lot
of momentum in recent years due to the unprecedented
potential it can bring by achieving a ubiquitous,
inconspicuous, and interconnected technological
surroundings that are sensitive, adaptive, and
responsive to the presence of humans. Multimedia
management including its creation, annotation,
storage, sharing, retrieval and delivery in such a
distributed and interconnected environment, is a
challenging task to perform. The challenge stems
primarily from the heterogeneity of multimedia data
that need to be managed and delivered while taking
into consideration the context of users, devices,
networks, and surrounding environments. In this paper,
we first state the requirements of an ambient
intelligence environment and keeping these
requirements in mind, propose a multimedia
management framework. We describe the several
activities of multimedia management, its related issues,
and provide the architecture of the proposed
framework. A prototype application is developed that
handles the capturing, storage, retrieval and delivery of
multimedia data. Finally, we present some
experimental results and outline future works in this
direction.
Keywords: Ambient intelligence, multimedia,
multimedia management, multimedia metadata,
context-aware, content repurposing.
1 Introduction
Ambient Intelligence (AmI) promotes a paradigm
where humans will be surrounded by intelligent and
intuitive interfaces provided by the interconnected
heterogeneous computing devices embedded into
everyday objects. The environment thus created will be
capable of recognizing and responding to the actions
and presence of individuals [16]. This vision initiated
several research projects that include smart home [14],
smart labs [22] and so on.
The interaction of people with technology-rich
ambient environment is expected to be seamless,
unobtrusive, and often invisible. Such a massively
distributed, integrated, and unconventional platform
poses huge challenges to the way multimedia
management is performed. The heterogeneity, mobility,
and ubiquity of ambient environment impacts the way
multimedia is captured, shared, annotated, stored and
delivered to different client contexts.
Multimedia is heterogeneous in forms (such as,
image, audio, video, graphics, animation, text and the
emerging haptic and sensory data) and complex in
nature (having spatial and/or temporal properties). Such
heterogeneity and complexity need to be addressed in
intuitive ways in the challenging world of ambient
intelligence, where technology moves to the cognitive
background of its users. Data, in particular multimedia
data are being generated, captured and stored in huge
volumes with more being generated every moment. In
such an environment, unique concepts like “media at
the fingertips” and “enhanced-media experiences” [9]
are quite challenging tasks to be analyzed and
implemented, especially when they must adapt the
specific needs of humans in natural and transparent
fashion.
The huge growth of heterogeneous data along
with the contextual data from sensors and usage
environment require specialized handling mechanism.
Particularly any multimedia data needs to be processed
either real-time or in batch processing mode depending
on the situation, and relevant metadata need to be
generated. The generation of multimedia metadata is an
important task that would facilitate identification,
browsing, navigation, retrieval and content
management. Some computing devices automatically
generate metadata when they capture the media. For
example, digital cameras automatically generate EXIF
[11] metadata when a picture is taken. However, these
generated metadata is mostly concerned with the low-
level features of the media. Therefore, new techniques
are needed to add more expressive and semantic
metadata with the media.
In this paper we focus on the different aspects of
multimedia management and state the following
contribution: First, we discuss several issues related to
multimedia management with respect to ambient
intelligence requirements. Second, based on the
requirements analysis of AmI, we propose a multimedia
management framework and present its architecture.
Third, we implemented some of the major components
of the framework. This includes the metadata extraction
service that extracts the embedded metadata from the
multimedia data, allows adding new metadata tags, and
provides a mapping mechanism among heterogeneous
metadata standards to conform to MPEG-7 schema. We
then apply the traditional relation-based query
mechanisms to retrieve the multimedia of interest.
Fourth, we designed and implemented a distributed
proxy-based content repurposing service that converts
multimedia from one format to another in order to
deliver the content to heterogeneous client devices.
2 Requirements of AmI
Research on AmI puts forward several criteria or
challenges for the design of ambient intelligence
environment. These can be summarized as:
• Ubiquitous: offering pervasive computing
and communication with many heterogeneous
networking and computing devices embedded
into everyday objects like furniture, clothing,
white goods, and even toys [10, 16, 20].
• Unobtrusive: the plethora of many distributed
devices required to achieve ubiquity must not
be undesirably noticeable [9, 23].
• Intelligent: enriched with learning and patterns
matching algorithms, speech and gesture
recognition, and situation assessment
capabilities [9, 20].
• Context-aware: such that the system can track
objects and recognize humans and their
situational context [10, 20, 23].
• Personalized: such that the system can
recognize humans and tailor itself to their
needs [10, 23].
• Adaptive: such that the system’s behavior can
change in response to a person’s actions,
surrounding environment and situation [10,
16].
• Anticipatory: such that the system
automatically understands a person’s desires to
varying degrees and proactively offers services
[10, 23].
• Multimodal: interaction capabilities include
voice, gesture, touch, haptic and tactile in
addition to traditional WIMP based techniques
[16, 23].
• Socially aware: special attention must be given
to the interaction of individuals in the AmI
environment to ensure that the collaborative
intelligence can be preserved and reused [20,
21].
• Secure, private, and trustworthy: systems will
inevitably handle private information and must
provide a framework to verify credibility [9,
16, 21, 23].
The progression towards AmI is likely to bring
about new opportunities. However, the above
requirements represent significant research and
development challenges for accessing the computer-
based services and products in a seamless fashion that
conforms to the “anywhere and anytime” vision. The
implementation method of AmI environment would
certainly be a very complex technological endeavor.
3 Multimedia management in AmI
Multimedia management is especially challenging
in ambient intelligence environment, due to the several
requirements of AmI systems described in Section 2,
coupled with the heterogeneity of multimedia resources.
This coupling creates a demand for new ways to
acquire, annotate, browse, retrieve, and deliver media
resources. In the following section we describe the
activities of multimedia management, which are closely
inline with the requirements of traditional knowledge
management [18] and mobility aware knowledge
management [7, 8] practices. Here we elaborate these
activities and its related issues in terms of AmI-specific
requirements.
Multimedia capture/creation: As the ambient
intelligence environment is highly dynamic and
interconnected with sensors, actuators, cameras, and
other computing devices, multimedia data are captured
real-time in different contexts and situations by these
pervasive devices. Since data are coming from
distributed sources, there needs to be a decision making
process in place as to which data to capture and which
to discard. Also, the captured multimedia data are
heterogeneous in nature that requires special mechanism
to address the heterogeneity and map them towards
unification for storage and future retrieval.
Annotation/tagging: The heterogeneous
multimedia data that are captured in AmI environment
or are already present in archives need to be annotated
or tagged in meaningful ways. The different types of
captured/archived media appear in different formats
ranging from simple image clips and texts to complex
video sequences, each conforming to different metadata
standards (e.g. EXIF [11] for image, ID3 [17] for audio
recordings etc.). It is important to mention that in some
media formats, low level media features such as color,
resolution, size and other metadata information are
embedded within the media during its capture.
However, these metadata lacks any semantic
information such as author, usage, identification etc,
unless explicitly tagged. Additionally, on their own,
these data are expressive enough to address the dynamic
interaction and context-based retrieval in AmI
environment. Therefore, in order to manage the
different media in an AmI environment, new tagging
schemes and styles need to be adopted. For example, a
complete video may be tagged with MPEG-7 [6]
manually or semi-automatically and use these tags to
search the video. However, techniques that would
facilitate us to search and retrieve particular video
sequences based on timestamps would be very helpful
[25] in ambient environment.
Storage: Heterogeneous multimedia resources
need to be stored in an appropriate format to be
accessed and utilized by others. One of the biggest
challenges of multimedia management is the data
storage planning, especially when the interacting
environment is like that of an ambient intelligence
environment. The mobile and distributed nature of an
ambient environment calls for delicate and generalized
actions for data management. It is equally important to
analyze and decide whether the media storage need to
be replicated, partitioned, or distributed. Research [27]
in this direction emphasizes the requirements of high-
level data management functionality to be managed by
the distributed middle layer. However, more research is
needed to handle other compelling criteria of AmI such
as ubiquity and context-awareness for the purpose of
managing heterogeneous multimedia data. Some key
research challenges in ubiquitous data management are
described in [19], while specific context-aware aspects
of the same issue are elaborated in [4], all of which
could be leveraged to design data storage service for
multimedia resources.
Sharing: Multimedia management systems, in an
AmI environment, needs to facilitate sharing of
captured and/or archived multimedia among distributed
parties communicating or collaborating on some
common issues. Such sharing of information would
promote enhanced social collaboration in ambient
environment. However, several security and privacy
related issues need to be addressed and resolved to
ensure the integrity of such collaborative practices.
Retrieval: Multimedia retrieval in an AmI
environment is one of the major tasks of the overall
management functionalities. The retrieval operation is
greatly influenced by the surrounding context [24]. The
said context can include various parameters of user’s
physical environment (such as, location, current time,
weather, light level, sound, etc.), their social
environment (such as, surrounding people, social
relations, social position, group relations, etc.), their
psychological situation (such as, anger, happiness,
interactions, etc.), the computational environment (such
as, surrounding devices) and so on [13, 24]. The
information retrieval modules should employ
sophisticated query processing mechanism that can
function considering these diverse set of contextual
parameters in order to provide humans with a
personalized media experience. Other important factors
of multimedia retrieval services include guarantying the
security and privacy of multimedia resources by
ensuring proper access control policy while at the same
time not encumbering users with inconvenient security
policies. Rule-based organization oriented access
control policy [1] may be investigated and extended for
AmI environment. However, we must keep in mind
there are some special issues in terms of mobile devices
in the ambient aware environment, where handhelds and
other portable computing devices may require
immediate and prompt access to services without
lengthy authentication protocols or even explicit user
actions. The emerging biometric identification
techniques [3] may provide a possible solution.
Delivery: The information retrieval process is
followed by the delivery of content to the requesting
client device. Compared to text documents, multimedia
data are large and normally takes more bandwidth in
delivery. The same multimedia data cannot necessarily
be delivered to different client devices the same way
because of differences in client contexts, terminal
capabilities, and networking characteristics. Content
repurposing [2], also known as content adaptation, is
used to resolve this issue by repurposing one media
content into another based on the context and
capabilities of the target environment. For example,
suppose an MPEG-2 movie needs to be delivered to a
client device that is connected with a Bluetooth-enabled
network. Since, the Bluetooth network has bandwidth
constraints, the MPEG-2 movie cannot be delivered to
the client device in this network. Therefore the movie
needs to be adapted for the client device by converting
MPEG-2 movie to some acceptable format such as
MPEG-4, which will meet the bandwidth requirements
of the Bluetooth network.
There are other activities and related issues for
multimedia management in AmI environment such as,
classification, filtering, presentation, and interaction
with the multimedia. The available distributed
multimedia data needs to be classified and filtered. It is
obvious that the classification of multimedia data may
not be distinct due to correlated events in the media
sequences. Therefore, some fuzzy clustering [12]
approach may be used in this respect. Additionally,
clustering or classification (in broader sense) of
multimedia data can be personalized to a specific user
group. The presentation of multimedia in an ambient
environment requires novel approaches. The ambient
display technology [21] shows an intuitive way for this
to be accomplished. However, the visual natural
interfaces and intuitive interaction techniques, in an
ambient environment, need further investigation.
4 Proposed framework
Figure 1 represents a high-level architecture of our
proposed multimedia management framework. Like
many other information and knowledge management
systems, it provides the functionality of media capture,
annotation, storage, sharing, retrieval, and delivery. In
the developed prototype, some of the functionalities are
less emphasized (such as, context capture,
personalization, security and privacy) while others
(such as, metadata extraction, metadata mapping, query
processing, content delivery) are exhaustively designed
and implemented.
Figure 1. Multimedia management system architecture
In this architecture, heterogeneous multimedia
content along with the ambient context (e.g. user’s
surrounding contexts) are captured from the user’s
usage environment. The Media Capturing Agent
coordinates the myriad sensors and other detection
devices to handle the incoming multimedia stream,
user’s location, and user’s interaction. The captured
multimedia is then passed to the Metadata Generator
module for further processing. The metadata generation
is a very complex task as it has to take care of the
situational context and involves complex processing of
the raw media. In our implementation the Metadata
Generator is not yet fully functional.
The Metadata Extractor module extracts the
embedded metadata from the multimedia data that are
generated by the capturing devices. Each captured
media may conform to different metadata standard (e.g.
EXIF, ID3 etc.). However, to provide a unified view of
the heterogeneous media, we map the different metadata
to MPEG-7 schema. This mapping represents one of the
major contributions of our work. The captured
multimedia and metadata are then stored in the
distributed Ambient Database through Database Agent
module. The storage of multimedia data represents a
substantial cost due to the sheer volume of the
multimedia data. Therefore, the system maintains the
metadata in the repository while the actual multimedia
data are stored in distributed file systems.
Context in the ambient environment plays a major
role for providing personalized and adaptive service.
The Ambient Context Handler is used to analyze and
process user’s interaction and usage environment
context. Our current prototype only uses several usage
environment contexts such as network characteristics,
device characteristics and user preferences. However, it
is not very clear how to handle other contextual
information such as user’s social interaction and
psychological situation. In order to consider these social
and psychological contexts in our framework, further
research is required. The Multimedia Retrieval Agent is
responsible to extract information from the repository in
response to user or system level query. The actual query
is processed by the Query Processor module.
Based on the query results the actual multimedia
content is delivered to the requesting client. This task of
multimedia delivery is not straightforward as it needs
mediation between heterogeneous client devices and
heterogeneous media formats. Our contribution in this
respect is that we developed a Content Repurposing
Proxy service that automatically converts one
multimedia format into another. This is needed to
deliver a single format of the media to different client
devices by considering the heterogeneity of the device,
network and user environment characteristics.
The AMI Agent coordinates and controls the
overall activities of the multimedia management
service. It monitors content capture, metadata
generation, metadata extraction, retrieval, delivery, and
other internal processes to ensure the consistency
among the modules. The Security and Privacy Engine is
responsible to guarantee the security and privacy of
multimedia resources and other services by ensuring
proper access control policy. Our current prototype does
not implement the functionality of this security module.
5 Implementation and result
We have implemented a prototype of the proposed
framework. Most of the implementation was done on
J2SE 1.5 platform. However, the content repurposing
services is built using Microsoft Visual C++. In the
following we only present the experimental results
obtained from the multimedia content repurposing
service, which demonstrates the viability of the
proposed multimedia management framework.
In the experiment, Motion JPEG [15], H.263 [26]
and H.264 [5] coding standards are considered for the
multimedia stream. Motion JPEG, creates video
sequences from a series of JPEG images captured from
a webcam. It is an extension (part 3) of ISO/IEC’s
JPEG standard. H.263 is designed for low bit rate video
communication while H.264 support both low bit rate
and high bit rate communication.
0 200 400 600 800 1000 1200
24
26
28
30
32
34
36
38
40
42
Bit Rate (Kbps)
PSNR (dB)
H.264 QCIF
H.264 CIF
MJPEG
Figure 2. Repurposing of MJPEG (CIF) stream into
H.264 CIF and H.264 QCIF stream
As shown in Figure 2, the motion video M-JPEG
(CIF) is encoded at a bit rate of 1.1 Mbps, and is
repurposed into H.264 CIF and H.264 QCIF at variable
bit rate ranging from 56kbps to 1Mbps. Average quality
gain (PSNR) for the repurposed H.264 CIF video
stream is 2.01 dB, while average quality gain for the
repurposed H.264 QCIF video stream is 1.1 dB. At very
high resolutions and high bit-rates, Motion-JPEG
outperforms H.264 in terms of average PSNR over
average bit-rate.
As shown in Figure 3, MJPEG (CIF) stream at 1.1
Mbps and at 30 fps is repurposed into the H.264 QCIF
stream at 64 Kbps and at 10 fps, which is appropriate
for low bandwidth video communications. This
repurposing involves conversion of coding standard and
frame resolution, and frame rate conversion. The
repurposed stream (H.264) outperforms the encoded
MJPEG by an average gain of 3.3 dB as shown in Fig 5.
020 40 60 80 100 120
28
30
32
34
36
38
40
42
44
46
No of Frames
PSNR (dB)
MJPEG
H.264
Figure 3. Quality comparison of repurposed H.264
stream and encoded MJPEG stream
6 Conclusions
The management of multimedia in ambient
intelligence environment is a very challenging task. We
discussed several issues of multimedia management
including media capture, annotation, storage, sharing,
retrieval, and delivery with respect to ambient
intelligence requirements. We also presented a
framework of multimedia management and provided
architecture of the framework. We further elaborated
the tasks of the individual module of the framework.
We implemented a working prototype of the
framework. We demonstrated different media
adaptation process, which is an integral part of an
ambient intelligence environment in order to deliver
multimedia content to heterogeneous client devices.
Future aspects of this research will include automatic
metadata generation, extended context analysis,
security, and privacy issues.
References
[1] A.A. El Kalam et al., “Organization Based Access
Control (Or-BAC)”, IEEE 4th International
Workshop on Policies for Distributed Systems and
Networks (Policy 2003), Italy, June 4-6, 2003.
[2] A. El Saddik and M.S. Hossain, “Multimedia
content repurposing”, Encyclopedia of Multimedia,
B. Furht, ed., Springer Book Series, 2006.
[3] A. El Saddik et al., “A novel biometric system for
identification and verification of haptic users”,
IEEE Transactions on Instrumentation and
Measurement (accepted, to appear).
[4] A.H. van Bunningen, L. Feng, and P.M.G. Apers,
“Context for ubiquitous data management”, In:
International Workshop on ubiquitous data
management (UDM2005), pp. 17-24, 2005.
[5] “Advanced video coding for generic audiovisual
services,” ITU-T Recommendation H.264, March
2005.
[6] B.S. Manjunath, P. Salembier, and T. Sikora, eds.,
Introduction to MPEG-7: multimedia content
description language, John Wiley & Sons, New
York, 2002.
[7] C. Valle et al, “MILK Mobile Support for
Knowledge Management”, CAiSE Short Paper
Proceedings, 2003.
[8] D. Balfanz, M. Grimm, and M. Tazari, “A reference
architecture for mobile knowledge management”,
N. Davies, T. Kirste, and H. Schumann, eds.,
Mobile Computing and Ambient Intelligence: The
Challenge of Multimedia, IBFI, Schloss Dagstuhl,
Germany, 2005.
[9] E. Aarts, “Ambient intelligence: A multimedia
perspective”, IEEE Multimedia, Vol. 11, No. 1, pp.
12-19, Jan.–Mar., 2004.
[10] E. Aarts and S. Marzano, eds., “The new everyday:
vision of ambient Intelligence”, 010 Publishing,
Rotterdam, The Netherlands, 2003.
[11] Exchangeable image file format,
http://www.exif.org/specifications.html
[12] F. Höppner et al., Fuzzy Cluster Analysis, Wiley,
Chichester, 1999.
[13] G. D. Abowd and E.D. Mynatt, “Charting past,
present, and future research in ubiquitous
computing”, ACM Transactions on Computer-
Human Interaction, Vol. 7, No. 1, pp. 29-58, 2000.
[14] IST AMIGO project. Available at:
http://www.hitech-projects.com/euprojects/amigo/.
[15] J. Pons, M.P. Malumbres, and R. García, “Scaled
MJPEG: A symmetric video compression method
for low bit-rates”, IASTED/ACTA Press (AI'2000),
pp. 302-308, 2000.
[16] K. Ducatel et al., “Scenarios for Ambient
Intelligence in 2010”, IST Advisory Group Final
Report, Seville, 2001. Available at:
ftp://ftp.cordis.lu/pub/ist/docs/istagscenarios2010.pdf.
[17] MP3 metadata, http://www.id3.org/index.html
[18] M. Alavi and D.E. Leidner, “Knowledge
management systems: issues, challenges, and
benefits”, Commun. AIS, Vol. 1, No. 2es, Feb.
1999.
[19] M. Franklin, “Challenges in ubiquitous data
management”, Informatics, pp. 24-33, 2001.
[20] N. Shadbolt, “Ambient intelligence”, IEEE
Intelligent Systems, Vol. 18, No. 4, pp. 2-3,
Jul./Aug. 2003.
[21] N. Streitz, C. Magerkurth, T. Prante, and C. Röcker,
“From information design to experience design:
smart artefacts and the disappearing computer”,
Interactions, Vol. 12, No. 4, pp. 21-25, Jul. 2005.
[22] Philip’s Home Lab. Available at:
http://www.research.philips.com/technologies/misc/
homelab/.
[23]
P.L. Emiliani and C. Stephanidis, “Universal access
to ambient intelligence environments: opportunities
and challenges for people with disabilities”, IBM
Systems Journal, Vol. 44, No. 3, pp. 605-620, 2005.
[24] P.J. Brown and G.J.F. Jones, “Context-aware
retrieval: exploring a new environment for
information retrieval and information filtering”,
Personal and Ubiquitous Computing, Vol. 5, No. 4,
pp. 253-263, Dec. 2001.
[25] R. Jain, “Tagging videos”, online blog,
http://ngs.ics.uci.edu/blog/?p=585 (accessed, Aug.
8, 2006).
[26] “Video Coding for Low Bitrate Communication”,
ITU-T Recommendations H.263, April 2005.
[27] W. Fontijn, J. Nesvadba, and A. Sinitsyn,
“Integrating media management towards ambient
intelligence”, Adaptive Multimedia Retrieval, pp.
102-111, 2005.
