Advances in Ambient Intelligence [1 ed.] 9781607502876, 9781586038007

Citation preview

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

ADVANCES IN AMBIENT INTELLIGENCE

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Frontiers in Artificial Intelligence and Applications FAIA covers all aspects of theoretical and applied artificial intelligence research in the form of monographs, doctoral dissertations, textbooks, handbooks and proceedings volumes. The FAIA series contains several sub-series, including “Information Modelling and Knowledge Bases” and “Knowledge-Based Intelligent Engineering Systems”. It also includes the biennial ECAI, the European Conference on Artificial Intelligence, proceedings volumes, and other ECCAI – the European Coordinating Committee on Artificial Intelligence – sponsored publications. An editorial panel of internationally well-known scholars is appointed to provide a high quality selection. Series Editors: J. Breuker, R. Dieng-Kuntz, N. Guarino, J.N. Kok, J. Liu, R. López de Mántaras, R. Mizoguchi, M. Musen, S.K. Pal and N. Zhong

Volume 164

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Recently published in this series Vol. 163. C. Angulo and L. Godo (Eds.), Artificial Intelligence Research and Development Vol. 162. T. Hirashima et al. (Eds.), Supporting Learning Flow Through Integrative Technologies Vol. 161. H. Fujita and D. Pisanelli (Eds.), New Trends in Software Methodologies, Tools and Techniques Vol. 160. I. Maglogiannis et al. (Eds.), Emerging Artificial Intelligence Applications in Computer Engineering – Real World AI Systems with Applications in eHealth, HCI, Information Retrieval and Pervasive Technologies Vol. 159. E. Tyugu, Algorithms and Architectures of Artificial Intelligence Vol. 158. R. Luckin et al. (Eds.), Artificial Intelligence in Education – Building Technology Rich Learning Contexts That Work Vol. 157. B. Goertzel and P. Wang (Eds.), Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms – Proceedings of the AGI Workshop 2006 Vol. 156. R.M. Colomb, Ontology and the Semantic Web Vol. 155. O. Vasilecas et al. (Eds.), Databases and Information Systems IV – Selected Papers from the Seventh International Baltic Conference DB&IS’2006 Vol. 154. M. Duží et al. (Eds.), Information Modelling and Knowledge Bases XVIII Vol. 153. Y. Vogiazou, Design for Emergence – Collaborative Social Play with Online and Location-Based Media Vol. 152. T.M. van Engers (Ed.), Legal Knowledge and Information Systems – JURIX 2006: The Nineteenth Annual Conference

ISSN 0922-6389

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Advances in Ambient Intelligence

Edited by

Juan Carlos Augusto School of Computing and Mathematics, University of Ulster at Jordanstown, UK

and

Daniel Shapiro

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Applied Reactivity Inc., USA

Amsterdam • Berlin • Oxford • Tokyo • Washington, DC

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

© 2007 The authors and IOS Press. All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without prior written permission from the publisher. ISBN 978-1-58603-800-7 Library of Congress Control Number: 2007938062 Publisher IOS Press Nieuwe Hemweg 6B 1013 BG Amsterdam Netherlands fax: +31 20 687 0019 e-mail: [email protected]

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Distributor in the UK and Ireland Gazelle Books Services Ltd. White Cross Mills Hightown Lancaster LA1 4XS United Kingdom fax: +44 1524 63232 e-mail: [email protected]

Distributor in the USA and Canada IOS Press, Inc. 4502 Rachael Manor Drive Fairfax, VA 22032 USA fax: +1 703 323 3668 e-mail: [email protected]

LEGAL NOTICE The publisher is not responsible for the use which might be made of the following information. PRINTED IN THE NETHERLANDS

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Advances in Ambient Intelligence J.C. Augusto and D. Shapiro (Eds.) IOS Press, 2007 © 2007 The authors and IOS Press. All rights reserved.

v

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

AmI revs up Since the early days of Ambient Intelligence I have been following at close range and with great curiosity the various developments and achievements that have been reported in the open literature on this novel paradigm for the disappearing computer. To obtain a proper feeling for the speed at which the AmI concept has been developing itself, I have been monitoring its Web presence over the past five years by registering on a monthly basis the number of hits Google returns upon the controlled search for the key words “Ambient Intelligence”. After a wavering start with a grand total of only a few hundred hits the number ramped up almost exponentially for a number of consecutive years. Since the beginning of 2006 the growth started to slow down to follow a more or less linear increase leading to current value of about 550,000 hits. The incredible speedup of the AmI Web presence can be attributed to the publication of the seminal document by the Society and Technology Advisory Group’s (ISTAG). It explains the AmI concept on the basis of four AmI scenarios and through this remarkably simple approach the issue attracted an enormous amount of attention. Furthermore, the European Commission follow up on the ISTAG research recommendations by launching the sixth Frame Work Programme in IST using the concept of Ambient Intelligence as its central research theme. This obviously has largely contributed to the enormous growth in the number of studies into the AmI concept. Evidently, the evolution in the number of Google hits reflects the typical development of a novel paradigm such as Ambient Intelligence. After a conception phase in which the concept is being shaped by a small community of truly convicted early adopters it spreads rapidly to become a leading theme for a large and broad audience of researchers. After the hype the growth slows down again to normal proportions and this is also the phase in which the concept is judged on its true merits. It needs reflection to get rooted; it needs criticism to separate the wheat from the chaff, and it needs reinvention to become robust. It simply needs time to become truly meaningful. To a certain extent this also can be viewed as an existential phase as it reveals whether there is any substance to the concept at all and whether it is truly capable of inspiring researchers to come up with new ideas and glue them together to become a fully integrated disruptive computing paradigm that is meaningful to society providing its participants with maximum benefit. The Second Workshop on Artificial Intelligence Techniques for Ambient Intelligence (AITAmI’07) has contributed to this purification process as it provided a critical forum for the discussion of the recent developments in the area. The present volume, which resulted from the workshop, does even more so contribute, as the editors have succeeded in compiling a most remarkable collection of chapters. Some chapters are extended version of the best papers presented at the workshop, others are high-quality invited contributions that provide critical reviews of specific developments. The chapters reflect a high scholarship and expertise of the contributing authors and the book is definitely a mandatory reading for anyone who is professionally active in this promising field of Ambient Intelligence.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

vi

It goes without saying that it is my strong belief that Ambient Intelligence will be become an influential new computing paradigm and I am confident that the present volume will contribute to the steady growth of the number of AmI Google hits as many other researchers will feel the need to refer to many of its most inspiring chapters. Eindhoven The Netherlands August 2007

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Emile Aarts Scientific Program Manager Philips Research

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

vii

Introduction

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Ambient Intelligence [1–4] concerns the development of digital environments that ‘... proactively, but sensibly, support people in their daily lives’ [5]. The field lies at the confluence of several trends: the continued decrease in cost and size of computing technology; the increasing availability of networking and communication infrastructure, the growing public familiarity/comfort with computing artifacts, and practical advances in artificial intelligence. These developments make it possible to contemplate the ubiquitous deployment of intelligent systems – prototypically in smart homes, but more broadly in public spaces, private automobiles, and on individual appliances and handheld devices – in applications ranging from entertainment through eldercare, to safety critical device control. Ambient Intelligence is a young field. As a result, it has been natural to wonder what the technology can do to improve the way we live. At the same time, it is becoming increasingly important to ask, “What do we want?” since the intent is to embed technology in new and pervasive ways. This volume compiles contributions closely related to those presented at the workshop AITAmI’07 [6–8]. These contributions were especially selected because they address both questions by providing a window into the visions and trends currently dominating the area of Ambient Intelligence. We have organized the volume into three sections. The first describes visions for the future of Ambient Intelligence, the second addresses core technology of the field, and the third provides an analysis of elements of the area which will demand special consideration during the future development of the area. First we present two vision pieces: Cyber Assist Project for Ambient Intelligence by H. Nakayima (Future University – Hakodate, Japan) focuses on the importance for systems and people to share a common semantics in order to understand each other and as a condition for AmI systems to be useful. Ambient Intelligence – From Personal Assistance to Intelligent Megacities by M. Berger, F. Fuchs, M. Pirker (Siemens) describes what is possible at a very broad level systems by considering AmI scenarios of increasing complexity from an engineering perspective. Next, we offer five articles reporting significant technical advances and illustrating key technology underlying Ambient Intelligence: Towards a Context Theory for Context-Aware Systems by R. Dapoigny and P. Barlatier (U. de Savoie – France) provides a formalization of contexts and their relationships and dependencies. An Active Classification System for Context Representation and Acquisition by F. Mastrogiovanni, A. Sgorbissa, and R. Zaccaria (U. of Genova-Italy) describes a multi-agent system with integration of numerical and symbolic reasoning systems that can effectively reason about situations and contexts.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

viii

Ambiance: A Mobile Agent Platform for End-User Programmable Ambient Systems by R. Razavi, K. Mechitov, G. Agha, and J-F. Perrot (U. of Luxembourg, UIUC – USA, U. Pierre et Marie Curie – France) explores new and better methods to engineer systems which have to be deployed over a wireless sensor network. A Human Activity Aware Learning Mobile Music Player by S. Dornbush, A. Joshi, Z. Segall and T. Oates (U. of Maryland – USA) reports on their experience of implementing a musical device that can learn from its user’s preferences and daily use. Two Issues for an Ambient Reminding System: Context-Awareness and User Feedback by N. Richard and S. Yamada (National Institute of Informatics – Japan) highlight the importance of AmI systems being sensitives to the users context and taking into account preferences, current occupations and other parameters in the design of a reminder system. We conclude with two articles outlining elements that need further consideration and evolution within the area: More Help From A Stranger (Media Arts In Ambient Intelligence Research) by M. Bohlen (Department of Media Study, SUNY at Buffalo, USA and University of Zürich, Switzerland) provides an original multidisciplinary view over a fundamental issue for AmI systems: how people perceives and reacts to artificial systems.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Run to the Hills! Ubiquitous Computing Meltdown by P. Huuskonen (Nokia) enriches our view of the area by making an objective and critical but still positive analysis of the implications that AmI systems can bring to our world if some current engineering trends are not rectified. Taken together, these articles offer a snapshot of the state of the art in Ambient Intelligence, and a window into its probable future directions. They paint a picture that promises exciting technology and powerful applications, while raising important questions about the social consequences of their deployment. Researches can influence that future. As a final word we would like to recognize and thank the people that collaborated on this volume throughout the publication process. This involved referees giving their feedback on earlier versions of this chapters before AITAmI’07 and during the edition of this book. Our sincere gratitude also to the IOS Press team for believing in this project, and supporting it throughout all its stages. Equally vital is the positive attitude that all the authors had throughout the writing of successive drafts of those chapters included towards achieving a high quality compendium of forward thinking contributions to the literature of the area. References [1] “IST Advisory Group. The european union report, scenarios for ambient intelligence in 2010”, 2001. URL: ftp://ftp.cordis.lu/pub/ist/docs/istagscenarios2010.pdf. [2] E. Aarts, R. Harwig, and M. Schuurmans. Ambient intelligence. In The Invisible Future, Peter Denning, Ed. Pages 235-250. McGraw-Hill, 2002. [3] D.J. Cook and S. Das (Eds.). “Smart Environments: Technologies, Protocols and Applications”. JohnWiley and Sons, 2004.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

ix

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

[4] J.C. Augusto and D. Cook. “Ambient Intelligence: applications in society and opportunities for AI”. Lecture Notes for the tutorial given during 20th International Joint Conference on Artificial Intelligence (IJCAI 2007). January 2007. Hyderabad, India. [5] J.C. Augusto. “Ambient Intelligence: the Confluence of Ubiquitous/Pervasive Computing and Artificial Intelligence.” In Intelligent Computing Everywhere. Alfons Schuster (Ed.). Springer Verlag, 2007. [6] J.C. Augusto and D. Shapiro (Eds.). “Proceedings of the 1st Workshop on Artificial Intelligence Techniques for Ambient Intelligence (AITAmI’2006)”. Riva del Garda, Italy. 29th August 2006. Co-located event of ECAI 2006. Printed by ECCAI. [7] J.C. Augusto and D. Shapiro (Eds.). “Proceedings of the 2nd Workshop on Artificial Intelligence Techniques for Ambient Intelligence (AITAmI’2007)”. Hyderabad, India. 6–7th January 2007. Colocated event of IJCAI 2007. Printed by IJCAI. [8] J.C. Augusto and D. Shapiro. “The First Workshop on Artificial Intelligence Techniques for Ambient Intelligence (AITAmI ’06)”, AI Magazine 28(1): Spring 2007, 86–87, AAAI Press.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

This page intentionally left blank

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

xi

Contents AmI Revs Up Emile Aarts Introduction Juan Carlos Augusto and Daniel Shapiro

vii

Cyber Assist Project for Ambient Intelligence Hideyuki Nakashima

1

Ambient Intelligence – From Personal Assistance to Intelligent Megacities Michael Berger, Florian Fuchs and Michael Pirker

21

Towards a Context Theory for Context-Aware Systems Richard Dapoigny and Patrick Barlatier

36

An Active Classification System for Context Representation and Acquisition Fulvio Mastrogiovanni, Antonio Sgorbissa and Renato Zaccaria

56

Ambiance: A Mobile Agent Platform for End-User Programmable Ambient Systems Reza Razavi, Kirill Mechitov, Gul Agha and Jean-François Perrot A Human Activity Aware Learning Mobile Music Player Sandor Dornbush, Anupam Joshi, Zary Segall and Tim Oates Two Issues for an Ambient Reminding System: Context-Awareness and User Feedback Nadine Richard and Seiji Yamada Copyright © 2007. IOS Press, Incorporated. All rights reserved.

v

81 107

123

Help from Strangers – Media Arts in Ambient Intelligence Research Marc Böhlen

143

Run to the Hills! Ubiquitous Computing Meltdown Pertti Huuskonen

157

Author Index

173

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

This page intentionally left blank

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Advances in Ambient Intelligence J.C. Augusto and D. Shapiro (Eds.) IOS Press, 2007 © 2007 The authors and IOS Press. All rights reserved.

1

Cyber Assist Project for Ambient Intelligence Hideyuki NAKASHIMA Future University - Hakodate, Hakodate, Hokkaido 041-8655 Japan Abstract. Information processing technologies should be able to support people in every aspect of their everyday life, with information processing units embedded in the environment which communicate with portable or wearable personal devices. The keywords are "here, now and me" rather than "wherever, whenever, whoever". We want to add one more element to the vision, the concept of situatedness. Ubiquitous computing and ambient intelligence put stress on having sensors in the environment to sense the context of the user. However, I define situation not as something to be sensed or computed, but something to be used as it is. We should not design a system that acts on the sense-represent-compute-act cycle. Rather, we should design a system that can set up the context appropriately. Cyber Assist project was conducted under the above philosophy. In this chapter, we overview the project under the light of ambient intelligence. Keywords. Cyber Assist, ubiquitous computing, Ambient Intelligence, situatedness

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Introduction In the information-oriented society, anyone should be able to utilize information processing technology (IPT) anywhere in the world at any time. IPT should not be confused with information communication technology (ICT). In ICT, computers are used to relay information from human to human. The typical examples of ICT include telephone networks, TV and radio broadcasting, and the Internet. In these examples, the sender and receiver of information are human. Computers should not change the information content. On the other hand, when we talk about IPT, computers should process the content of information. When we talk about ambient intelligence, IPT must be the focus and ICT is the vehicle on which IPT is implemented. Traditional IPT was accessible only through computers sitting on the desktop, or worse, in the computer room. Its accessibility is broadening recently with the spread of mobile devices such as PDA and cellular phones that are capable of accessing networks. With the keywords of “ubiquitous computing” and/or “ubiquitous networks” future life scenes are visualized by many IT research labs. Nevertheless, in reality, only a small fraction of people enjoys such technologies in rather few scenes of their daily life. IPT should be able to support each of us in every aspect of our everyday life, with information processing units embedded in the environment which communicate with portable or wearable personal devices. The keywords are "here, now and me" rather than "wherever, whenever, whoever". IPT should help me by automatically booking a seat in a train ac-

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

2

H. Nakashima / Cyber Assist Project for Ambient Intelligence

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

cording to my schedule, by guiding me through a shopping mall while providing necessary information about commodities and services available here and now, or by automatically calling a taxi or requesting a dial-a-ride bus (see section 3.5.1) service if I needed it now. This technology can boost the quality of life in all respects including economy, politics, culture, and education. For IPT to provide meaningful and valuable assistance to people, computers should share the semantics of the tasks with humans. We must pull computers out of their digital world into our physical world by providing various sensors and visual and acoustic recognition technologies. For instance, if your mobile phone can understand that you have a visitor and are discussing an important issue, it may be able to automatically forward incoming phone calls to your secretary. This kind of new requirements affects the direction of research and development. The direction of the technology used for ubiquitous computing and ambient intelligence (AmIhereafter) may be the opposite of those required for global networking such as the Internet. The network architecture of Internet does not fit our needs because it is designed to connect computers throughout the world in a rather fixed topology. The direction is toward high bandwidth and high throughput communication. For AmI, we have to put our focus on local but rich communications. For example, we need a "responsive" network with dynamic reconfiguration and connection with very low latency. As we describe later (for example, in section 3.1), we sometimes have to manipulate physical (in contrast to digital) information. We launched Cyber Assist Project aiming at the construction of a system to provide situated information that I-want-here-now through natural interface. The rest of this chapter is organized as follows. We first describe the overall concept of Cyber Assist project in section 1. In section 2, the design philosophy of AmIis described. In section 3, various research themes tackled by CARC are presented, followed by their applications to the real world in section 4.

1. Overview of Cyber Assist Project Cyber Assist Research Center (CARC) was established in April 2001, as one of the fifty research units of AIST, National Institute of Advanced Industrial Science and Technology. The primary mission of the research centers of AIST is to apply research results to the real world problems for industrial and other practical purposes. We had to cover research areas from the bottom to the top, because the other communication technologies could not be used as they are, as we just described. CARC is organized as in Figure 1. However, it is not a linear bottom-up/top-down structure. Each research component on the circumference is a means to another component, and they are interconnected with each other. Content is what the user is primarily interested in. However, row content may be hard to interpret. Good architecture to support its conversion is necessary. Good interface to present the content to user is also important. The interface cannot be designed independently from device design. Device cannot be operative without good software, and the operation of the software depends on the service architecture. At the same time, we had to attack many new problems. New kind of problems arose as information accesses become ubiquitous. The first one is “information overload”. We

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

H. Nakashima / Cyber Assist Project for Ambient Intelligence

3

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 1. Organization of Cyber Assist Research Center

need some technology to help us access only information relevant to our needs. The second one is “digital divide”. A person has to learn a lot to use IT to get proper support. A major cause of information overload and digital divide is the semantic gap between humans and computers; humans are adapted to dealing with deep meaning, while machines excel at processing formal syntax. The only feasible way to fill this gap systematically is to make the semantic structure of information content explicit so that machines can manage them too. Web content processing with ontology and semantic retrieval is proposed in the Semantic Web project. In this connection, it is essential that a huge number of people contribute their knowledge in machine-understandable form to the socially shared knowledge base, in order to provide ubiquitous information services in every situation of our everyday life. The intelligent-content technologies support this collaborative creation and maintenance of the huge common intellectual property. To this end, we worked on semantic authoring, by which people can create better content more easily than when using traditional authoring-support tools such as word processors and outline processors that manipulate only surface structure of documents. This semantic authoring system uses two authoring interfaces: the idea-processor window through which to make unordered, networkstyle, content, and the word-processor window through which to manipulate ordered, tree-style, content. The third problem is “privacy protection”[18]. In the society where everyone is connected to the network all the time, private data are easy to access and hard to protect. Moreover, on-line data are easy to copy and distribute. There are laws against abuse of on-line private data, but technological support for enforcing those laws is very limited. Encryption does not protect data from abuse of the server itself that manipulates the data. Our goal is to develop human-centered information assistance systems (intelligence boosters) usable without professional knowledge or training, while addressing the problems of information overload and privacy.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

4

H. Nakashima / Cyber Assist Project for Ambient Intelligence

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

2. Design Philosophy of Ambient Intelligence Ubiquitous computing[33] and MIT’s Oxygen project[3] among many others propose similar concepts to ambient intelligence[29]. A typical scenario uses a living room or a conference room. Those rooms are equipped with many kinds of sensors, including cameras and microphones, to sense behavior of people in the room. Sensed information usually goes through heavy computation (is this the part called “intelligence”?) to decide the proper action of the system. For example, when a certain person enters the living room and sits on a sofa, his location is sensed by cameras and/or weight sensor of the sofa, and then his favorite music is played from the proper speakers. When he points to a TV screen, the direction of his finger is computed from the camera image and TV is tuned on. The system also keeps track of an individual and delivers message when contacted from outside. In a sense, the ultimate goal of those projects is to set up a computer supported environment such as seen in Star Trek movies. Long history of Artificial Intelligence tells us that this approach would never become perfect. The problem is called “the frame problem”[7,2] and it points out that representational model of a person and the surrounding environment cannot be complete. There always a possibility of misrepresentation or mis-inference. We want to suggest a slightly different approach. It is the concept of situatedness (in the sense of [1]). Ubiquitous computing and AmIput stress on having sensors in the environment to sense the context of the user. However, I define situation not as something to be sensed or computed, but something to be used as it is. We should not design a system that acts on the sense-represent-compute-act cycle. Rather, we should design a system that can set up the context appropriately. In other words, the system should better have the ability to set up an environment so that natural behavior of a person enables a natural transfer of information. We need a system to be able to couple with the person to form an autopoietic[16] system, or in other words, to provide affordance[6] to the users[27]. For example, instead of computing people’s location and give different service according to it, we could set up the environment so that the proper service is receivable only in the area. I will come back to this issue later with a concrete example (section 4.3). In a sense, sense-represent-compute-act model is of the Western view. The system observes the users as scientists observe natural phenomena. The system designer is detached from the system in action or the user. Let us call this “birds-eyes-view”. Another possibility is “agents-view” where the system designer is one component of the system. Eastern view of the world is also this agent-view. There is an interesting experiment to highlight the differences in the view between Western and Eastern cultures. The following example illustrates the difference of views embedded in linguistic expressions between English and Japanese. A sentence is taken from a famous Japanese novel “Snow country” by Yasunari Kawabata, who is a winner of the 1968 Nobel Prize in Literature. English translation of its first sentence by E. Seidensticker is “The train came out of the long tunnel into the snow country.” original Japanese sentence is something like “When went through the long tunnel, * was (in) the snow country.” * has no corresponding element in Japanese sentence. It could be “it”, “the train” or “I”. Most of readers of Japanese sentence are reported to image something like the lefthand side of Figure 2, while readers of English sentence are reported to image something

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

H. Nakashima / Cyber Assist Project for Ambient Intelligence

5

Figure 2. The Scene from Agent-View (left) and Bird-View (right)

like the right-hand side of Figure2. Natural expression in Japanese takes agent-eye view. Natural expression in English, on the other hand, takes bird-eye view. For the system to share the situation with the user, the designer and the system must take the agent view, in contrast to birds eyes view. In other words, having agent view is the key for supporting “enactivity”[32] of the user.

3. Research Themes

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

3.1. Location-based Communication CARC developed new communication methodology based on location. Location plays important roles in information processing technology (IPT). It can act as a glue to integrate/amalgamate IPT and the real physical world through situated communication[20]. The proposed mechanism uses the physical locations as the source/destination addresses of message delivery. The two presently major means of communication, the Internet and phone network, use global ID’s, IP addresses and phone numbers, as the target address. With the use of globally unique ID, the target can be identified throughout the world to realize global communication. On the other hand, since the ID is unique, the recipient can easily be identified. This produces privacy hazard. If we use physical location as the address of service delivery, we can reduce the possibility of identification of the recipient. We developed a compact battery-less information terminal (CoBIT) system (section 3.2) to implement our idea. CoBIT uses infrared light beam for communication. Information is delivered to a specific area of the 3D space. Voice is the major modality for downloading information to a CoBIT terminal. Vision is used for uploading information. IR camera or IR code detector in the environment receives information of the user. We are also trying to apply the idea of location-based communication to wireless (RF) networks. UBKit (section 3.3) is a set of software tools for building a middleware system in ubiquitous computing environment. Wireless LAN as well as the abovementioned CoBIT system is within the scope of UBKit. Ad-hoc network over wireless LAN usually uses (temporally issued) IP addresses for communication, but we are investigating possibilities of using location instead.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

6

H. Nakashima / Cyber Assist Project for Ambient Intelligence

Figure 3. The original CoBIT

Figure 4. CoBIT with RFID

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

3.2. Device We have developed CoBIT, Compact Battery-less Information Terminal [21] as the first example of location-based communication (section 3.1). The first CoBIT[22] is composed of an earphone, which is directly connected with a solar cell, and a cornerreflecting sheet (Figure 3). User of the CoBIT receives voice or sound information through the earphone via amplitude modulated (AM) infrared light. The corner-reflecting sheet is used to detect the location and/or movement, or to classify the type of device without further communication. Features of the CoBIT system are lightweight and compact dimensions, low cost of the manufacturing and the maintenance of the device. We also developed the reflectivity modulating technique to upload the data from the device to the environment with much lower power consumption[11]. The reflectivitymodulating module has been installed in two types of PDA’s, and those characteristics were evaluated. Various types of liquid crystal (LC) light modulators, a polymer dispersed type, a polymer networked type, and a ferroelectric type, have been specially developed for lower operational voltage or higher speed operation. Using the developed LC modulator with a low powered processor (PIC), we developed an ID or a data-launching CoBIT, which operates without a battery (Figure 4). Base station: We have been developing several types of the base station, named ilidar, which locates, traces, and communicates with My Buttons. First prototype of the station had used a single infrared laser beam. The location of My Button was calculated from its direction and the distance. The direction is obtained by angles of galvanometer mirrors, and the distance is obtained by measurement of beat frequency from chirped

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

H. Nakashima / Cyber Assist Project for Ambient Intelligence

7

real world voice

infrared

UBKit

SOAP

middleware

Microservers

IEEE802.11

positioning

DNS Bluetooth

network Figure 5. UBKit

laser beam. The system had high accuracy of location (5 sec.) To improve the finding time (10 bps), reflectivity modulated signal detection using an infrared LED illuminator and CCD camera has been used for the second prototype. To improve the uploading speed and to reduce the dimensions of the beam scanner, a machine-vision camera (1000 frames/sec.) and electromagnetically driven MEMS beam-scanning mirror was adopted for the third prototype of i-lidar.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

3.3. Software Software team has been focusing on construction of software tools for the middleware, namely the UBKit. The UBKit (Ubiquity Building Toolkit)[19] is a set of software/hardware components to be used as modular building blocks for ubiquitous computing in the same spirit as “LEGO” blocks. It aims at “off the shelf” development of embedded wireless computer networks and also at a handy platform for programming the real world using lightweight middle-ware and sensor network gateways. Some components are newly developed in collaboration with companies such as Mitsubishi Electric Cooperation and Nihon Unisys, Ltd.; some are adopted from existing open-source software. The UBKit has been carefully assembled addressing scalability, interoperability and ease of deployment so that a large number of heterogeneous devices and services can work together. In technical terms, we regard the following aspects as essential in popularizing ubiquitous computing in the cyber-assisted environment we envision: • A zero configuration network, where IP address and related parameters are configured without manual operation or the help of special (such as DHCP) servers. • An automatic service discovery through service registry/lookup mechanisms, in which requests from humans or computers are automatically directed to a nearby component that can provide the service. • A real world user interface, by which devices and services are accessed by ordinary names (such as “printer” and “TV”) and locations. • A context sensitive user support, where the user’s location, preferences, and activity records are used to identity tasks to be done.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

8

H. Nakashima / Cyber Assist Project for Ambient Intelligence

The above observations have led us to include the following components in the toolbox (Figure 5). • Microservers: We have tiny and power-efficient microcomputers running Linux on 32bit RISC microprocessors developed at Mitsubishi Electric Co. A microserver may have interfaces such as CF (mainly for networking), SD (mainly for storage), USB, infrared, and DSP. The microprocessor part is under continuous development while the current implementation has 300 MHz clock speed. Microservers are used to host local content and may be embedded in various devices to make them network-aware and remotely controllable. • Sensor network and its gateway to IP network: Sensor networks are used to get real world data. We have experimental use of MICA Motes [8] developed at UC Berkeley in the Smart Dust project. The main module has a microprocessor and an RF chip; the sensor module has the sensors for light, temperature, magnetism, acceleration, and sound. As the role of sensor networks is constant monitoring, a special low-power radio communication is usually in use. Therefore, gateways between sensor networks and computer networks are needed; their task is to monitor incoming sensor data and invoke hook operations when trigger events are detected. • Middle-ware: To have applications working over heterogeneous embedded networks requires lightweight mechanisms of service directory and remote procedure invocation. We use the multicast DNS (domain name service) service discovery for the former functionality and the web-based technologies for the latter such as CGI for simple server-side processing and SOAP (simple object access protocol) for network wide distributed processing.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

3.4. Multiagent Architecture for Ubiquitous Computing To control the environmental systems, we need a multiagent architecture (for example, see [5]). One of our major contributions to ubiquitous computing is the introduction of multiagent architecture. The ubiquitous computing technology usually regards embedded devices but fails to consider the software architecture for human support on top of those devices. Without a proper architecture for information integration and control, the whole environment will be a mess of uncontrolled local communications; each device will be accessed by many services without arbitration; and too much irrelevant or un-intelligible information will be delivered to people. We proposed a multiagent architecture CONSORTS[30] to facilitate ubiquitous flow of information among various devices and users. The architecture is shown in Figure 6. The service agents provide information services with using information on users’ situation such as the distance of a user and a display and as the direction of a user’s face. Services provided in CONSORTS museum scenario is shown in Figure 7. The position of the users is captured by sensory network and a spatio-temporal reasoner manages geometrical relations among users and displays. When certain condition on the geometrical relations (e.g., a user gets into the visible area of a display) is satisfied, the spatialreasoner sends a message about the condition to related service agents. The service agent that received the message provides suitable services to the user.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

H. Nakashima / Cyber Assist Project for Ambient Intelligence

9

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 6. CONSORTS

Figure 7. Image of Museum Guidance System

3.5. Multiagent Simulation The concept of AmIshould not be restricted within a room or a building. With maturation of ubiquitous computing technology, it became feasible to design new systems to improve our urban life. However, unlike our laboratory environment where only small numbers of parameters are relevant and they can be controlled, the real world consists of many parameters and some of them are not controllable. Multiagent simulation plays an important role to design such real world application. We can investigate many different

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

10

H. Nakashima / Cyber Assist Project for Ambient Intelligence Changes of Ave. Score by Demand Frequency 5.5 5 4.5 4

ave. score

3.5 3 2.5 2 fixed path full demand

1.5 1 0.5 0

50

100

150

200

250

300

demand frequency

Figure 8. Simulation with Fixed Number of Buses

parameter settings through simulation. We will see the power of simulation in section 3.5.1. Moreover, there are many cases where simulation plays an essential role not only in designing a system but also in real-time control of the ambient intelligence. When a (multiagent) simulation is coupled with the real data from sensor networks, it can predict the status of the near future, in contrast to the status of the near past or of the current moment. The weather forecast is a good example. We will see the power of simulating the future in sections 3.5.2 and 3.5.3.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

3.5.1. Dial-a-Ride Bus We had an interesting experience on power of large scale simulation. In Japan, it became popular to see so called a “demand bus” that detours from prefixed route upon request from passengers. Even a full version of the concept, a bus without any fixed route or time table, was put into field test. It is called a “full demand bus” in Japanese. The bus goes anywhere to pick up or to drop a passenger within a designated area in the city. Two field tests were conducted in a large city with population of about 0.8 million, and a smaller city with population of about one hundred thousand. The result was semi-success in the smaller city but a total failure in the larger city. Our simulation[26,25] confirmed the result (Figure 8). However in simulation, we can try other parameter settings. It is natural to put more buses into operation where there are more passengers. Figure 9 shows the result of such simulation. The number of passengers is kept equal. If the number of passengers is doubled, the number of the buses in service is also doubled. This new simulation revealed the completely opposite result. Bus-on-call becomes more efficient as the population of the city gets larger. It was overlooked in the real field test because they did not have the luxury to involve more buses before knowing that it works. Note that to install the system in full scale, we need ubiquitous computing environment. A user must have a device through which he issues a request to the bus system. The

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

11

H. Nakashima / Cyber Assist Project for Ambient Intelligence Changes of Ave. Score by # of Bus and Demand Frequency 6

fixed path full demand (freq. = 1 * #bus) full demand (freq. = 2 * #bus) full demand (freq. = 3 * #bus) full demand (freq. = 4 * #bus) full demand (freq. = 5 * #bus) full demand (freq. = 8 * #bus) full demand (freq. = 16 * #bus)

5

ave. score

4

3

2

1

0 1

10

100

1000

# of bus (in log scale)

Figure 9. Simulation with Fixed Number of Passengers

request should be accompanied with information of the user’s current location and the destination. The device must also be able to negotiate with the system about allowable waiting time or estimated time to the destination and report them to the user. The user then decides to wait for the bus or take a taxi instead.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

3.5.2. Global Car Navigation Every car navigation system in operation has the current position of the vehicle, the destination, and the currently chosen route to the destination. If vehicles in a city can share that information, they can use traffic information to globally plan semi-optimal routes for each vehicle. In Japan, we now have the second generation car navigation system with VICS(Vehicle Information Communication System). VICS delivers the traffic status of the moment (to be precise, the information is of a slightly past in at least several minutes to an hour because it takes time to measure the traffic flow) to car navigation systems. A car navigation system then uses the information to plan the quickest route to the destination. It may include some detour route to bypass congested areas. There is an interesting research[31] that shows the efficiency of VICS-based car navigation. A simulation showed that the maximum performance of the system is obtained when only 20% of the cars are equipped with the system. When more than 80% uses the system, it makes the traffic worse (Figure 10) The reason is simple: Because the traffic information delivered by VICS is of the past, the whole system forms a feedback system with time delay, which is known to oscillate in certain conditions. In a simpler term, when the traffic condition of a road section gets bad, all the cars get the same information at the same time and they all try to avoid the route. We proposed a cooperative car navigation system with route information sharing (RIS)[34]. Each vehicle transmits route information (the current position, the destination, and the route to the destination) to a route information server, which estimates future traffic congestion using a simulation based on gathered information and feed it back to

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

12

H. Nakashima / Cyber Assist Project for Ambient Intelligence

Figure 10. Simulated effectiveness of VICS-based car navigation

the vehicles. Each vehicle then uses the estimation to re-plan its own route. This cycle is repeated. Our simulation confirmed the effectiveness of the proposed RIS system. The average travel time of drivers using RIS system is substantially shorter than those who do not. Moreover, as the number of RIS user increases, the total amount of traffic congestion of the city decreases.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

3.5.3. Rescue Multiagent simulation was also applied to rescue planning in RoboCup Rescue[13] research. Application of simulation in this domain has at least three potential uses - the first one is for training of rescue personnel, the second one is to test effectiveness of various tactics before actual disaster takes place, and the third one is for rescue planning after actual disaster takes place[28]. The third use is similar to the global car navigation in a sense that simulation is based on actual data and is used to predict future status[24]. 3.6. Intelligent Content Intelligent content is electronic information content with explicit semantic structure — the basis of Semantic Web in the broad sense of the term. Ubiquitous computing calls for intelligent content in order to promptly provide information best meeting the situated needs of the users. For the sake of grounding IT onto the semantics of the human lifeworld, we are exploring annotation-based intelligent content[8,9], whose semantics is shared by machines and humans. A major application of intelligent content is semantics-based information retrieval. Although minute semantic structures (typically those of semantic network) have been believed to be useless in information retrieval, we have demonstrated otherwise. Even with automatically (hence rather poorly) analyzed natural-language document database, our interactive semantics-based retrieval method exhibits twice as high efficiency as the traditional methods, which do not use such semantic structures. This should contribute to

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

H. Nakashima / Cyber Assist Project for Ambient Intelligence

13

the spread of semi-automatic, rich semantic annotation of information content, and thus promote high-quality information services based on such annotation. Currently the most promising killer application of Semantic Web is our semantic authoring[10], by which humans create machine understandable document from the beginning. This means that we enjoy machine support all the way through creation of documents, maintenance of contents, and intelligent semantic search. We integrated intelligent content and location-based communication to implement in-door navigation for people. On the part of the content technology, we developed a new, affordance-based, method for the semantic description of indoor situations, which highlights the human-sensible functions of places and artifacts. Another content-oriented development of ours to employ in various information services not restricted to in-door navigation is personal encounter support; Relationships among people are quantitatively calculated from the co-occurrences of their names in Web pages. We provided this service to the participants of a conference and demonstrated the potential.

4. Application to the Real World We believe it important to collaborate with and apply the technology to the real world. Through the real world applications, we can find many wrong assumptions which we initially had when we began research and development. We can even find new value and new research themes.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

4.1. Intelligent Home Appliances In early 2003, we conducted a project to test user interface technologies for intelligent home appliances. We designed a universal integrated remote controller for the next generation home appliances that are connected to home LAN. A user does not have to control each device separately. He or she even may not be aware of which device they are controlling. All they have to do is to specify requirements. We implemented voice recognition system with full sentence parsing capability to respond to human request in any form. A user may say “I want to see HNK news”, or “Show news on Iraq warfare”. The user may not be aware that the news is currently broadcast or it is recorded. Conceptually, this application is similar to intelligent room applications seen in other projects. Because the user interacts with the real world environment, it is important to have a “natural” user interface on the agent device in addition to the standard web browser interface. We use infrared communication for the physical reference, where the IP address of the agent device and the URL of the service on the microserver that is pointed at by the user are exchanged. We also use voice recognition for specifying services, where the user’s voice is sampled on the agent device and then transmitted to the voice recognition server by the VoIP (voice over IP) technology. The result may be used with the multicast DNS service discovery. Another interesting method we have is location-based service lookup. For example, one can specify printers in the meeting room by “meetingroom._lpr._tcp.local” using location as a capability. We are collaborating also with Yokohama City to carry out an experiment on intelligent content involving the general public. The experiment should demonstrate the

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

14

H. Nakashima / Cyber Assist Project for Ambient Intelligence

feasibility of a massive circulation among people of knowledge on every aspect of their daily life, such as shopping, health care, education, transportation, medication, finance, and local administration.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

4.2. Event Space Support Event spaces such as conferences, museums, expositions, concerts, parties and so on consist of rich content and many attendants. We developed and actually applied event support systems. The constituent of an event space is not only exhibit or a performer but also attendants themselves. One of the targets of the project is to create and grow communities of attendants for those events. In an event such as an academic conference, attendants can enjoy and utilize the event fully by having access to proper content and other attendants. The system delivers rich content even before the event and attendant’s community begins form through use of Internet technologies such as Blog and mail. In that process, the system also offers support with various AI technologies. The first platform for an annual conference of Japanese AI society (JSAI) has been realized using CoBIT system and Web technologies. The system has been used since JSAI 2003 to date. We installed ID sensors, PCs and monitors in the environment (entrances to lecture rooms, reception halls and so on). The content of the proceedings (papers and schedules) are presented to the users according to their interests and locations[17,23]. Another service provided was “interactive map”, which has various service icons arranged considering each person’s preferences as well as his/her history of browsing or interaction. When an attendant comes close to an “information kiosk” monitor, the system senses his/her ID and appropriate map is displayed. A camera on the monitor detects CoBIT motion so that is can be used as a mouse to interact with the map. Participants use CoBIT to browse various kind of information at kiosks in the event space. Information is normally displayed on the terminal at kiosks but if they want more personal or detail information, it can be downloaded into their own terminals. In event spaces, it is important to design the system from social perspectives. Interactions among participants are sometimes the key component of events, conferences in particular: If a participant fails to talk to persons who he wants to, he may be unsatisfied with the event. Relationships of researchers, automatically retrieved from Web [15] (Figure 11) are used to support meetings. For example, if you want to meet a particular researcher who you do not know personally, the system can list you a chain of people to reach the person. This service is independently provided as Polyphonet[14] throughout a year. 4.3. Aichi World Expo The largest application of our technology was in Aichi World Exposition held in 2006[12]. It is another example of event space support. However, since the target is a general audience, we could not use human relationships. As the result, the communication channel was one way, from the event planner to attendants only. We used successors of CoBIT, named Aimulet(TM) 1. Two kinds of Aimulet’s are used: Aimulet-GH (for Global House) and Aimulet-LA (for an installation by Lorie An1 Aimuletis

the trademark of AIST. This name comes from information + amulet, or AI + amulet.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

H. Nakashima / Cyber Assist Project for Ambient Intelligence

15

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 11. JSAI social network automatically extracted from the Web

Figure 12. A page from JSAI 2005 support system

derson). They both have the capability of delivering audio sound. The basic mechanism is the same as CoBIT: Analog sound is transmitted from LED’s to the solar cells of Aimulet’s. The internal speaker is directly activated by the voltage produced by the solar cell. Two kinds of Aimulet-GH’s are used: One for Japanese and one for English. We originally planned to have five different languages but had to abandon it. Since we stuck to analog audio systems, band-path filters must be used to have multiple channels. However, this results in narrower wave bandwidth for each channel and we could not deliver enough power audible in rather noisy environment of the exhibition. Aimulet-GH’s are also equipped with RFID tags in addition to the acoustic devices. It was used in the Global House, a pavilion by the Japanese government, to listen to detailed explanation of exhibitions (Figure 14) and to keep track of movements of individual users. The recorded trajectories of user location are used only to analyze the appropriateness of the layouts. This analysis was achieved off-line after the Expo to be

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

16

H. Nakashima / Cyber Assist Project for Ambient Intelligence

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 13. AimuletGH

Figure 14. Usage of AimuletGH

used for future exhibitions. We could have used the lactation on-line to provide more personalized information but gave it up in concerning privacy issues. Aimulet-LA was used outdoor. To prevent the strong infrared emission of the Sun from overloading the solar-cells, they are set downward. The light source is therefore set at the ground level (see Figure 16). We found this setting quite satisfactory because some portion of infrared beam is reflected by the cloths of the user and made the reception better. The design target was a cheap dispensable device that can be given out as a ticket to the garden. This will lower management cost to recover terminals at the exit 2 . 2 The similar, but more expensive, audio devices are used at museums. They are usually handed out in exchange of some personal identification such as driver’s licenses and then recovered at the exit.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

H. Nakashima / Cyber Assist Project for Ambient Intelligence

17

Figure 15. AimuletLA

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 16. Usages of Aimulet(Emitters are visible at the bottom of the fence)

The casing of Aimulet-LA was made of bamboo to meet the theme of the expo: ecology. We are happy to announce that Aimulet-LA got Japanese G-mark 3 for ecology in 2006. 4.3.1. Situatedness Issue of Aimulets To emphasize the situatedness of Aimulets, let us compare the concept with another one. The XPod system [4] integrates awareness of human activity and musical preferences to produce an adaptive system that plays the contextually correct music. It is a good example of sense-represent-compute-act system. The system’s inputs are galvanic skin response, mean and standard deviation of the acceleration, skin temperature and heat flow of the user, together with other information like time of day and day of week. The XPod system is comprised of a standard3 playing device and a human body sensor. The device tracks and stores a record of song meta-data as a song is played, including artist, album, genre, title, and beats-per-minute. In addition, the system records the time 3A

good design mark accredited by the Ministry of International Trade and Industry

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

18

H. Nakashima / Cyber Assist Project for Ambient Intelligence

of day, a user’s rating of the music, and a full range of physical responses from the user’s body. Each of the symbolic attributes, artist, album, title, and genre are all expanded into a large number of binary attributes. This was done so that the symbolic attributes could be accurately handled by the numerical algorithms, such as SVMs and neural networks. Aimulets, on the other hand, have no input or feedback on the devices themselves. The environment side system, which consists of computer controlled information source and infrared emitters located in various points in the environment, controls the content to be delivered at each location and time. The environment as a whole is used to set-up a context and the user with Aimuletmay just wonder around in the context to pick-up whatever the content delivered in the situation. Therefore, Aimuletsas a personalized information terminal has the following shortcomings: 1. When more than one users are at the same location, the system cannot deliver different content and/or in different timing. For example, at each exhibition, the system cannot coordinate its explanation timing with each user’s arrival. 2. In the same reason, the system cannot deliver personalized content adjusted to each user when there is more than one user at each information delivery point.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Acknowledgment The research described in this chapter is carried out by the members of Cyber Assist Research Center. The following names are only part of them. Koiti Hasida, the vice-director of CARC, led the intelligent content team. Koichi Kurumatani led the multiagent team. Itsuki Noda was in charge of the simulation for dial-a-ride bus system and found its efficiency. Tomohisa Yamashita worded on the traffic simulation to verify the efficiency of our global car navigation system. Akira Mori led the software team. He conducted a field test of intelligent home appliances. It is one of the real world applications of CARC technologies. Takuichi Nishimura led the interface team. He is the main inventor of CoBIT. He also contributed to set up and then led JSAI support systems. Yutaka Matsuo’s Polyphonet system was one of the main services used there. Hideo Ito led the device team. His main field is optoelectronics and supported many designs of infrared communication systems. We also thank members of the Advisory Board to their continuous support and helpful comments: Shun-ichi Amari (Riken Brain Science Institute), Yuichiro Anzai (Keio University), Rodney Brooks (MIT), Kokichi Futatsugi (JAIST), William Mark (SRI International), Yoshio Tanaka (IBM Japan; currently at Microsoft Japan), Kouji Ohboshi (NTT DoCoMo), Stanley Peters (Stanford University), Ikuo Takeuchi (Univ. of ElectroCommunication; currently at Univ. of Tokyo), Jun’ichi Tsujii (Univ. of Tokyo), Wolfgang Wahlster (DFKI GmbH), Steven Willmott (Universitat Politecnica de Catalunya) and Akinori Yonezawa (The University of Tokyo).

References [1]

Jon Barwise and John Perry. Situations and Attitudes. MIT Press, Cambridge, Massachusetts, 1983.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

H. Nakashima / Cyber Assist Project for Ambient Intelligence

19

[2] D.C. Dennett. Cognitive wheels : the frame problem in AI. In Z.W. Pylyshyn, editor, The robot’s dilemma : the frame problem in artificial intelligence. Ablex Publishing Corp., 1987. [3] Michael L. Dertouzos. The future of computing. Scientific American, July 1999. [4] Sandor Dornbush, Jesse English, Tim Oates, Zary Segall, and Anumpam Joshi. Xpod: A human activity aware learning mobile music player. In Proceedings of the Workshop on Ambient Intelligence, IJCAI2007, 2007. [5] Krzysztof Gajos, Luke Weisman, and Howard Shrobe. Design principles for resource management systems for intelligent spaces. In Proceedings of the 2nd International Workshop on Self-Adaptive Software, Budapest, Hungary, 2001. [6] James J. Gibson. The Ecological Approach to Visual Perception. Houghton Mifflin, 1979. [7] Steve Hanks and Drew McDermott. Default reasoning, nonmonotonic logics, and the frame problem. In Proc. of AAAI-86, pages 328–333, 1986. [8] Kôiti Hasida. Global document annotation. In Proc. the Natural Language Processing Pacific Rim Symposium 1997, pages 505–508, 1997. [9] Koiti Hasida. Gda: Annotated document as intelligent content. invited talk. In COLING 2000 Workshop on Semantic Annotation and Intelligent Content, 2000. [10] Koiti Hasida. Distributed semantic authoring as foundation of semantic society. In Hasida and K. Nitta, editors, New Frontiers in Artificial Intelligence: Joint Proceeding of the 17th and 18th Annual Conferences of the Japanese Society for Artificial Intelligence, page to appear. Springer, 2005. [11] Hideo Itoh, Kazutaka Kosugi, Xin Lin, Yoshiyuki Nakamura, Takuichi Nishimura, Kuniharu Takizawa, and Hideyuki Nakashima. Spatial optical point-to-point communication system for indoor locarionbased information services. In Proc. of 2004 ICO International conference Optics and Photonics in Technology Frontier, 13D2-28, 2004. [12] Hideo Itoh, Xin Lin, Ryosaku Kaji, Tatsuya Niwa, Yoshiyuki Nakamura, and Takuichi Nishimura. Compact battery-less guest guidance system at the expo 2005, aichi japan. In International Conf. on Control , Automation and Systems (ICCAS 2005), UA10-1, 2005. [13] Hiroaki Kitano and Satoshi Tadokoro. RoboCup rescue: A grand challenge for multiagent and intelligent systems. AI Magazine, 22(1):39–52, 2001. [14] Yutaka Matsuo, Junichiro Mori, Masahiro Hamasaki, Keisuke Ishida, Takuichi Nishimura, Hideaki Takeda, Koiti Hasida, and Mitsuru Ishizuka. Polyphonet: an advanced social network extraction system from the web. WWW 2006, pages 397–406, 2006. [15] Yutaka Matsuo, Naoaki Okazaki, Kiyoshi Izumi, Yoshiyuki Nakamura, Takuichi Nishimura, Koiti Hasida, and Hideyuki Nakashima. Inferring long-term user properties based on users’ location history. In Proc. IJCAI 2007, pages 2159–2165, 2006. [16] Humberto R. Maturana and Francisco J. Varela. Autopoiesis and Cognition: The Realization of the Living. D. Reidel Publishing Co., Dordrecht, Holland, 1980. [17] Yoshiyuki Nakamura, Takuichi Nishimura, Hideo Itoh, and Hideyuki Nakashima. A system of card type battery-less information terminal: Cardbit for situated interaction. In Prof. of IEEE International Conference on Pervasive Computing and Communications (PerCom 2003), pages 369–377, 2003. [18] Hideyuki Nakashima. Cyber assist project and its security requirements. In Proc. The First International Conference on Security in Pervasive Computing. Springer, 2003. Invited Talk. [19] Hideyuki Nakashima, Masatomo Hashimoto, and Akira Mori. Ubkit for cyber assist. In Active Media Technology: Proceedings of the 2nd International Conference, pages 46–56. Knowledge Systems, 2003. [20] Hideyuki Nakashima and Koiti Hasida. Location-based communication infrastructure for situated human support. In Proc. SCI 2001(World Multiconference on Systemics, Cybernetics and Informatics), 2001. "Location-Based Wireless/Optical Communication for Situated Human Support"-Invited Session(IS00165). [21] Takuichi Nishimura, Hideo Itoh, Yoshiyuki Nakamura, Yoshinobu Yamamoto, and Hideyuki Nakashima. A compact battery-less information terminal for real world interaction. In Pervasive 2004, pages 124–139. Springer LNCS 3001, 2004. [22] Takuichi Nishimura, Hideo Itoh, Yoshinov Yamamoto, and Hideyuki Nakashima. A compact batteryless information terminal (cobit) for location-based support systems. In Proceedings of International Symposium on Optical Science and Technology (SPIE), 4863B-12, 2002. [23] Takuichi Nishimura, Yoshiyuki Nakamura, Hideo Itoh, and Hideyuki Nakashima. System design of event space information support utilizing cobits. In Proc. of the 24th International Conference on Distributed Computing Systems Workshops (ICDCSW’04), pages 384–387, 2004.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

20

H. Nakashima / Cyber Assist Project for Ambient Intelligence

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

[24] Itsuki Noda. Database for information sharing and integration for rescue robot and simulation. In Itsuki Noda, editor, Proc. of Third International Workshop on Synthetic Simulation and Robotics to Mitigate Earthquake Disaster (SRMED 2006), pages 30–35, 2006. [25] Itsuki Noda, Masayuki Ohta, Yoichiro Kumada, Kosuke Shinoda, and Hideyuki Nakashima. Usability of dial-a-ride systems. In Proc. AAMAS 2005, pages 1281–1282, 2005. [26] Itsuki Noda, Masayuki Ohta, Kosuke Shinoda, Yoichiro Kumada, and Hideyuki Nakashima. Evaluation of usability of dial-a-ride systems by social simulation. In Multi-Agent-Based Simulation III. 4th International Workshop, MABS 2003 (LNAI-2927), pages 167–81, 2003. [27] Donald A. Norman. The design of everyday things. Doubleday, 1990. [28] Masayuki Ohta, Tomoichi Takahashi, and Hiroaki Kitano. Robocup-rescue simulation: In case of fire fighting planning. In RoboCup 2000: Robot Soccer World Cup IV, Lecture Notes In Computer Science; Vol. 2019, pages 351 – 356. Springer-Verlag, 2001. [29] The European Union Report. Scenarios for ambient intelligence in 2010. 2001. [30] Akio Sashima, Noriaki Izumi, and Koichi Kurumatani. Consorts: A multiagent architecture for service coordination in ubiquitous computing. In Koichi Kurumatani, Shu-Heng Chen, and Azuma Ohuchi, editors, Multi-Agent for Mass User Support, International Workshop, MAMUS 2003, volume 3012 of Lecture Notes in Computer Science, pages 190–216. Springer, 2004. [31] Iwao Tanahashi, Hironobu Kitaoka, Miyako Baba, Hiroko Mori, Shigeo Terada, and Eiji Teramoto. Global traffic simulator netstream (in japanese). R&D Review of Toyota CRDL, 37(2), 2002. [32] Francisco J. Varela, Evan Thompson, and Eleanor Rosch. The Embodied Mind: Cognitive Science and Human Experience. MIT Press, 1991. [33] Mark Weiser. The computing for the 21st century. Scientific American, 265(3):94–104, 1991. [34] Tomohisa Yamashita, Kiyoshi Izumi, Koichi Kurumatani, and Hideyuki Nakashima. Smooth traffic flow with a cooperative car navigation system. In Proc. AAMAS 2005, pages 478–485, 2005.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Advances in Ambient Intelligence J.C. Augusto and D. Shapiro (Eds.) IOS Press, 2007 © 2007 The authors and IOS Press. All rights reserved.

21

Ambient Intelligence – From Personal Assistance to Intelligent Megacities Michael BERGER1, Florian FUCHS, Michael PIRKER Siemens AG, Corporate Technology, Intelligent Autonomous Systems, Germany

Abstract. Ambient Intelligence as a vision for future information systems is currently receiving more and more attention in research and industry. In this article, we investigate how related technology domains, such as Artificial Intelligence, Multi-Agent Systems and the Semantic Web can provide methods and IT infrastructures to support Ambient Intelligence middleware and applications. By means of three case studies ranging from personal assistance to megacity management we identify functional and non-functional requirements. Based on this study we propose AmIRA as an Ambient Intelligence Reference Architecture with a focus on methods, architectures and components. We illustrate the instantiation of AmIRA by describing the implementation of three industrial applications related to the previously presented case studies. Keywords. Ambient Intelligence, reference architecture, Artificial Intelligence, Intelligent Agent and Multi-Agent Systems, mega trends, personal assistance, preventive health care, intelligent infrastructure management

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Introduction and Background The notion of Ambient Intelligence (AmI) describes the vision of electronic environments, which actively, personalized and situation sensitively act to support humans and to fulfill their desires [1]. Technologies like sensor-based situation understanding, middleware for distributed and heterogeneous devices, attractive multimodal user interfaces, predictive as well as situation adaptive decision making, and autonomous as well as cooperative system behavior are needed to realize that goal [2]. Artificial Intelligence and Intelligent Agent and Multi-Agent Systems technology play a central role as technologies for intelligent behavior [3]. Today, many AmI developments and solutions already exist as prototypes and demonstrators. Current application domains are already broad and reach from personal assistance by intelligent mobile devices [4,5], intelligent clothes [6], and entertainment solutions [7,8] to integrated smart office and meeting rooms [9] and home environments [10,11,12]. The way towards products is still limited, but on the border to emerge. Consumer product companies, such as Philips and Nokia with first products 1

Corresponding Author: Dr.-Ing. Michael Berger, Siemens AG, CT IC 6, Otto-Hahn-Ring 6, D-81739 Munich, Germany; E-mail: [email protected]

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

22

M. Berger et al. / Ambient Intelligence – From Personal Assistance to Intelligent Megacities

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

like the Ambilight TV and the Intelligent Phone, are currently the main drivers behind the productization. In addition to that, we do not only see end-user driven applications of AmI. It will also play an important role for future industrial infrastructures and public services scenarios. Therefore, AmI represents a key technology for coping with future mega trends such as demographic change, health and elderly care, increasing mobility, safety/security and urbanization/megacities as well as saving natural resources and environmental care. For example, AmI will optimize traffic, water and energy networks, realize dynamic supply chains in globally organized companies, and enable emergency prevention and security surveillance. The applicability will reach from personal environments (room, home, office, car) to single public places and buildings (trains, train stations, airports, stadiums) up to the whole megacity with its infrastructure. Many of the current developments of AmI systems are done from scratch and implement domain-specific and completely heterogeneous AmI systems. What is still missing, for the acceptance of AmI and its commercial success, is a common AmI reference architecture with defined interfaces where different components and AmI systems can be integrated. This is even more important, if we look at the challenges of the mega trends. Systems are getting more and more complex and have to be operated in an optimized way. The integration of several sub-systems is required, e.g. for monitoring and controlling megacities. Keeping these considerations into account, this chapter will develop a software engineering approach for technology integration in a re-usable AmI framework to enable applications in the before mentioned domains. In section 2, we will first describe in detail three different scenarios from different AmI application domains: personal assistance, smart home health care and megacity infrastructures. Section 3 will then analyze requirements shared by typical AmI systems, while Section 4 is defining the AmI reference architecture and (especially AI) technology components based on these requirements. Section 5 is describing the prototypic realization of the reference architecture based on our own projects related to the three domains described in section 2. Section 6 give as short overview of related work and Section 7 summarizes the results and concludes with a view to further work followed by references.

1. Ambient Intelligence Scenarios This section describes three representative scenarios for AmI systems in domains reaching from personal assistance to intelligent megacities. They serve as a basis for identifying requirements on a common AmI reference architecture. 1.1. Personal City Guide Individuals will find it challenging to find their way around future megacities. Personal AmI systems will assist both inhabitants and visitors in their activities. Imagine Ralf, who is living on the outskirts of a megacity called Metropolis. He plans to see a rock concert at the Victoria stadium in central Metropolis tonight. As it is his free afternoon, Ralf – an architecture aficionado – seizes the opportunity and asks the megacity’s visitor guide system for a route to the stadium which will take him past some of the architectural highlights of Metropolis. Being aware of Ralf’s interests, the current traffic situation and Ralf’s time constraints, the system compiles an appropriate route

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

M. Berger et al. / Ambient Intelligence – From Personal Assistance to Intelligent Megacities

23

and transfers it to his PDA. On the road, the system guides him to the relevant sights and automatically adapts the route when he decides to spend more time than expected at a particular place. Reaching the Inner Ring Road in time, road-side displays indicate that access to the city center has been banned for individual transport until further notice due to increased fine particulate air pollution. The city’s traffic management system provides information to Ralf’s in-car navigation system for planning a travel schedule to the stadium using public transport. Then he is guided to a free car park located nearby an appropriate train station. When Ralf leaves the car, his travel schedule is transferred to his PDA and guides him to the correct platform. As the city’s public transport operator has predicted increased traffic towards the Victory stadium, additional trains have been deployed to the rapid transit network. Entering one of the trains, displays at the entrance show passengers areas of the train with free seats available and low noise level. Using public transport Ralf and thousands of others reach the stadium faster than if they would have used their own car. During his trip, Ralf’s route (more precisely: the travel card he is carrying) has also automatically been tracked using several positioning systems and the fare has been deducted from his balance. Ralf is happy about the relaxed journey to the stadium and enjoys the concert. At the end of the concert, access to the city center has been granted again to individual traffic. At the same time, the city’s traffic management is prepared to move the thousands of people leaving the stadium back to their homes. As it has suddenly started snowing, however, they are not directed to the light rail system running on the surface, but to the metro running in the underground. Still, Ralf prefers to take a taxi back to the car park where his car is located. For this, he instructs his personal assistant: Using an ad-hoc network, the assistant identifies nearby people who are headed towards a similar destination. As soon as it has found a sufficiently large group, it proposes to share a taxi in order to safe money. This way, both the environment and Ralf’s wallet are taken care of, and Ralf has a comfortable trip to his car. On his final drive home in his own car, electronic traffic signs guide him around an area where an accident has happened only minutes ago. Taking a different route home, he can hear the emergency vehicles speeding towards the scene as they take advantage of the progressive green light the traffic management has set up for them. 1.2. Smart Home and Preventive Health Care The effect of demographical and social changes in most industrial countries has an enormous effect on either public or private emergency services, especially in megacities like Metropolis. Already today, more and more elderly people still live in their preferred environment at home and use commercially available call systems or patient monitoring systems. However, the evaluation of emergency service usage statistics shows that a high percentage of initiated emergency calls either do not have a significant emergency indication and can therefore lead to inefficient solutions or are initiated too late to provide help at all. Since there is no anticipation of emergency situations this leads to increased hospitalization in case of emergency situations and increased relocation to nursing homes after rehabilitation in hospitals. Future AmI systems should provide a better solution. Imagine the old lady Agathe which is connected to and using an AmI emergency management system. The system knows Agathe’s diseases and health history and is monitoring Agathe’s daily routine tasks by unobtrusive surveillance. The system detects abnormalities, reasons about

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

24

M. Berger et al. / Ambient Intelligence – From Personal Assistance to Intelligent Megacities

emergency situations, and proactively anticipates emergency situations to decrease the amount of invalid emergency situations and detect arising emergency situations earlier. When an upcoming emergency situation is anticipated, different resolution strategies are employed depending on the criticality of the situation: 1. The first strategy (daily support component) enables the interaction with the affected person by various AmI technologies to prevent emergencies in daily life. A daily routine task model including regular medical treatments should be followed in order to resolve the arising emergency situation. Additionally, the patient could taught how to eat diet food and how to take the medicine. 2. By a second strategy (social network support component), other people like family, friends and care givers are integrated into the daily routine task model in order to prevent emergency situations with adequate help. Audio/video channels could establish a connection to the far living family. But the system could also organize assistance from near living friends, e.g. for talking to Agathe or by going shopping to the supermarket. Telemedicine and social competence centres could be involved. 3. In urgent cases, the public emergency service with acute medical care support will be informed (by the emergency support component). Because of the detailed emergency information provided by the emergency support component, the most appropriate service can be selected. If needed, even a bed in the nearest hospital will be negotiated in the background and care takers, friends, and the family will be involved for the recreation process. Leveraging AmI technologies for preventive health care systems, e.g. in the case of elderly people suffering from particular chronic diseases, will help them stay longer in their preferred environment. AmI technologies are needed for the implementation of all the listed components (the surveillance component and the three possible resolve components).

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

1.3. Megacity Infrastructures The city of Metropolis has a complex system of several large-scale infrastructures such as traffic, water and power distribution infrastructures. The traffic infrastructure, for example, is itself composed of several large-scale sub-infrastructures like public and private transport (the road network, bus and railway systems, airports or taxis). Subinfrastructures are managed through command-and-control infrastructures for supervising traffic and controlling traffic lights and road-side displays, for example. Keeping track of all relevant aspects of the overall traffic infrastructure at such a large scale can only be realized in an automated way. Therefore the monitoring of the many different infrastructure facets is realized in Metropolis by a huge number of distributed, heterogeneous information sources: Various types of sensors are attached to critical infrastructure parts for timely detection of critical situations (e.g. dangerous wear and upcoming blackouts) and for optimizing the overall system throughput (e.g. avoiding traffic congestions and improving average speed). Additional information is collected from external data sources such as information about weather and events (e.g. fairs, concerts) as well as by human inspection (for places, situations and systems that can not appropriately be monitored by sensors). A network of geographically distributed Traffic Command and Control centers (TCCCs), each of which responsible for a certain district of the city collects, integrates and uses this monitoring information to manage the city’s overall traffic infrastructure.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

M. Berger et al. / Ambient Intelligence – From Personal Assistance to Intelligent Megacities

25

During normal operation, each TCCC’s major task is to collaboratively control public and private transport in an intelligent way, based on information gathered from many heterogeneous sources. Traffic and crowd movement is dynamically optimized based on identification of traffic hot spots and prediction of required transport capacity: Hot spots, for example, are derived from crowds due to events, like fairs, known from event databases. Also incipient traffic and crowd congestions, measured by sensor systems like inductive loops and video camera systems, are used to identify hot spots. Transport capacity predictions are, for instance, computed from known rush hours, foreseeable higher demand of public transport due to bad weather conditions or from a significantly higher number of incoming people in local airports. To handle these different situations, the TCCCs can dynamically adapt the frequency, routes and types of public transport. For private transport, the TCCCs can remotely open or close lanes and change speed limits with road-side displays, switch between different modes of progressive green light favoring different traffic streams, and influence personal driving behavior and utilization of public transport by providing individual navigation services. Additionally, the TCCCs provide traffic-related convenience services to all residents, like seamless payment solutions for public transport or traffic related information services (nearest car park, traffic hot spots, public transport based routing). In the case of traffic infrastructure-related incidents, the responsible TCCCs identify, orchestrate and execute the necessary actions, while minimizing the impact on the remaining parts of the traffic infrastructure: x Incident example 1: If the emergency management system of the elderly person Agathe detects a severe health problem, the TCCC in Agathe’s district is automatically contacted by the emergency hotline that was informed by the emergency management system. It is the responsibility of the TCCC to find out, perhaps collaboratively together with other involved TCCCs, the most appropriate routes: On the way to Agathe the fastest route, avoiding traffic hot spots, must be selected. On the way to the hospital, when Agathe is on board, the most appropriate route, depending on Agathes physical condition, is selected (e.g. taking a longer route over less bumpy roads). The routes are computed based on current information like congestion, construction sites, events, weather conditions, current conditions of roads (bad paving), and on static information like high-speed routes. After the route has been planned and Agathe is on her way to the hospital, the TCCC(s) dynamically initiate a progressive green light for the critical crossings. x Incident example 2: The Metropolis Command and Control Center (MCCC) is a central hub for all infrastructure related CCCs. It just received the notification about a power blackout in district 12 from the Power CCC. Based on the current context (type of incident, time, geographic location, etc.) the MCCC identifies all relevant CCCs. A TCCC, also responsible for district 12, is informed. The TCCC uses all information received from the MCCC and its own current information about the traffic system to adapt the public and private transport system: Power-dependent public transport is rerouted, extra busses are scheduled based on a transport capacity estimation, private navigation systems, like the one of Ralf, receive a notification to replan and avoid the parts of the district where a lot of traffic lights are down. Additionally, the TCCCs responsible for the neighboring districts, which were not notified by the MCCC, are informed that extra public and private traffic load must be handled.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

26

M. Berger et al. / Ambient Intelligence – From Personal Assistance to Intelligent Megacities

2. Common Requirements on AmI Systems Based on the previously described application scenarios, this section identifies requirements that are common to all scenarios and therefore have to be reflected in a generic AmI reference architecture. We distinguish between functional and nonfunctional requirements. Functional requirements describe the necessary system behavior in terms of functions that define the internal workings of the system. Nonfunctional requirements describe the overall characteristics of and constraints on the system design and implementation. They can also be used to assess system operation in the form of key performance indicators.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

2.1. Functional Requirements A common requirement for all described AmI systems is to integrate information from distributed and heterogeneous sensor and data sources. Examples from scenario one are weather information and traffic information. In scenario two, information from vital sensors, surveillance cameras and medical databases are relevant, and in scenario three, sensors mounted at the rail and power infrastructure, but also traffic information are required. The goal is to obtain an integrated view of the world state that is relevant to the AmI system. This implies that a middleware is required that provides connectivity to heterogeneous sensor devices and other data sources. It has to provide management services for the distributedly produced data and needs to realize semantic interoperability among information from heterogeneous sources. Also common to all presented scenarios is that particular situations have to be recognized and detected. In scenario one and two, these situations have a local scope such as being on the way home and critical health state. In scenario three, the relevant situations are increasingly large-scale such as detecting critical infrastructure states. All AmI systems therefore need to support situation understanding. This requires appropriate middleware and algorithms for identifying relevant aspects of the known world state as well as enabling efficient recognition. The core functionality of each described AmI system is its intelligent behavior. This is specific to the particular system. For example, scenario one involves route planning, ad-hoc group formation and other algorithms. Scenario two requires assessing situations and deriving appropriate actions, but also planning routes and scheduling actions. Scenario three requires diagnosing root causes and deriving effects of particular failures. Although highly application-specific, all functionalities build on fundamental artificial intelligence (AI) problem solving techniques such as searching, knowledge representation and reasoning, planning and learning. A common requirement in all described scenarios is therefore to have a collection of problem solving components available that can be instantiated for the particular problem. Finally, in all scenarios the system provides some form of decision support to its user or even acts autonomously. In scenario one, Ralf is recommended an optimal route for traveling home. Road-side traffic signs direct traffic along preferred routes. In scenario two, appropriate measures for handling an emergency situation such as alerting emergency services and controlling traffic lights are autonomously taken. In scenario three, infrastructure operators are provided with decision support as potential causes for failures and expected effects are pointed out. This is why all AmI systems also require appropriate human machine interfaces and actuators.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

M. Berger et al. / Ambient Intelligence – From Personal Assistance to Intelligent Megacities

27

2.2. Non-Functional Requirements

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

User focus is required to guarantee that AmI systems actually provide benefits to their users. Usability (e.g. simplicity of use), relevance (how many people are addressed), accuracy (e.g. reliability of information) and performance (e.g. response times and timeliness of information) are central to the acceptance of such an AmI system. Also cost plays an important role: The cheaper the offered services the higher the acceptance for the average user. The services should also be customizable and reconfigurable at runtime to fit personal user needs in the best way possible. Administration and maintenance considers overall system handling and life-cycle issues from the perspective of the system operators. A well-designed system design that is highly modular and extensible is essential for facilitating maintenance and administration. Also administrative tools and up-to-date documentation are required. Robustness of an AmI system is a prerequisite for achieving trust by users. The objective is to minimize Mean Time Between Failures (MTBF) and guaranteeing minimum system functionality for core services (like health services, traffic services, etc.). This requires adherence to quality standards such as ISO and efficient use of system resources such as network bandwidth, energy, memory, processing power. Also, measurements for disaster recovery have to be taken. Scalability is required for enabling system growth and improved system functionality. Compatibility with existing standards, portability, modularity and extensibility play an important role to maintain an acceptable level of system performance while user numbers are growing. Cost, legal and licensing issues at future system migration and extension are also important factors in the long run. Security and privacy are required for guaranteeing the secure operation of the system and preventing the abuse of person-related information. A number of techniques have been developed and are available for this. Furthermore, system services can be certified for increased trust by users, e.g. in payment transactions. Interaction among users should be modeled using trust and reputation mechanisms.

3. AmI Reference Architecture (AmIRA) Based on the previously identified common requirements of typical AmI systems, this section proposes the AmI reference architecture AmIRA. AmIRA is intended to serve as a blueprint architecture for AmI systems that provides a proven system design and facilitates reuse of AmI system components across different AmI system implementations. We propose a layered functional decomposition. Each layer provides a particular functionality as a service with a predefined interface. The advantage of a layered architecture is that each layer depends only on services from lower layers and does not have knowledge about higher layers. This approach maximizes maintainability and flexibility. We separate the application logic from the system model and distinguish two functional building blocks (see also Figure 1): The first building block is concerned with perceiving and understanding the current state of the environment in order to be able to take it into account (the Ambient part in “Ambient Intelligence”). The second building block is concerned reasoning and acting on this knowledge in order to provide assistance to users or to act towards a certain goal (the Intelligence part in “Ambient

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

28

M. Berger et al. / Ambient Intelligence – From Personal Assistance to Intelligent Megacities

Actuators and Human Machine Interface 1)

Reasoning and Acting

1b) 2)

Problem Solving Perceiving and Understanding Situation Understanding

2b)

World State Integration

World Model

2a)

Problem Situation Model Model

1a) Decision Support / Autonomous Behaviour

3a)

3b)

3c)

Sensors and Data Sources Figure 1: AmIRA – the proposed AmI reference architecture

Intelligence”). Both functional building blocks are based on explicit models that enable interoperability between the layers. We distinguish between world model, situation model, and problem model. All parts of the architecture will be described in the following.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

3.1. World, Situation, and Problem Model An important aspect of the presented architecture is that we propose explicit models that are separate from the application logic. Having an explicit model carries several advantages: Designing a complex system becomes much easier as it allows to separate model engineering from application development. Maintainability and extensibility of the system is higher as the system model can be adapted independently from the application logic. Furthermore, it facilitates development of other AmI systems as models or parts of them may be reused. We propose to use explicit models for three purposes: 1. The World Model describes the world states that are relevant to the AmI system. It represents the observations that can be made about the states of the relevant environment and is closely related to the types of sensors and data sources that are available to the AmI system. The World Model is used by the Situation Understanding layer for accessing information about current and previous world states from the World State Integration layer. 2. The Situation Model describes the situations that are relevant to the AmI system. Situations are defined as specific combinations of world states. They are usually not directly observable, but have to be inferred from the observed world states. The Situation Model is used by the Problem Solving layer for accessing information about current and previous situations from the Situation Understanding layer. 3. The Problem model describes the problems that are relevant to the AmI system and their solutions. It is used by the Decision Support / Autonomous

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

M. Berger et al. / Ambient Intelligence – From Personal Assistance to Intelligent Megacities

29

Behavior layer for accessing information about potential problem solutions from the Problem Solving layer. Depending on the specific application, different modeling formalisms can be used. Knowledge representation formalisms are particularly suited for representing the world and situation models. A very expressive example are formal ontologies. These are based on a decidable fragment of first-order logic, a so-called Description Logic (DL) [13]. This facilitates interoperability among systems and increases extensibility and maintainability. Above all, ontologies provide machine-readable semantics that can be automatically interpreted by logical reasoners. They also support consistency checks. OWL DL [14] is a W3C standard for representing formal ontologies. 3.2. Perceiving and Understanding

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

The Perceiving and Understanding building block builds on sensors and data sources that provide data about the current state of the relevant environment. Typical examples are sensor systems, such as positioning systems, and other data sources, such as timetable data. This building block is responsible for integrating this data into a homogeneous world state (world state integration) and using this integrated view for recognizing relevant situations (situation understanding). The World State Integration layer provides a single interface for querying the current and past world states as represented with respect to the world model. The service provided by this layer can be further structured into three steps: 1. It acquires observations about the relevant environment from sensors and data sources and manages this data appropriately. This includes handling data streams and historical data. Database technologies can be used for storing this data. Depending on data dynamics different optimizations may be possible. 2. It preprocesses and transforms these observations into a common representation as given by the World Model. Each sensor and data source, in principle, requires tailored preprocessing. However, preprocessing algorithms may be reused for the same type of sensor data / data source. For example, there could be an image analysis component, which creates an ontology-based scene description from a video stream 3. It integrates this data into a consistent instance of the World Model. This includes detecting and resolving inconsistencies among contradicting sensor data. Formal ontologies provide a flexible framework for this. The Situation Understanding layer provides a single interface for querying current and past situations as represented with respect to the Situation Model. This layer uses the World State Integration layer for accessing current and historical world states in order to understand situations. Recognizing situations based on world states is a complex task. It is up to the implementation of this layer to choose an appropriate technology or technology combination. Examples are ontology-based approaches where situations are defined as complex concepts and the realization service of a reasoner is used for deducing matching situations based on the available facts about the current world state [15]. Another logics-based approach are rule engines with appropriate rule sets [16]. Probabilistic situation understanding technologies, like Hidden Markov Models and

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

30

M. Berger et al. / Ambient Intelligence – From Personal Assistance to Intelligent Megacities

Bayesian Networks [17], include probabilistic classifiers that rank several predefined situations according to their probability. 3.3. Reasoning and Acting The Reasoning and Acting building block builds on the Situation Understanding layer that provides information about current and historical situations. It uses this information for performing application-specific and situation-dependent problem solving. Resulting solutions are the basis for providing decision support to users and performing autonomous behavior. Decision support and autonomous behavior are implemented using human machine interfaces and actuators. The Problem Solving layer provides a library of problem solving methods that can be instantiated in an application-specific manner by each AmI system. Problem solving techniques may access the Situation Understanding layer for taking into account the current or historical situation. Problem solving methods can include constraint solvers, linear optimizers, rule engines, reasoners and planners. In distributed settings, where coordination is necessary, also Multi-Agent Systems (MAS) are relevant. The Decision Support and Autonomous Behavior layer is responsible for effectuating solutions as determined by the Problem Solving layer. It does this by either informing the user about potential solutions and the required actions for getting there. If it is allowed to act autonomously, it has to perform the required actions and monitor their effects using the Perceiving and Understanding building block. Typical technologies include traditional graphical user interfaces on stationary or mobile devices as well as multi-modal and cross-device interfaces.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

4. Examples for the Instantiation of AmIRA This section explains the application of the previously proposed AmIRA reference architecture to the design and implementation of specific AmI systems. To this end it refers to the scenario descriptions in Section 2 and sketches the realization of appropriate AmI systems. Implementation of scenario one is described in detail while for the other scenarios primarily the particularities are pointed out. The mapping of the mentioned implementation components to the layers of AmIRA is also illustrated in Figure 2. 4.1. Personal City Guide System This scenario is actually a collection of a number of separate AmI systems such as the dynamic city guide [18], the traffic management system and the mobile grouping service [19]. The proposed AmI architecture is applicable to all of them. As a small, but exemplary case, we select the train seat recommender service and describe it in detail. This service uses wireless sensor networks in the train wagons and compartments for observing the current world state. Each wireless sensor is equipped with a light, a temperature and a noise level sensor. Sensor nodes are placed at the compartment door, at the compartment window and in each compartment seat.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

M. Berger et al. / Ambient Intelligence – From Personal Assistance to Intelligent Megacities

31

The World Model describes the observations made by these sensors, such as “high temperature on seat 1” and “low light level at compartment door”. The Situation Model builds on these world states in order to define complex situations. For example, it defines a “compartment that is suitable for reading” as a compartment with at least one free seat, high light level, and low noise level. A free seat would in turn be defined as a seat with low temperature (no person sitting on it) and high light level (no object placed on the seat). Other situations are “Compartment is fully occupied”, “Compartment has available seating”, and “Compartment can be used for sleeping”. Furthermore, the Problem Model describes the current occupancy state of a set of compartments and an optimal target state. All models are represented as OWL DL ontologies. The Perceiving and Understanding building block is implemented as follows: World state integration is performed on wagon level: Individual sensor nodes transmit their measurements via the wireless sensor network to the wagon’s super node. There they are preprocessed and transformed into a representation of the World Model. Temperature measurements are, for example, discretized into two classes, low and high temperature, and annotated with a timestamp. World state integration based on this common representation is also performed by the super node, which is equipped with persistent storage. Situation Understanding is implemented using a DL reasoner. With the Situation Model being defined based on the World Model, it can use the current world state provided by the World State Integration layer as an input for the reasoner. Using realization (a DL reasoning service that infers the most specific concept for a particular instance), the reasoner can then infer the current situation. For example, it infers that seat1 is occupied by a person and that, if all seats are occupied, this compartment is fully occupied. The Reasoning and Acting building block is implemented as follows: The Problem Solving layer for this AmI system is quite simple: The goal is to achieve a balanced occupancy rate of all compartments across wagons. So it first retrieves the current occupancy rate from the Perceiving and Understanding layer using the Situation Model. Based on this information, it uses a linear optimizer for determining an optimal load for the overall train. This solution is represented with respect to the Problem Model. The Decision Support / Autonomous Behavior layer uses the calculated solution as the basis for determining the directions to entering passengers that are required for reaching the intended load distribution. It uses the displays at the train entrances in order to present this information to the passengers. These directions are permanently adapted when a new situation is recognized and when a new problem solution is therefore delivered by the Problem Solving layer. 4.2. Smart Home and Preventive Health Care System This system [20] will encompass a number of environmental sensors that monitor particular aspects of a person’s daily routine unobtrusively at home. Examples are simple detectors at light switches, taps and pill boxes, but also elaborated video surveillance systems. The integration of these heterogeneous data sources is done at the world state integration layer based on an ontological world model that integrates several data source-specific ontologies. The data is stored in a central RDF repository located at each home in order to store historical data. Situation models are defined as ontological concepts and are based on the world state model. Detecting situations is done by the situation understanding layer. As the

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

32

M. Berger et al. / Ambient Intelligence – From Personal Assistance to Intelligent Megacities

complete information is available at the central repository, this can be done using a standard ontological reasoner. Understanding the current situation is then the basis for several components on the reasoning and acting layer: The daily support service monitors the detected situations and compares them with an internal reference model (problem solving). If there is a deviation, it takes an appropriate action (autonomous behavior) such as reminding the patient to take the medicine if this was not detected during the expected timeframe. Using a rule engine for this purpose enables quick and dependable adjustment of the system if different behavior should be desired in the future. The social network support service can also be realized using a rule engine and customized rule sets. In addition to capabilities for in-home communication, this service provides communication capabilities for contacting remote family, friends and care givers. The emergency support service takes care of critical situations. It uses the available information about a patient’s current condition in order to call an ambulance and take other appropriate measures. This may also involve routing and planning as the availability of medical facilities, their geographical location and reachability should be taken into account.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

4.3. Megacity Infrastructures System Relevant data sources for metropolitan infrastructures are existing sensor systems of existing infrastructures. These provide the basis for the world state integration layer. By means of adapters, this layer uses an ontology-based world model as the data model for integrating data from traffic infrastructure [21], power infrastructure [22] and healthcare infrastructure [20]. The data is physically stored in distributed repositories. Situations are defined in the situation model as an ontology based on the worldmodel. These include the load state of power systems, punctuality of transportation systems as well as the state accessibility state of road sections. The situation understanding layer employs distributed reasoning for recognizing situations across the distributed world state. This serves as the basis for the subsequent control. Several problem-specific components can be found on the problem solving layer. One example is a routing component that determines the optimal path through the street network according to patient requirements and taking into account the current state of the street network. Another example is an event correlation system that uses a dependency model for determining the root cause of a failure and its potential effects. On the decision support layer different components pick up the results from the problem solving layer and initiate the appropriate actions [23]: The ambulance router configures the traffic light system for a progressive green light. The Metropolis Command and Control Center uses the diagnosis results for taking appropriate counter measures and alerting other relevant command centers.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

M. Berger et al. / Ambient Intelligence – From Personal Assistance to Intelligent Megacities

Video & Audio Displays

Communication Traffic Controls Links

Infrastructure Controls

33

…

Reasoning and Acting

1)

Ontology

1a)

Guidance System Rule-based Behavior Scheduling

1b)

Linear Optimizer Case-B.Reas. Event Corr.

…

Route Planning

Rule Sets

… 3a) Ontology

Perceiving and Understanding

2) 2a)

DL Reasoner

Distributed DL Reasoners

… 3b) … Ontology

Central RDF Repository

2b)

Wireless Sensor Network

Video Cameras

Distributed RDF Repository

…

Power Tap Traffic Infrastructure Sensors Infrastructure Pill Box Sensors Sensors Sensors

… 3c) …

Figure 2: Illustration of some components on the AmIRA layers as mentioned in the instantiation examples

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

5. Related Work Much research and prototyping work related to generic architectures was carried out in the fields of Artificial and Ambient Intelligence, but up to now no commonly agreed reference architecture has been developed. In the AI domain, the following approaches have been proposed over the last years: The Real-time Control System [24] provides a reference architecture for control problems already applied in diverse domains like automotive industry, manufacturing or astronautics. It consists of a hierarchically layered set of processing modules connected by communication links. The Soar [25] architecture has been developed over many years. Its design is based on the hypothesis that all deliberate goal-oriented behavior can be cast as the selection and application of operators to a state. The Ikarus [26] architecture defines an integrated architecture for intelligent agents based on longterm memories for concepts and skills. In addition to these monolithic approaches, architectures for the flexible integration of different AI modules have been proposed recently: The Psyclone architecture and software platform [27] is based on a blackboard system. It includes support for multimodal perception of vision, audio, speech and other sensor input, as well as modular cognitive processing and multimodal output generation, such as speech and animated output. The CAVIAR system [28] is based on one global controller and a number of modules for information processing. Each module provides a complete description of itself including its input and output datasets. Both CAVIAR and Psyclone [29] support addition and removal of modules with minimal reconfiguration. In conclusion, the presented AI architectures are applicable to the problem solving aspect of AmI systems, but lack support for other AmI requirements as identified above.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

34

M. Berger et al. / Ambient Intelligence – From Personal Assistance to Intelligent Megacities

In the area of AmI architectures, there are primarily domain-specific approaches: [30] describes a system for the manufacturing industry. By taking into account the current context of human operators, it increases the observability of the whole manufacturing system. Another domain with several existing approaches is ambient assisted living. Here, the ongoing project [31] aims at identifying fundamental functional components, data flows and communication patterns in order to provide for interoperability between different AmI implementations. [32] reports on a middleware for AmI systems that provides core services such as communication, events, lookup, similarity search, logging, and context. [33] does not aim at developing a generic middleware, but rather proposes an abstract reference framework for AmI. This is to enable the integration of existing systems, while still being able to deploy new technologies. [34] focuses on an overall, but very generic AmI reference architecture. It identifies core technologies such as middleware for the integration of devices and technologies, communication methods and technologies, new sensor and actuator systems, semantic technologies for the interpretation of environment conditions, and interaction technologies provided for the user. While all these systems share some ideas of the presented AmIRA reference architecture, to the best of our knowledge, none of them has so far specified a generic architecture that is domain-independent and covers all functionality required.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

6. Summary and Outlook In this article, we investigated how related technology domains, such as Artificial Intelligence, Multi-Agent Systems and the Semantic Web can provide methods and IT infrastructures to support Ambient Intelligence middleware and applications. By means of three case studies ranging from personal assistance to megacity management we identified functional and non-functional requirements. Based on this study and our experience with several other Ambient Intelligent solutions we proposed AmIRA as an Ambient Intelligence Reference Architecture with a focus on methods, architectures and components. We illustrated the instantiation of AmIRA by describing the implementation of three industrial applications related to the presented case studies. Currently, some of the non-functional requirements, such as communication security, misuse of systems, information privacy and safety are not yet covered in the architecture. These are issues to be considered and introduced stepwise in the near future. Furthermore, the architecture does currently not cover regulatory frameworks, local laws and political conditions for using intrusive and un-obtrusive sensors. By applying the reference architecture to other solutions it will be completed and established and could also be part of still missing standardization activities in the domain of Ambient Intelligence.

7. References [1] [2] [3]

K. Ducatel et al.: Scenarios for Ambient Intelligence in 2010. IPTS-ISTAG, EC: Luxembourg, 2001. E. Aarts and J. Encarnação: True Visions – The Emergence of Ambient Intelligence, Springer, Berlin, Germany, 2006. M. Berger, J. Müller, Ch. Seitz: Multi-Agent Technologies for Ambient Intelligence – an industrial view. In: IT – Information Technology Journal, 47 (2005), Themenheft Agententechnologie, Oldenbourg, 2005.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

M. Berger et al. / Ambient Intelligence – From Personal Assistance to Intelligent Megacities

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

[4] [5]

35

Siemens Ay System. URL: http://www.siemens.com/aysystem/ [accessed: 18.08.2007] N. Richard et al.: Two Issues for an Ambient Reminding System: Context-Awareness and User Feedback. This volume. IOS Press. 2007. [6] M. Boronowsky et al.: Empowering the mobile worker by wearable computing - wearIT@work. In: Journal of Telecommunications and Information Technology, 9-14, 2006. [7] B. de Ruyter et al.: Assessing the effects of building social intelligence in a robotic interface for the home. Interacting with computers, 17, 522-541, Elsevier, 2005. [8] S. Dornbush et al.: A Human Activity Aware Learning Mobile Music Player. This volume. IOS Press. 2007. [9] A. Waibe1 et al.: CHIL: Computers in the human interaction loop. In: 5th Int’l Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS), Lisbon, Portugal, 2004. [10] B. de Carolis, G. Cozzolongo and S. Pizzutilo: An Agent-based Approach to Personalized House Control. In: Workshop on Artificial Intelligence Techniques for AmI at ECAI’06. Riva, 2006. [11] E. Arts, E. Diederiks: Ambient Lifestyle – From Concept to Experience. Eindhoven, August 2006. [12] H. Nakashima: Cyber Assist Project for Ambient Intelligence. This volume. IOS Press. 2007. [13] F. Baader et al.: The Description Logic Handbook: Theory, Implementation, and Applications. Cambridge University Press, 2003. [14] D. L. McGuinness and F. van Harmelen: OWL Web Ontology Language Overview, http://www.w3.org/TR/owlfeatures/. W3C Recommendation, 2004. [15] A.-Y. Turhan, T. Springer, M. Berger: Pushing Doors for Modeling Contexts with OWL DL: A Case Study. In: Proc. IEEE Intl. Conf. on Pervasive Computing and Communications Workshops, IEEE Computer Society, Washington, DC, USA, 2006. [16] H. Chen: An Intelligent Broker Architecture for Pervasive Context Aware Systems. PhD Thesis, University of Maryland, 2004. [17] Y. Zhang, Q. Ji: Active and dynamic information fusion for multisensor systems with dynamic Bayesian networks. In: IEEE Trans. on Systems, Man, and Cybernetics, Part B, 36(2):467-72, 2006. [18] R. Kramer, M. Modsching, K. ten Hagen: Development and evaluation of a context-driven, mobile touristguide. In: International Journal of Pervasive Computing and Communication (JPCC), http://www.pervasive-computing.org, 2005. [19] C. Seitz, M. Berger: Towards a General Approach To Mobile Profile Based Distributed Grouping. In: Proc. 17th Intl. Conf. on Architecture of Computing Systems (ARCS’04), Augsburg, Germany, 2004. [20] EC-funded project: Emergency Monitoring and Prevention (EMERGE). URL: http://www.emergeproject.eu/ [accessed: 17.07.2007] [21] F. Fuchs et al.: Towards Semantics-based Monitoring of Large-scale Industrial Sytems. In: IEEE Conference on Computational Intelligence for Modelling, Control and Automation. Sydney: IEEE Computer Society, 2006. [22] V. Terziyan, A. Katasonov: Global Understanding Environment: Applying Semantic Web to Industrial Automation”, In J. Cardoso, M. Hepp, M. Lytras (Eds.), Real-world Applications of Semantic Web Technology and Ontologies. Springer, 2007. [23] F. Fuchs et al.: Applying Semantic Technologies to Railway Decision Support. In: Proc. of Semantics’06, Austrian Computer Gesellschaft, Vienna 2006. [24] J. Albus. The NIST Real-time Control System (RCS): An Application Survey. In: Proc. of the AAAI 1995 Spring Symposium Series, Stanford University, 1995. [25] J. E. Laird, C. B. Congdon, and K. J. Coulter: The Soar User’s Manual. Electrical Engineering and Computer Science Department, University of Michigan, 2006. [26] P. Langley et al.: A Value-Driven Architecture for Intelligent Behavior. In: Proc. IJCAI-2003 Workshop on Cognitive Modeling of Agents and Multi-Agent Interactions (pp. 10-18). Acapulco, Mexico, 2003. [27] K. R. Thórisson et al.: Constructionist Design Methodology for Interactive Intelligences. A.I. Magazine, Vol 25, Issue 4, pp 77-90, 2004. [28] T. List et al.: A Plug and Play Architecture for Cognitive Video Stream Analysis. IEEE CAMP05, 2005. [29] T. List et al.: Two approaches to a plug-and-play vision architecture CAVIAR and Psyclone. In: AAAI05 Workshop On Modular Construction of Human-Like Intelligence, Pittsburg, Pennsylvania, 2005. [30] D. Stokic, U. Kirchhoff, H. Sundmaeker: Ambient Intelligence in Manufacturing Industry - Control System Point of View. In: Proc. Intl. Conf. on Control & Applications, IASTED, Quebec, Canada, 2006. [31] PERSONA project. URL: http://www.aal-persona.org/ [accessed: 17.07.2007] [32] M. Anastasopoulos et al: Towards a Reference Middleware Architecture for Ambient Intelligence Systems. In: Proc. Workshop for Building Software for Pervasive Computing (OOPSLA05), 2005. [33] A. Buchmann, M. Mezini, A. Schürr: Architecture for Ambient Intelligence. Report Thema Forschung, Technical University of Darmstadt, Issue 1, pp 12-14, 2007. [34] J. Encarnação, M. Mühlhäuser, R. Wichert: Ambient Intelligence – Research and Applications. Report Thema Forschung, Technical University of Darmstadt, Issue 1, pp 4-8, 2007.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

36

Advances in Ambient Intelligence J.C. Augusto and D. Shapiro (Eds.) IOS Press, 2007 © 2007 The authors and IOS Press. All rights reserved.

Towards a Context Theory for Context-aware systems Richard DAPOIGNY a,1 and Patrick BARLATIER a a Université de Savoie, LISTIC/Polytech’Savoie, Domaine Universitaire B.P. 80439, F-74944 ANNECY Le Vieux Cedex, France Abstract. The basic goal of context-aware systems is to make software aware of the environment and to adapt to their changing context. For that purpose, the core problem is to have a powerful context model. While significative formalizations have been proposed, context models are either expressed through logical formalisms or with ontology-based approaches. The major problem with all that approaches is that they suffer from the chronic insufficiency of first-order logic to cope with dynamic change and especially, to solve the frame problem. Therefore, building context-aware software is a complex task due to a lack of appropriate formal models in dynamic environments. In this paper, we propose a model which combines the strengths of both approaches while trying not to carry their specific weaknesses into the resulting formal framework. For this purpose, the formal model relies both on a knowledge representation with ontologies and on a logical reasoning with Dependent Record Types (DRT) based on Intuitionistic Type Theory and the Curry-Howard isomorphism. This logic modelling aims to be applied to any kind of process-based applications.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Keywords. contexts-aware, Intuitionistic Type Theory, Dependent record types, Ontology

Introduction Much research has been done in the area of context modelling, representation and reasoning in the past few years. It concerns a number of research communities ranging from human-computer interaction to pervasive computing. Unfortunately, there is no consensus about context definition, representation or modelling. Therefore, an epistemologically adequate and sound model is required which includes mechanisms for context integration and usage. There is a number of context definitions in the literature and it is obvious that they are greatly influenced by the target application framework [7]. From an ontological perspective, the concept of context belongs to the class of "moment universal" [18], which means that the context existentially depends on other concepts to exist [8]. Since contexts are subsumed by the concept of property, we should speak of context-of rather than simply context. In this paper, we focus on the context dealing with physical 1 Corresponding Author: Richard Dapoigny, Université de Savoie, LISTIC/Polytech’Savoie, Domaine Universitaire B.P. 80439, F-74944 ANNECY Le Vieux Cedex, France; E-mail: [email protected].

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

37

processes in which the central element is that of a task. As a result, we must speak of the context of an action since it is the intentional core of the task. It is an important subclass of the real-world applications in which any physical process can be modelled by incorporating contextual information. This subclass includes processes such as engineering and control applications, Web engineering, clinical tasks [36] or business processes [1]. Context inherits from a number of well-known characteristics. Context is dynamic and is changing especially in mobile applications. Context information itself can be considered hierarchical, since basic context data can be further mapped into higher-level categories, and so on. Context knowledge is also partial and first order logic is not adequate to fulfill this constraint. The context-aware framework is composed of semi-autonomous software programs known as Agents. These agents represent real-world objects (e.g., people, devices such as RFID tags, GPS, etc.). These agents communicate with each other, deduce goals from context information and automatically activate tasks to achieve these goals. Contexts must be derived to a large extent automatically which argues for modularity. In addition, there are a number of issues that need to be addressed in context aware computing, including how context information is modelled and how we can reason about context information. To cope with these issues, most prominent context models developed in pervasive computing and artificial intelligence, are either logic-based [31,22] or ontology-based [38]. After considering recent comparative studies2 of significant techniques in context modelling [39,33], it appears that, first the ontology-based approaches are the most appropriate and second that logic-based and ontology-based approaches are rather complementary, especially on distributed composition and partial validation. While the ontology representation specifies the rules and relations, the inference engine works on these rules to derive meaningful inferences. The process of deduction and reasoning is highly dependent upon how rules are represented. Logic languages can work on first order logic (FOL) formulae and logic formulas with quantification. However, the core problem with current approaches is not the lack of powerful rules engine, but rather the lack of ways to easily represent real-world scenarios in terms of rules. To cope with this issue, we adopt the framework of intuitionist logic, more realistic than FOL. We suggest a logical framework based on intuitionism and on type theory to represent contexts from an AI perspective. The Intuitionistic Type Theory (ITT) is able to serve as a background for expressing contexts in Artificial Intelligence via Dependent Record Types. We address the problem of partial validation with Dependent Record types while the hierarchical property is expressed with sub-typing. Moreover, ITT is functionally decidable [44]. The resulting context theory integrates both a context logic based on Intuitionistic Type Theory, and a domain ontology to enable the systematic development of context-sensitive applications. The first section discusses the concept of context and summarizes relevant works on context formalization in AI. In section 2, we present the context logical framework through Dependent Record Types in intuitionistic logic including context types and context tokens. The third section relates the domain ontology to the logical framework. In the fourth section, some examples witness the ability of Context Record Types (CRT) to adapt to different domains. Some account of the implementation are given in section 5. The last section discusses major characteristics of CRT and present future ongoing works. 2 based

on general requirements

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

38

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

1. Context in AI 1.1. The need for a Context Model The issue of context has been emerged in various areas of AI, including knowledge representation, natural language processing, and intelligent information retrieval. In contextaware systems, a well-known definition is expressed as follows "a system is contextaware if it uses context to provide relevant information and/or services to the user, where relevancy depends on the user’s task" [17]. Such a definition requires first to categorize explicitly the concept of context. The second issue to address is to clearly define the strong link between contexts and situations. This is highlighted in [22] where three different possibilities are detailed (a fourth one is the combination of the three basic ones). In the first one, a given situation is related to multiple contexts, each being a different approximate of the situation. We argue that these contexts are different because they correspond to different goals that we have in mind, speaking for a context-of definition. In his one-to-one correspondence between situation and context, the author gives an example involving a time evolution of a physical process. Considering first that a physical process has a global goal and second, that this goal can be composed of sub-goals, each of them being achieved successively, it results that, at a given time a single context-of is available (i.e., the context-of the related sub-goal). The third possibility, in which a single context corresponds to many situations reflects the traditional approach of Situation Calculus in which contexts are not used. The last issue relates to the underlying logic. Almost all the work in the literature on context reasoning take the approach of first-order logic (FOL). However, it results in the great difficulty to express partial validation and dynamic change.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

1.2. Related Works Ontology-based approaches are a promising framework to specify concepts and their interrelations [43,25]. They are particularly suitable to project parts of the information describing and being used in our daily life onto a data structure utilizable by computers. The development of context models based on ontologies take advantage of their knowledge sharing, logic inferencing and knowledge reuse capabilities. In [46], the authors have created an upper ontology which captures general features of basic contextual entities and a collection of domain specific ontologies with their features in each subdomain. The CoBrA system [11] provides a set of ontological concepts to characterize entities such as persons, places or several other kinds of objects within their contexts. It implements a broker-centric agent architecture to provide runtime support for context-aware systems. A recent survey of the most significant context modelling approaches [39] argued that ontology-based approaches seem one of the most promising assets for context modelling. In logic-based models, contextual information is introduced, updated and deleted in terms of facts or inferred from a set of rules. In the early ninety, the most significant general foundations for reasoning about contexts have explored logics rooted in the Situation Calculus. In [31], contexts are first class objects and the formula ist(c, p) asserts that the formula p is true in the context c. Lifting rules are able to relate the truth in one context to the truth in another one. In addition, operations such as entering and leaving contexts

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

39

were introduced. This work was the starting point for a formalization in propositional logic [9] extended to first-order languages [10]. In [23,22], the formalization relies on two assumptions, namely the locality principle where reasoning happens inside a context and the principle of compatibility where relationships can occur between reasoning processes in different contexts. The approach known as Multi-context Systems considers a context as a subset of the complete state of an individual. Contexts are seen as partial theories which can interact with each other through bridge rules allowing to infer a conclusion in one context from a premise in another one. More recently, an approach of contexts exploring type-theoretic ideas has been proposed in [40,41]. Three primitive syntactic constructions are derived from type theory, namely identity, functional application and lambda abstraction. Built on a previous work on intensional logic [32], propositions are defined as sets of possible worlds and a context is identified with a set of propositions. In Human Computer Interactions, context is seen as a feature of interaction (a property of information, or of objects) and the proposed a model includes context and activity which are mutually constituent [19].

2. The Context Logical Framework

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

2.1. Intuitionistic Type Theory While the approaches described in section 1.2 cover a wide spectrum of AI modelling, they show some difficulty to manage the intensional aspect (some of them present modality as a solution). One important aspect already underlined in [22], is the locality principle. This principle which states that different portions of knowledge are defined, is taken as a basis in the present work. Moreover, first-order logic is not a natural tool for describing dynamic systems. Then, observing that physical objects are a central element of process activity and building on recent works [5,37,45], we propose a context model based on Intuitionistic Type Theory (ITT) for engineering applications. We consider the context-of the application, that is, the context-of the global goal (since the application is characterized by an objective). The analysis of contextual reasoning requires sound and expressive formalisms. Widely used in Natural Language processing [5,37,12] and in Programming Languages [6,35,14], ITT [29] has been proven to be appropriate to support the linguistic nature of physical situations. We extend these works with the description of contexts. However, contexts are not situations but they are related to them and since contexts describe knowledge extracted from situations, they have a natural expressivity with types. Moreover, ITT provides an intensional type theory for representing most of the features characterizing contexts in an AI perspective. The Intuitionistic Type Theory is able to serve as a background for expressing knowledge in Artificial Intelligence via Dependent Record Types. From a practical point of view, there is a strong motivation for using intuitionistic logic, since it has the existence property. In rough words, intuitionism says that only those mathematical concepts that can be demonstrated, or constructed are legitimate. ITT exploits both intuitionism and theory of types which provides a rich type system in which any property in predicate logic can be interpreted as a type. It is the basis for important works in logic and AI (theorem provers such as Coq, LEGO, ...), functional languages, logics (linear logic, Martin Löf’s type theory, ...). One of the major benefits of Type Theory is the proofs-as-

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

40

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

programs paradigm3 which associates a constructive proof with a program realizing the proven formula [27] (see table 1). Proposition

Set

A&B A∨B

A × B, cartesian product A + B, disjoint union

A→B

A → B, functions from A to B  (AB), cartesian product of a family B(x) of types indexed on values of type A  (AB), the disjoint union of a family B(x) of types indexed on values of type A

(∀x ∈ A)B(x) (∃x ∈ A)B(x) ⊥

∅ Table 1. Proposition as Set

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Another benefit is the computability of any judgement: Intuitionistic Theory of Types is functionally decidable [44]. The most fundamental notion of ITT is the typing judgement a : T classifying an object a as being of type T . Examples of typing judgements are: x : Animate_ind # x is of type Animate_ind s : run(x) # run(x) is a type of a proof In ITT, a useful innovation is the introduction of dependent types and dependent record types [4,28]. Dependent types are types expressed in terms of data. They are able to express more of what matters about data since they are much more flexible than conventional type systems (i.e., they offer a continuum of precision from the basic assertions up to a complete specification of a problem). As suggested in a previous work [16], the concept of context can be expressed as Dependent Record Types including individuals as well as propositions. Context types (what is possible) are distinguished from context tokens (what is the realm). They clearly separate the specification of potential contexts through types with their implementation through tokens. A further attraction is that this subdivision also supports ontological engineering for the specification of the applications during the design step. 2.2. Dependent Record Types In the following, we assume that capital letters denote context types whereas small letters denote context tokens. Definition 1 A dependent record type is a sequence of fields in which labels li correspond to certain types Ti , that is, each successive field can depend on the values of the preceding fields: ⎡

l1 : T 1 ⎢ l2 : T2 (l1 ) C =⎢ ⎣. . . ln : Tn (l1 . . . ln−1 ) 3 also

known as the Curry-Howard isomorphism

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

(1)

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

41

where the type Ti may depend on the preceding labels l1 , ..., li−1 . A similar definition holds for record tokens where a sequence of values is such that a value vi can depend on the values of the preceding fields l1 , ..., li−1 : ⎡

l1 = v1 ⎢ l2 = v2 c=⎢ ⎣. . . ln = vn

(2)

We can also make use of tabular notation to represent records. The empty sequence is a dependent record type and the type Ti is a family of types over the record type l1 : T1 , ..., li−1 : Ti−1 . Assuming that Γ is a valid context4 , we can express the record type formation rules provided that l is not already declared in R: Γ : record − type Γ  R : record − type Γ  T : record − type → type Γ < R, l : T >: record − type

(3)

(4)

We also assume that the general rules of construction for Set and P rop types are valid and that primitive syntactic constructions (i.e., equality, functional application and lambda abstraction) hold (for more details see [29]).

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

2.3. Context Record Types The context theory builds on the works of [2] which assesses that, by contrast with a "world" determining the value of each proposition, a situation must reflects the limited part of reality we perceive and reason about. Basic themes of Barwise’s situation semantics have been incorporated into many frameworks and we also introduce its assessment through Dependent Record Types and Tokens. In the framework of physical processes, we introduce the Context Record Type (CRT) described by a record in which the fields detail the physical knowledge (i.e., objects, their properties and their constraints). Their ability to provide a simple structure that can be reused to specify different kinds of structured semantic objects is a key element of the model. Another important aspect of CRT is that sub-typing is allowed, for example a context token with additional fields not mentioned in the type is still of that type. CRT can range from the context of a single variable involved in a physical equation to the context of a real life situation. In context-aware systems, scenarios often consider spatial and temporal information. Spatial information, such as location, can range from geographical location to information about other objects’ location that are related to the user’s current location. Inside a building for example, sensors provide fine-grained information about the location whereas outside the building location information does not require such accuracy and a satellite-based locating service can be used for tracking the user’s location. The temporal 4 A valid context in type theory is a sequence x : T , . . . x : T such that there is a judgment having it as n n 1 1 left side of a sequent.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

42

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

information, such as time of day and date, are mostly used in triggering actions. Making explicit the temporal component of context allows to disambiguate some contextdependent events and discover connections between them. Obviously, the assignment of time values fulfil real time constraints provided they are not too hard (i.e., the total time consumed for both the acquisition of a time value and the type checking process must be less than the evolution time of the system). Let us consider the following example including both spatial and temporal information. ⎡

x : V ehicle ⎢ y : RegistrationN umber ⎢ ⎢ l1 : GP SLongitude ⎢ ⎢ l2 : GP SLatitude ⎢ ⎢ te : evT ime ⎢ ⎢ tm : maxT ime ⎢ ⎢ q1 : has_identif ication(x, y) ⎢ ⎢ q2 : has_Longitude(x, l1 ) ⎢ ⎣ q3 : has_Latitude(x, l2 ) c1 : has_lowerV alueT han(evT ime, maxT ime)

⎡

x = truck ⎢ y = 2678KX69 ⎢ ⎢ l1 = 12.0987 ⎢ ⎢ l2 = 67.2365 ⎢ ⎢ te = 2/11/06.11 : 33 ⎢ ⎢ tm = 2/11/06.12 : 00 ⎢ ⎢ q1 = p1 ⎢ ⎢ q2 = p2 ⎢ ⎣ q3 = p3 c1 = ct1

CRT are First-Class Objects in which the corresponding typing judgments hold: p1 is a proof of has_identif ication(truck, 2678KX69), p2 is a proof of has_Longitude (truck, 12.0987), p3 is a proof of has_Latitude(truck, 67.2365) and ct1 is a proof that evaluation time is less than the maximum time (2/11/06.11 : 33 < 2/11/06.12 : 00). Since the Curry-Howard isomorphism identifies proofs with programs, it can be used to prove a specification, or in other words, to select which definitions are needed for the specification to work properly.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Manifest Types Pre-defined values can be introduced with manifest types [15]. Definition 2 Given x of type T , x : T , a singleton type Tx is such as: y : Tx iff y = x

(5)

Given a record, a manifest field is a field whose type is a singleton type. We will adopt the notation of [13,21]: ⎡

... r : ⎣l = x : T ...

(6)

Records are also recursive, which means that a value corresponding to a label may itself be a record type. Complex Record Types Manifest fields allow to specify constant values for simple type fields as well as more complex structures such as record types. As a consequence, any kind of constraints may result in a record expression. Let us consider a constraint originated in a given situation. That situation can be drawn as a record in the following way:

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

⎡

43

⎡

x : V ehicle ⎢ v = ⎣ y : RegistrationN umber ⎢ ⎢ p1 : has_identif ication(x, y) ⎢ ⎢x ⎢ ⎣ p1 p2

: record − type : Ind : man(x) : own(x, v)

This type corresponds to the context in which a man owns a vehicle having a given registration number. The ownership is true just in case there is a record of type: ⎡

x : V ehicle ⎣ y : RegistrationN umber p1 : has_identif ication(x, y)

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Intention As discussed in [42], achievement of goals within a process is contextdependent. As a consequence, the notion of context-of is formalized with a relation between the context and the goal type. Context must be related to an intentional concept and an important application field related to this aspect concerns processes such as tasks or human activity. The intention is made more concrete by means of a goal expression (i.e., a proposition). In order to preserve the model structure, this goal is itself a dependent record type. The theory of record types as a part of general type theory allows to define functions and function types providing a version of the typed λ-calculus. The intention can be drawn as a function from a context type to a record type describing the related intended goal resulting from the action on the environment:  λci : Ci .( g : [g1 : ... ) (7) It can be seen as a link in which an agent observing an action acting within the  context type Ci will predict the existence of a goal of type g1 : . . . . Another important aspect of this modelling with CRT is that a context can have any number of fields (there is no upper limit). As a result, a clear understanding of the subtyping concept is required. 2.4. Sub-typing with Contexts The question of sub-typing requires the knowledge of all possible coercions used for a given term and their precise effect, which is untractable in practice. This problem can be avoided by imposing semantic constraints on coercions [4]: this is the case in recordbased subtyping that we shall adopt here. Definition 3 Given two record types C and C  , if C contains at least every label declared in C and if the types of these common labels are in the inclusion relation then C is included in C  which is written: C  C Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

(8)

44

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

The extension of a physical context type C to a context type C  corresponds to the process of getting more information. Since in type theory, the analogue of a proposition is the judgement, we can conclude that the judgement in C is lifted to the judgement in C  . Every record token of type C is also a token of type C  , since it contains components of appropriate types for all the fields specified in C  . Type inclusion and corresponding proof rules generalize record type inclusion to dependent record types and propagate it to all the types within the language. R : record − type Γ  R 

(9)

It follows from eq.9 that any dependent record type is a subtype of the empty record (the record that imposes no constraints). Context by itself doesn’t make sense since it must be related to an intentional concept (it is ontologically speaking, a moment universal)[8]. As a result the context knowledge strongly relates to an ontological framework.

3. The Domain Ontology

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Context modelling abstractions are required to support common understanding and to facilitate communications between agents in distributed environments. A standard representation of context meta-data is necessary for agent-based frameworks in order to guarantee their inter-operability. Semantic ontology can provide such a standard representation. Ontology-based systems are used in current AI research to build common sense in machines, so ontology seems like the obvious choice for making intelligent systems. A recent approach [18] has proposed to characterize the concept of context within an

Figure 1. The Core Ontology.

ontological framework. The authors introduce some concepts such as the "containment Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

45

context" which associates multiple entities within a context, the quality requirement in which entities are bearers of qualities (intrinsic moment), the formal relations between individuals and material relations in which entities share a common property. Despite its interesting definitions of entities and contexts, this conceptual model of context is not sufficient to cover all aspects of context reasoning since the underlying logic is not explicit. Moreover, the context concept described as a moment universal doesn’t formalize the notion of context-of. Alternatively, a context-aware system requires the ability to represent objects and maintain information about their attributes and relationships with other objects. Such a theory can be partially defined through an ontology describing concrete or abstract entities and their relationships [30]. Most Ontology-oriented approaches ignore issues concerning constraints and quantitative relations and focus more on the goals of knowledge sharing [26,11]. The ontology should provide a vocabulary for representing knowledge about the domain and a common understanding of the structure of context information. However, to capture contexts in a pervasive computing environment is a difficult task. Instead of defining a specific "ContextEntity" like in [24], it is preferable to start with

l Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 2. The Context Theory.

basic and simple concepts like entity and relation that are the basis of all ontologies. The main idea is to substitute the context concept in [18] with the more atomic concept of property. As a result we obtain the Core Ontology described in figure 1. The kind of possessive, allows to semantically relate the entity noun to a property noun in a specific "has_" relation. Through the fundamental categories of Entity and Property, all the basic components of the physical world are defined. Above these assumptions, the containment context is nothing else than a context extension (sub-context-of) shifted in the logic, the intrinsic properties and the material relations by "has_" while formal relations are directly expressed by propositions in the logic. Classical subsumption relations and material relations between entities and their properties explicitly conceptualize in a simple way a given application. The resulting upper level ontology does’nt formalize the context, but rather its elementary components, i.e., the entities and their properties. However, the ontological part is not sufficient by itself and must be extended with an appropriate logic. In such a way, a Context Theory can be formulated with a complete model (see figure 2). The first component (Conceptual part) built in the design step defines the ontology of the application and follows the construction rules of the upper level ontology. The second component (logical part) addresses the logical aspect re-

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

46

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

lating the concepts defined in the previous layer through the generation of CRT including entities and propositions. It is only in this part (also built during the design step) that the context appears in a dynamic way. From an extended perspective, the Context Theory presents capabilities either for more applied processes or for a more conceptual approach. Since ontology has been the prevalent representation medium, there is some basis for sticking to ontology.

4. Applications 4.1. Web Services Description with Contexts

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

The first example is extracted from works on WSMO (Web Service Modeling Ontology), an alternative to OWL-S, which provides a formal specification of concepts for Semantic Web Services. WSMO relies on four main elements: ontologies, giving the terminology used by other WSMO elements, Web service descriptions, expressing the functional and behavioral aspects of a Web service, goals that represent user desires, and mediators, which aim at automatically handling inter-operability problems between different WSMO elements. In the WSMO model each concept can have a finite number of concepts that serve as a super-concept for some concept. Being a sub-concept of some other concept in particular, means that a concept inherits the signature of this super-concept and the corresponding constraints. As a result, for each concept C, all attributes of all super-concepts can be applied to instances of C as well. In the example below, a trip in Europe is defined as sub-concept of a general trip. Note that the semantics of predicates symbols is the common one for predicates in First-Order Logics. concept tripInEurope subConceptOf trip origin ofType loc: location destination ofType loc: location departure ofType xsd: dateTime arrival ofType xsd: dateTime definedBy forAll ?x (?x memberOf tripInEurope equivalent ?x memberOf trip and ?x.departure.loc : locateIn = loc:europe and ?x.origin.loc : locateIn = loc:europe).

In this example, the domain of the possible attribute values for origin is restricted to instances of the concept location. The same example described with contexts in Intuitionistic Type Theory will give the following CRT and sub-CRT: Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

47

⎡

orig : location ⎡ ⎢ dest : location orig : location ⎢ ⎢ dep : dateT ime ⎢ dest : location ⎢ ⎢ ⎢ arr : dateT ime ⎣ dep : dateT ime ⎢ ⎣ y : GeographicRegion arr : dateT ime p1 : has_localization(orig, y) TripInGeographicRegion

Trip

The CRT provide a more natural and simple way of expressing knowledge (2 conceptual structures for CRT against 7 for WSMO). Both formalisms make use of type, but the underlying logic of CRT is characterized by a more expressive power. 4.2. The GLM Problem

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Alternatively, the example, originally proposed by McCarthy, concerns an air travelling domain in which a traveller (named John) must plan a Glasgow-Moscow flight. He knows that there are two successive flights, one from Glasgow to London and another from London to Moscow. The objective is to formalize the reasoning process in which the traveller may be in Moskow after two flights unless he loses the ticket in London. The context model must cover the lost of a ticket in London. The first step is to define the goal to achieve, that is the proposition g = At(John, M oskow). This goal can be divided in two subgoals, i.e. g1 = At(John, London) and g2 = At(John, M oskow) respectively related to the context types C1 and C2 . In this simple example, the two goal

Figure 3. Excerpt of the GLM ontology

tokens stem from a single goal type, i.e., At(e1 , e4 ) in which e1 is of type P erson and e4 Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

48

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

of type location. Therefore, a context type C1 can be easily constructed from the domain ontology. Entities which range from e1 to e4 can be either intangible entities (e.g., e3 and e4 ) or spatial entities (e.g. e1 and e2 ). With e1 = John, e2 = t0015, e3 = BA515 and e4 = M oskow, an assignment like p3 = w corresponds to the typing judgement w : has_Destination(BA515, M oskow). It is the present situation which makes true (or not) the relevant context of a given goal (see section 2.3).

T yp(c)

e1 e2 p1 e3 p2 e4 p3 ...

: : : : : : : :

P erson T icket own(e1 , e2 ) F light has_F light(e2, e3 ) Location has_Destination(e3 , e4 ) ...

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

The context propositions derived from the ontology can describe intrinsic properties as well as relational properties which relate two (or more) entities through their property. If the context is valid, after the two flights the traveller will be at Moskow but if he loses its ticket at London, the plan will fail since p1 is false and then, the context token c1 is no longer valid. Figure 3 shows the conceptual model for an agent which implements the GLM example. The difficulty with this problem is that the number of possible obstacles to the plan such as the loss of ticket, crew on strike, ... can be huge for high-level goals and it is not possible to take all of them into account. But if we consider elementary goals, it becomes possible to attach the possible obstacle(s) with them (see [8]). The consequence is that a mechanism of goal composition with their related contexts must be investigated.

4.3. The Smart Phone

This simple scenario is extracted from [20]. Let us assume an hotel in which rooms are equipped with smart phones, that is phones which are able to receive contextual information from sensors (or other devices), to aggregate them and to reason about contexts in order to make decisions. The Smart Phone is supposed to have sufficient computing power for context processing and reasoning. In this scenario, the user is inside the room and performs some activity. The smart phone receives an incoming call which is not an emergency communication and it must decide whether it must ring, produce only a vibration or forward the call to a voice mail. These actions depend strongly on the user activity and, for that purpose, the smart phone must aggregate information from the environment such as running electric devices, motion detection, physical state (temperature, brightness, ...). In other words, the smart phone has to define the context of its task. Since it have to deal with three tasks, three contexts types are attached to these tasks as follows:

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

49

⎡

x : hotel − room ⎢ y : user ⎢ ⎢ w : call ⎢ ⎢ p1 : has_location(y, x) C1 = ⎢ ⎢ p2 : has_LowP riority(w) ⎢ ⎢ z : smart − phone ⎢ ⎣ s : beep p3 : has_activity(y)

→



g1 : provide(c1 .z, c1 .s)

⎡

x : hotel − room ⎢ u1 : user ⎢ ⎢ u2 : user ⎢ ⎢ w : call ⎢ C2 = ⎢ ⎢ p1 : has_location(y, x) ⎢ p2 : has_LowP riority(w) ⎢ ⎢ z : smart − phone ⎢ ⎣ s : buzz p3 : Interact_with(u1 , u2 )



→

g1 : provide(c2 .z, c2 .s)

⎡

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

x : hotel − room ⎢ y : user ⎢ ⎢ w : call ⎢ ⎢ p1 : has_location(y, x) C3 = ⎢ ⎢ p2 : has_LowP riority(w) ⎢ ⎢ z : smart − phone ⎢ ⎣ v : phone − message p3 : Asleep(y)

→



g1 : f orward(c3 .z, c3 .v)

In addition, information incoming from the environment may extract context record types from libraries in the following way: • information from an electric device can populate the fields: ⎡

x : light ⎣ q1 : has_stateOf f (x) ... : ... • information from a tv set may feed some fields such as: ⎡

x : tv ⎢ q1 : has_stateOn(x) ⎢ ⎢ y : channel ⎢ ⎢ v : volume ⎢ ⎢ q2 : has_selectedChannel(x, y) ⎢ ⎣ q3 : has_stateOf f (v) ... : ...

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

50

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

• information from infra-red detectors or infrared cameras may validate the fields: ⎡

u1 : user ⎢ r : hotel − room ⎢ ⎢ q1 : has_activity(u1 ) ⎢ ⎣ q2 : has_multipleU sers(r) ... : ... The result of sensor information aggregation provides a vector of knowledge describing the current situation. From this situation, context types are checked and output the appropriate proposition (i.e., the goal proposition). Excerpts of such context types are: ⎡

C10

u1 : user ⎢ x : light ⎢ ⎢ q1 : has_stateOf f (x) ⎢ =⎢ ⎢ r : hotel − room ⎢ q2 : has_singleU ser(r) ⎢ ⎣ t : tv q3 : has_stateOf f (t)

→



g1 : Asleep(c10 .u1 )

⎡

C11

u1 : user ⎢ r : hotel − room ⎢ ⎢ q1 : has_activity(u1 ) =⎢ ⎢ q2 : has_multipleU sers(r) ⎢ ⎣ x : light q3 : has_stateOn(x)

→



g1 : Interact_with(c11 .u1 , c11 .u2 )

Let suppose now that the following items are checked in a particular situation: Copyright © 2007. IOS Press, Incorporated. All rights reserved.

⎡

c10 .u1 = Jack ⎢ c10 .x = l#5677 ⎢ ⎢ c10 .q1 = has_stateOf f (x) ⎢ ⎢ c10 .r = #1345 ⎢ ⎢ c10 .q2 = has_singleU ser(r) ⎢ ⎢ c10 .t = P anasonic#1322 ⎢ ⎢ c10 .q3 = has_stateOf f (t) ⎢ ⎢ c3 .x = #1345 ⎢ ⎢ c3 .y = Jack ⎢ ⎢ c3 .w = #217884 ⎢ ⎢ c3 .p1 = has_location(y, x) ⎢ ⎢ c3 .p2 = has_LowP riority(w) ⎢ ⎣ c3 .z = Alcatel#BSX120 c3 .v = #4365499 The six first items validate the context C10 , which itself is a proof that Asleep(c10 .u1 ) is valid. As a result, these six items can be replaced by Asleep(c10 .u1 ) and since c10 .u1 and c3 .y correspond to the same entity token, the context c3 becomes itself valid. Finally,

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

51

Figure 4. Entities of the scenario

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

the action leading to the goal proposition f orward(c3 .z, c3 .v) is selected. Of course, this scenario is only supposed to explain the mechanism of CRT checking and in real case more information are added. Notice that the numbers identifying entities can also be replaced by URIs.

5. Implementation features The implementation must satisfy two sets of constraints, one resulting from the design phase and the other from the run-time phase. The design phase requires, i) to check the semantic validity of concepts, ii) to check the syntactic CRT composition. In the runtime phase, CRT are compared to the current situation expressed as a list of propositions and entities in the spirit of [42]. For the first phase, the ontology ResearchCyc is used (details are given below). For the second phase, a Lisp checker acts as theorem prover. Concerning the first phase, we make use of Augmented Transition Networks (ATNs) that are a form of parser widely used for parsing natural language. The ATN system is implemented as a front-end with the database. With an ATN-driven interface attached to the system, then instead of making a formal query, users can ask questions in a constrained form of English. As a result, the present tool provide better compatibility with NLP-based systems. The CRT modelling relies on the closure property which i) provides a powerful way of representing dependent variables in Lisp, ii) facilitates the design of DRT structures

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

52

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

l

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 5. The Context Record Types formation.

which possess a state. The DRTs type constructors used in the tool are monads that capture various notions of sequential computation. Their main benefit holds in the restriction of the programming skills with functions instead of global variables. Macros Lisp generate DRT structures which are stored in a hash table and inside an ontology (in RDF language). The ontology which is expressed in RDF or OWL describes the sub-typing relations between DRTs. For that purpose, the Allegrograph tool is used as a high performance Datawarehouse allowing a high level of memory space and high speed transactions. With that tool, data are stored as RDF triples and OWL. The resulting knowledge base is competitive, autonomous and sharable between applications. Since ontologies give programs the ability to access a shared knowledge across a network, programs can modify their knowledge at run-time. The application ontologies whose restricted validity is limited to the task execution are a suitable tool for the specifications of context types with their relations. A dynamic ontology has been designed including an intentional definition of concepts and their relations with operators allowing for the composition of new concepts and their relations. A main benefit of this approach is to avoid the combinatory explosion inherent to static approaches. The Knowledge Base consists of terms - the vocabulary of Cyc - and assertions which relate these terms. These assertions include both simple ground assertions and rules. The representation of context types in Cyc permits the programs to share a concept base and allows inference and reasoning mechanisms requiring this base. The resulting software architecture is reported on fig. 6. Notice that consistency of what people put in the contexts, however, is the responsibility of the user. The programming language is powerful enough to let people get hung up in a loop, and the practical reasoning system is powerful enough to let people say things that are inconsistent.

6. Conclusion As underlined in [3], a common (and basic) functionality of AmI systems requires "a collection of problem solving components available that can be instantiated for the par-

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

53

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

l

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 6. The Global Architecture.

ticular problem". To solve that problem, the present approach introduces records and dependent record types and details how they can be used to represent and relate context as pieces of knowledge. In comparison with the related works, the prominent feature of this approach is that the context is checked by theorem proving. We have suggested a context theory as a framework for representing contextual information and inter-contextual information flow. Structuring facilities are conceptually grounded within a core ontology according to conceptual theories of philosophy and cognitive sciences. Checking the existence of entities with their properties (i.e. propositions) at run-time would make this approach suitable for context-aware applications. Since ITT with dependent record types has been widely applied in NLP, this approach may further facilitate linguistic applications of the theory of contexts. The use of types is both clear and simple while at the same time it is sufficiently expressive to be related with a highly structured ontology. The acquisition and processing of contextual information can be realized much easier in the future version since every information item will be referenced and accessed using URIs. Representing information with standard languages on the basis of XML is also a significant advantage for distributed applications. This paper has also given some account of key features of CRT such as dependent types, propositions as types and types as objects. Instead of defining ontological relations between entities and contexts, the CRT introduces only two sorts (i.e., entities and propositions) and expresses all other relations through context extension. The major interest in the CRT comes from their ability to express context dependencies in a dynamic way through context extension and their scalability through recursion. A prototype has been developed to build context types (design step) and a Lisp_like engine allowing dynamic typing at run-time. This architecture is able to support the whole matching process of

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

54

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

CRT with situations. However, since the amount of the available contexts and the size of ontology models can be huge it is necessary to reduce the search space during problem solving. We will further explore the distributed extension of this approach through information flow (IF) mechanisms.

References [1] [2] [3] [4] [5] [6] [7] [8]

[9] [10] [11] [12]

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

[13] [14] [15] [16]

[17]

[18]

[19] [20]

[21]

P. Balabko and A. Wegmann: "Context Based Reasoning in Business Process Models", Procs. of the IEEE Int. Conf. on Information Reuse and Integration, 2003, 120–128. J. Barwise: "Scenes and other Situations", Journal of Philosophy, 1981, 77, 369–397. M. Berger, F. Fuchs and M. Pirker: "Ambient Intelligence - From Personal Assistance to Intelligent Megacities", this volume, IOS Press, 2007. G. Betarte: "Type checking dependent (record) types and subtyping", Journal of Functional and Logic Programming, 2000, 10(2), 137–166. P. Boldini: "Formalizing Context in Intuitionistic Type theory", Fundamenta Informaticae, 2000, 42, 1–23. A. Bove and V. Capretta: "Nested General Recursion and Partiality in Type Theory", TPHOL, R.J. Boulton and P.B. Jackson eds., 2001, LNCS 2152, Springer, 121–135. P. Brézillon and S. Abu-Hakima: "Using Knowledge in its Context", Report on the IJCAI’93 Workshop, AI Magazine, 1995, 16(1), 87–91. P. Brézillon and J.-Ch. Pomerol: "Modeling and using context for system development: Lessons from experiences", Journal of Decision Systems, P. Humphreys and P. Brézillon Eds, Decision Systems in Actions, 2001, 10(2), 265–288. S. Buvac, V. Buvac and I.A. Mason: "Metamathematics of Contexts", Fundamentae Informaticae, 1995, 23(3), 412–419. S. Buvac: "Quantificational logic of context", Procs. of the 13th National Conference on Artificial Intelligence, 1996. H. Chen, T. Finin and A. Joshi: "Using OWL in a Pervasive Computing Broker", Procs. of Workshop on Ontologies in Open Agent Systems (AAMAS’03), 2003. R. Cooper: "Mixing Situation Theory and Type Theory to Formalize Information States in Dialogue Exchanges", in Procs of TWLT 13/Twendial ’98: Formal Semantics and Pragmatics of Dialogue, 1998. R. Cooper: "Records and Record Types in Semantic Theory", J. of Logic and Computation, 2005, 15(2), 99–112. C. Coquand and T. Coquand: "Structured type theory", Workshop on Logical Frameworks and Metalanguages, 1999. T. Coquand, R. Pollack and M. Takeyama: "A Logical Framework with Dependently Typed Records", Fundamenta Informaticae, 2005, 20, IOS Press, 1–22. R. Dapoigny and P. Barlatier: "Dependent Record Types for Dynamic context representation", Research and Development in Intelligent Systems XXIII Procs. of AI-2006, The Twenty-sixth SGAI International Conference on Innovative Techniques and Applications of Artificial Intelligence, Max Bramer, F. Coenen and A. Tuson eds., Springer, 23, 2007, 419 p. Anind K. Dey and Gregory D. Abowd: "Towards a Better Understanding of Context and ContextAwareness", Procs. of the CHI 2000 Workshop on The What, Who, Where, When, and How of ContextAwareness, 2000. P. Dockhorn-Costa, J.P. A. Almeida, L. F. Pires, G. Guizzardi and M. van Sinderen: "Towards Conceptual Foundations for Context-Aware Applications", Procs. of the AAAI’06 Workshop on Modeling and Retrieval of Context, AAAI Press, 2006, 54–58. P. Dourish: "What we talk about when we talk about context", Personal and Ubiquitous Computing, 2004, 8(1), 19–30. J. Euzenat, F. Ramparany and J. Pierson: "Gestion dynamique de contexte pour l’informatique pervasive", Procs of the 15th French Conference on Pattern Recognition and Artificial Intelligence (RFIA’06), 2006, Tours (France), 113–123. J. Ginzburg: "Abstraction and Ontology: Questions as Propositional Abstracts in Type Theory with Records", Journal of Log. and Comput., 2005, 15(2), 113–130.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

R. Dapoigny and P. Barlatier / Towards a Context Theory for Context-Aware Systems

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

[22]

55

F. Giunchiglia: "Contextual Reasoning", Istituto per la Ricerca Scientifica e Technologica, 1992, report 9211-20. [23] C. Ghidini and F. Giunchiglia: "Local Models Semantics, or Contextual Reasoning = Locality + Compatibility", Artificial Intelligence, 2001, 127(2), 221–259. [24] T. Gu, X.H. Wang, H.K. Pung and D.Q. Zhang: "An Ontology-based Context Model in Intelligent Environments", In Procs. of Communication Networks and Distributed Systems Modeling and Simulation Conference (CNDS 2004), 2004, 270–275. [25] N. Guarino: "Formal Ontology and Information Systems", Procs. of the 1st International Conference on Formal Ontologies in Information Systems, 1998, 3–15. [26] A. Held, S. Buscholz and A. Schill: "Modeling of Context Information for Pervasive Computing Applications", Procs. of the 6th conf. on Systemics, Cybernetics and Informatics (SCI), 2002. [27] W. A. Howard, J.P. Seldin and J.R. Hindley: "To H.B. Curry: Essays on Combinatory Logic, Lambda Calculus and Formalism", The formulae-as-types notion of construction, Academic Press, 1980, 479– 490. [28] A. Kopylov: "Dependent Intersection: A New Way of Defining Records in Type Theory", in Procs. of the 18th Annual IEEE Symposium on Logic in Computer Science, 2003, 86–95. [29] P. Martin-Lof: "Constructive Mathematics and Computer Programming", Logic, Methodology and Philosophy of Sciences , 1982, 6, 153–175. [30] C.J. Matheus, M.M. Kokar and K. Baclawski: "A core Ontology for Situation Awareness", Procs. of the Sixth International Conference of Information Fusion , 2003, 1, 545–552. [31] J. McCarthy: "Notes on Formalizing Context", Procs. of the 13th Int. Joint Conf. on Artificial Intelligence, 1993, 555–560. [32] R. Montague: "Pragmatics and intensional logic", Synthèse, 1970, 22, 68–94. [33] H. Mügge, T. Rho, D. Speicher, J. Kuck and Armin B. Cremers: "Towards an Infrastructure for ContextSensitive Intelligence", SAKS Workshop on Context-sensitive Intelligence, 2006. [34] P. Ozturk and A. Aamodt: "A context model for knowledge-intensive case-based reasoning", International Journal of Human Computer Studies, 1998, 48, 331–355. [35] L.C. Paulson: "The Foundation of a Generic Theorem Prover", Journal of Automated Reasoning, 1989, 5, 363–397. [36] M. Peleg, S. Tu, J. Bury, P. Ciccarese, J. Fox, Robert A. Greenes, R. Hall, Peter D. Johnson, N. Jones, A. Kumar, S. Miksch, S. Quaglini, A. Seyfang, Edward H. Shortliffe and M. Stefanelli: "Comparing computer interpretable guideline models: A case-study approach", Journal of the American Medical Informatics Association, 2003, 10(1), 52–68. [37] A. Ranta: "Grammatical Framework: A Type-Theoretical Grammar Formalism", Journal of Functional Programming, 2004, 14(2), 145–189. [38] T. Strang, C. Linnhoff-Popien and K. Frank: "CoOL: A Context Ontology Language to enable Contextual Interoperability", Procs. of DAIS 2003 LNCS 2893, 2003, 236–247. [39] T. Strang and C. Linnhoff-Popien: "A Context Modeling Survey", Sixth International Conference on Ubiquitous Computing (UbiComp2004), 2004, 34–41. [40] R.H. Thomason: "Representing and Reasoning with Context", Procs. of the International Conference on Artificial Intelligence and Symbolic Computation, 1998, LNCS 1476, Springer, 29–41. [41] R.H. Thomason: "Type theoretic foundations for context, part 1: Contexts as complex type theoretic objects", Modeling and Using Contexts, Paolo Bouquet et al. eds., Springer, 1999, 352–374. [42] Roy M. Turner and Robert A.G. Stevenson: "ORCA: An adaptive, context-sensitive reasoner for controlling AUVs", Procs. of the 7th Int. Symp. on Unmanned Untethered Submersible Technology (UUST’91), 1991, 423–432. [43] M. Uschold and M. Grüninger: "Ontologies: Principles, methods, and applications", The Knowledge Eng. Review, 1996, 11(2), 93–155. [44] S. Valentini: "Decidability in Intuitionistic Type Theory is functionally decidable", Mathematical Logic, 1996, 42, 300–304. [45] J. Villadsen: "Multi-dimensional Type Theory: Rules, Categories, and Combinators for Syntax and Semantics", Int. Workshop on Constraint Solving and Language Processing, 2004, H. Christiansen et al. eds., 160–165. [46] Xiao H. Wang, T. Gu, Da Q. Zhang and Hung K. Pung: "Ontology Based Context Modeling and Reasoning using OWL", Procs. of the 2nd IEEE Conference on Pervasive Computing and Communications, 2004, 18–22.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

56

Advances in Ambient Intelligence J.C. Augusto and D. Shapiro (Eds.) IOS Press, 2007 © 2007 The authors and IOS Press. All rights reserved.

An Active Classification System for Context Representation and Acquisition Fulvio MASTROGIOVANNI 1 , Antonio SGORBISSA and Renato ZACCARIA Laboratorium DIST, University of Genova, Italy Abstract. This work deals with a distributed knowledge representation and active information fusion system suitable for use in Ambient Intelligence applications. The architecture - which is modeled as a multi-agent system - exploits a sound formal model to relate numerical data to symbolic representations, thus being able to reason about predicates, situations and contexts. In particular, agents collaborate to perform an intelligent multi-sensor data fusion according to the guidance of an active classification layer. Experimental results performed both in simulation and in a real set-up are discussed with respect to a number of implications and future directions related to the system approach. Keywords. Knowledge Representation, Information Fusion, Symbol Grounding.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Introduction During the past few years, many research efforts were undertaken to improve the quality of life and to foster the independent living of elderly and people with special needs (for a survey, see [9] and the references therein). These goals involve the integration of technologies related to many disciplines, such as remote health care monitoring [4,16,12] and intelligent surveillance and control [20,8], with several research fields, as knowledge representation [2,3], man-machine interfaces [21], etc. Among the proposed approaches, the Ambient Intelligence (AmI) paradigm is particularly interesting, as it introduces intelligent spaces, i.e., systems responsible for distributed information acquisition and representation, context management, and context-based user interaction through intelligent interfaces [10,32]. The semantic information associated with this process is of the uttermost importance either for an intelligent monitoring system or for a human caregiver to infer the user status and to provide assistance when needed. In order for such a system to be proactive and not intrusive, sound knowledge representation and data fusion techniques must be made effective. The aim of information fusion is twofold: whereas contradictions arising from different information sources must be filtered away, the useful information content acquired by heterogeneous sources is to be maximized, so that relevant situations and events can be inferred. Hovewer, data fusion can not operate in isolation: on the contrary, it must be supported by a priori knowledge which guides data interpretation (i.e., one or more ontologies), and it must 1 Corresponding

Author: Fulvio Mastrogiovanni, Laboratorium DIST, University of Genova, Via Opera Pia 13, 16145, Genova, Italy; E-mail: [email protected]. Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

57

cooperate with services providing data acquisition and feature extraction. In particular, a challenging task is to perform fusion of symbols originating from numerical values, thus facing the symbol grounding problem [15]. Several data fusion approaches have been designed and successfully used in different applications reported in literature. The extended JDL (Joint Directors of Laboratories) model [36,35,24] hypothesizes that the information acquisition process can be exemplified using a 5-layer architecture, ranging from information acquisition to learning and process refinement. With respect to this model, current AmI systems address numerical data fusion using techniques which are largely derived from the bayesian framework, such as Kalman filters [1], Markov networks [23,26,37], probabilistic principal component analysis [29], the Dempster-Shafer theory [11] or sub-symbolic techniques, such as Fuzzy Logic [8,14], thus giving up the use of symbolic (and semantic) information. Nonetheless, fusion of symbolic information has been addressed in various research fields. Among other applications, it has been used to integrate large databases, to coordinate automatic reasoning systems [5,6] or to merge structured descriptions of uncertain information [18,19]. In logic-based information fusion, the following constraints are usually assumed:

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

• The information sources are supposed to be independent from each other: this simplification allows the system to avoid checking in advance pairwise consistency of data sources, which must be considered by information fusion. • The information provided by different sources is given the same priority, which is time-independent: this assumption is violated in real scenarios, where the reliability and the quality of the information provided by sensors depend both on their characteristics and on the algorithms used for data processing. At the same time, recent fundamental achievements in symbolic cognitive architectures - specifically related to linguistic cognition - have given new insights into developing interfaces between numerical and symbolic information. [34] presented an Integrated Connectionist/Symbolic (ICS) cognitive architecture, where symbolic processes mirror low level, numerical, activities. In our case, we focus on the problem of active monitoring elderly and persons with special needs: in particular, we want to design a pervasive system which is responsible for supervising their activities, monitoring their actions and reacting to emergencies whenever they arise. Furthermore, the system is expected to feed man-machine interfaces designed for caregivers or remote relatives. In this work we envisage a multi-agent cognitive architecture which is able: (i) to acquire numerical information from distributed sources and to build corresponding symbolic representations according to a sound formal model; (ii) to define groups of symbolic fusion operators which are mapped onto the system architecture; (iii) to fuse base symbols, thus being able to reason about predicates, situations and contexts; (iv) to actively select the sources of information needed to classify current contexts, according to some predefined pattern. Specifically, we focus on the architectural design which is at the basis of our work, as it turns out to be very important for the cognitive processes involved in a typical AmI scenario. The contribution of this work is to provide a framework to simplify acquisition of base symbols and to provide a method to build and organize complex structures. We believe that a promising approach to system validation is to be searched in supervised learning of relations among concepts networks: this is not addressed here.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

58

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

The paper is organized as follows. Section 1 introduces the main concepts related to our architectural approach. In Section 2 we describe the knowledge acquisition process with respect to the JDL architecture detailing the symbolic level. In particular, we focus on the interface between numerical and symbolic information, on the knowledge representation system and on the reasoning subsystem. Next, actual implementation and experimental results are presented and discussed. Conclusion follows.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

1. A Distributed Approach to Data Fusion An Intelligent Space can be easily represented as a distributed system where several cognitive entities interact to manage the information fusion process. In AmI, such a process is expected to deal with heterogeneous information, and - in particular - to support numerical and symbolic data fusion. As a consequence, proper design choices have been made to interface numerical and symbolic information. In order to manage such a wealth of heterogeneous data, we adopted a fully qualified multi-agent approach, able to guarantee consistent interfaces either among agents or between agents and external applications and devices, such as web interfaces and touch screens. Therefore, we satisfied two important objectives: (i) the system is scalable and extensible, despite the availability of heterogenous technologies to be integrated; (ii) the overall data fusion process is mapped onto the system architecture according to the roles played by cognitive agents. In the following discussion we adopt the FIPA abstract architecture [13] to describe the involved agents and their relationships. An Intelligent Space is a collection of agents performing different tasks. Each agent is provided with a name, a locator and one or more attributes. name is used to univocally identify an agent. locator describes a protocol used to communicate with the agent itself. Finally, we identify three attributes: (i) the agent capabilities, i.e., a vector of goals which the agent can contribute to achieve; (ii) the vector of input message types, i.e., the information needed by the agent to operate; (iii) the vector of output message types, i.e., the information produced by the agent. Each attribute vector is stored in an agent-entry which is managed by an agent-directory-service. Agents can aggregate in more complex entities which are characterized by common goals, defined as the result of many cooperating agents. With respect to a higher level of complexity, groups of agents can be modeled as macroagents and, as such, they are possibly part of larger groups. When an agent is looking for a particular information, it searches through the agent-directory-service for selected message types produced by other agents: once the resource has been selected, a communication link is established by instantiating a message-transport-service. Agents are able to communicate through messages. A message is a key-value tuple encapsilating a content: i.e., a data structure (possibly with numerical as well as symbolic information) which is grounded with respect to an ontology. In particular, this information is stored in a knowledge-base, which is represented within a content-kb-service. In our implementation, the ontology is maintained by a particular agent, which exposes the status of its representations according to a hierarchical fashion. Finally, agents can be roughly classified into two disjoint groups: whereas device-agents manage resources for the system, such as sensors for data acquisition, actuators or interfaces towards other software applications, cognitive agents perform computational or abstract reasoning to guide the overall system behavior.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

59

device-agents manage groups of distributed services which are then made available to cognitive agents. In the context of AmI systems, these services deal with either hardware or software entities, which provide cognitive agents with up-to-date information (e.g., hardware devices such as smoke or light sensors, or software interfaces able to collect data from other systems) or with the ability to act in the real world (e.g., opening or closing doors when needed). They are able to perform a preliminary analysis of the collected information and to react to anomalies (e.g., checking if numerical values provided by sensors are over a given dangerous threshold). In particular, Each service is characterized by a name, a type (e.g., door or camera) and a locator. From the computational perspective, device-agents can be modeled as light-weight processes, thus being executed on resource-constrained devices. On the opposite, cognitive agents are software entities applying algorithms and control rules to incoming data in order to ultimately issue low level references back to device-agents. They can be organized in groups collectively performing tasks related to data filtering, feature extraction and knowledge representation, thus being able to perform numeric and symbolic information fusion, logic inferences and planning. The ontology service is managed at this level: a proper cognitive agent maintains a collection of different ontologies, each one focused on a particular aspect of the actual architecture or the current scenario. Cognitive agents usually require more computational power than device-agents. Hovewer, they should guarantee real-time responsiveness. It is worth mentioning that the approach to data fusion presented here originates from the notion of Artificial Ecosystem [28], an architecture for Robotics applications developed by the authors which - during the past years - paved the way for interesting research (e.g., see [25]).

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

2. The Data Fusion Process The JDL extended model is a 5-layer architecture specifically designed to deal with heterogeneous information. In the following paragraphs, the JDL model is used as a framework to discuss the information fusion capabilities of the proposed architecture. From a multi-agent perspective, agents belonging to different JDL levels are indistinguishable: in general, agents functionally assigned to a JDL level exploit services which are provided by agents belonging to lower levels. 2.1. Level 0: Sub-object Assessment According to the JDL model, Level 0 involves: (i) acquiring, filtering and processing raw data in order to extract feature based information and (ii) imposing references to actuators. Therefore, Level 0 is related to numerical information processing. When symbolic information is not available within a system, designers are usually forced to consider complex data in order to infer as much information as possible from numerical features. The proposed architecture takes the opposite approach, i.e, to consider simple data and to infer information through semantic structures. Level 0 functionalities are achieved through a tight coupling between services (encapsulated in device-agents) and cognitive agents. Usually, a suitable group of agents can be arranged according to the goals of Level 1. Each service makes one or more ac-

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

60

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

tions available, according to its type. Therefore, a device-agent refers to these actions to retrieve information from one or more base services or to assign them with numerical references to be issued to actuators. This is accomplished through the exchange of proper kinds of messages. Once data has been collected, cognitive agents acquire raw information from device-agents in order to perform their tasks. Consider the following example, which describes a typical scenario within systems considering only numerical information: a 2D user tracking system based on camera images. If we would like to implement such a system, Level 0 could involve the fulfillment of a well-specified goal, i.e., to obtain bounding boxes features from image data. The system is composed by two physical devices (i.e., the two cameras), which are modeled as services: namely camera-1-service and camera-2-service. Suppose that they expose their current image through message passing, e.g., camera-1-msg and camera-2-msg, implemented as HTTP streams. camera is a device-agent designed to receive camera-1-msg and camera-2-msg using a transport-message. Once the two images are collected, this information could be passed to bbe (where bbe stands for “bounding boxes extractor”), which implements algorithms for filtering - if necessary - raw images and extracting bounding box features. These features are then submitted to another agent, da (where da stands for “data association”), which goal is to perform data association across different images, thus being able to provide Level 1 agents with robust information. For example, da posts an agent-entry declaring itself as a producer of user-pos-msg, i.e., the measurement of user position according to camera images. Obviously, the error probabilities associated with da and user-pos-msg are prone to big errors and even faults. Here, a completely different approach would be preferred: avoiding the use of rich sensors and complex algorithms, the same results could be achieved using several, poor sensors providing information characterized by a low error probability, to be integrated using semantic structures.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

2.2. Level 1: Object Assessment The goal of Level 1 is to arrange the information acquired by Level 0 agents in models and templates which relate each information either to numerical structures or to a properly defined set of symbols. The first option leads to numerical data fusion techniques, while the second involves the use of a well-defined theory to face the symbol grounding [15]. Still unsolved in its general formulation, the symbol grounding problem arises whenever a symbolic component is added to an artificial system. Within this architecture, ontology is responsible for managing symbolic information fusion, by expliting three services. Numerical data fusion and data association techniques do not deserve further investigation. It suffices to note that, with respect to the introduced 2D tracking problem, we could add the following agents: (i) a kalman-filter, which receives messages from da; (ii) a motion-model, used to guess the user position between measurements. Collectively, these agents can track a user within a 2D planar environment. The opposite approach is considered in this work: i.e., information fusion is delegated to ontology. This architecture deals with symbolic knowledge representation and data fusion by introducing a knowledge base represented by a Description Logic (DL). The knowledge base is encapsulated within an is-kb-service (where is and kb stand for, respectively, “intelligent space” and “knowledge base”) which is handled by

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

61

ontology. ontology is thus responsible for the management of different kind of knowledge: it provides access to ontologies for intelligent space modeling, message content and data interpretation. Furthermore, it is used to syncronize the access to the different representations. According to the FIPA abstract architecture specifications, each ontology can be accessed in a hierarchical fashion: e.g., intelligent space-related concepts are accessed through ontology.is-kb. A Description Logic consists of a Terminology Box (TBox), representing concepts, descriptions and relationships (i.e., roles) among concepts, and an Assertional Box (ABox), describing the actual scenario using the ontology provided by the TBox. ontology allows for symbolic information fusion by coordinating the behavior of three knowledge base related services: (i) ontology.is-kb is a representation of the intelligent space, i.e., of “objects” and devices belonging to the physical world; (ii) ontology.content-kb describes the meaning and the structure of data associated with messages: this provides the system with the ability to reason upon the behavior of different agents, for they are represented as well in ontology.content-kb; (iii) ontology.context-kb, a model for context management, used to reason upon concept instances (henceforth called “predicates”, “situations” and then “contexts”) from base symbols corresponding to sensor data. The first two services perform object assessment, while the latter is aimed at situation assessment: therefore, it is described in Section 2.3. The resulting ABox must be updated in real time, assuming the availability of agents providing ontology with up-to-date information, which is to be translated in symbols. In the following section, we describe first ontology.is-kb, the representation of the intelligent space we adopted; next, we discuss ontology.content-kb, detailing how message-contents are given a semantic meaning; then, we detail the interface between numerical and symbolic information, adopting a formal model inspired by [34]; finally, we describe how these different parts cooperate to implement symbolic information fusion for object assessment.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

2.2.1. Modeling Intelligent Spaces For the sake of brevity, we describe only a subset of the concepts defined within the ontology.is-kb.tbox. Nonetheless, they should provide the reader with an exhaustive idea about the structure of the TBox itself. The corresponding ontology.is-kb.abox depends on the actual scenario. Concepts and relationships are defined as follows: • Entity is a base concept for Object, User, Action and Area, among others. It is provided with a couple of roles: the first is name, used to unambiguously identify each entity; the second is description, a string used to provide user interfaces or other applications with brief descriptions of the ontology representations. • Object is introduced to model interesting objects which are present within the intelligent space. In addition to the roles inherited from Entity, it is characterized by a role called location. Object subsumes two concepts: Furniture (used to detail the furniture which is present within the intelligent space, e.g., Stove or Table) and Device. • Device models physical and virtual devices enriching the intelligent space. They are managed by services in a multi-agent scenario, and can be specified as being instances of either Sensor or Actuator. A Sensor is characterized by its scope, i.e., a collection of Areas. Modeling an Actuator deserves a special attention:

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

62

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

whenever possible, we augment the Actuator description with a simple discretetime model of its behavior, which can be used to infer possible faults (see Section 2.4). For each furniture which state is to be monitored (e.g., windows, doors, etc.), a corresponding Device is introduced (e.g. WindowDevice or DoorDevice). • The User concept models users inside the intelligent space. It is characterized by two roles, namely name (which is inherited by Entity) and location. In the case of users, the name is very significant as it allows the user identification by matching, e.g., RFID readings. location is also very important in the context of location-based services. Each User performs Actions, which are represented by ontology provided that the system is able to recognize them. • Action is a concept modeling either user actions detected by the system or actions purposively performed by the system itself. It is characterized by two roles: name (in this case enumerating action types) and performed-by (filled by instances of User or IntelligentSpace). • The Area concept is defined as a polygonal region, which is characterized by its vertices. An Area contains one or more Objects. This concept is of the uttermost importance in order to provide the user with location-based services, or to infer patterns of user activities (see for example [30]). The modeling of the environment topology is an interesting research field in its own right: here, we adopt an approach which is very close to [33]. Area is the building block used to represent the topology of the intelligent space. ToiletteArea and StoveArea are examples of areas with a particular significance in our scenario. Place is composed by one or more (not necessarily contiguous) Areas. Thus, a Room is modeled as a collection of Places. Interesting Places can be identified and explicitly represented: e.g., Kitchen. In intelligent spaces it is useful to define also Apartment, and then Building.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

2.2.2. Modeling Agents and Contents ontology.content-kb is responsible for grounding message contents. It can be divided into two parts: the former is a classification of the Agents belonging to the multi-agent architecture; the latter is a taxonomy of the various data types and message contents exchanged within the system. ontology.content-kb depends on concepts defined in ontology.is-kb. Here, we enumerate interesting concepts which will prove useful in the following discussion:

• Agent is an Entity modeling software agents. According to the definition given in Section 1, each Agent is characterized by several roles: a name (which is inherited by Entity), a locator, and a number of attributes. locator is used to describe a message-trasport to communicate with the Agent. The attribute role models a data structure enumerating message types required and produced by the Agent. The Service concept models services, i.e., software agents which provide functionalities to the system. It inherits the Agent definition, adding a role that specifies the controlled Device. It is worth noting that the agent-directory-service exploits functionalities provided by ontology.content-kb to perform its tasks. • The Data concept is used to model abstract data structures. Each concept derived from Data is characterized by a role called time-stamp, which models the update time of the data packet. Specific concepts, such as IntegerData, FloatData or

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

StringData

63

can be easily defined: for example, IntegerData is described by a

value role that is an integer. Using these base concepts, we represent all the data

exchanged in our scenarios: e.g., data originating from devices like PIR (Passive Infra Red) sensors or smoke detectors can be easily modeled using IntegerData; light or temperature sensors produce instances of FloatData, etc. We do not need to build more complex Data using the basic types, as it would be, e.g., in the case of BoundingBox. Instances of Data are obviously associated with instances of Agent: they fill such roles as sensing (in case of Sensors) or imposing (in case of Actuators). • The Message concept is described using a content role that encapsulates the current instance of Data to be exchanged. According to the classification of Data, it is possible to build a taxonomy of exchanged Messages. For example, PIRMessage or SmokeSensorMessage restrict the content role to be, respectively, of type PIRData and SmokeSensorData.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

2.2.3. Interfacing Numerical and Symbolic Information The process of translating numbers to symbols is carried out through a formal and welldefined interface. The specification of such an interface is possible only because - in our approach - the numerical information is rather simple and it is not highly structured. In [34] the Integrated Connectionist/Symbolic cognitive architecture is introduced, describing cognition as a 2-layer process: at the micro level cognition can be described as a sort of parallel distributed processing based on neural evidences, whereas at the macro level it is modeled as a symbolic computational system. These symbolic computations depend on the structure of the micro level: they mirror the activation-spreading mechanisms of the underlying structure, which is assumed to be based on a connectionist paradigm. Our model for interfacing numerical and symbolic information differs from the one in [34] in that: (i) we exploit cognitive algorithms specifically designed to be executed at the symbolic level and to concurrently run with numerical ones; (ii) we do not assume a low-level connectionist architecture: on the contrary, we adopted a fully multi-agent paradigm; (iii) our aim is not to explain cognition, but to use insights from cognitive science to develop a sound interface to deal with symbol grounding. One of the main contribution of the ICS architecture is the definition of a formal interface between numerical and symbolic representations. Taking inspiration from [34], we define such an interface as follows: 1. Cognitive representations are defined by numerical or ordered values provided by appropriate agents. These representations collect distributed sources of data according to predefined symbolic structures {C}, which are grounded with respect to an ontology. 2. Each symbolic structure {C} is defined by a vector r of n individual roles {ri }, each of which may be occupied by a vector fi of one or more fillers {fij }, according to the role definition: C = ∧{ri } = {r1 ∧ r2 ∧ ... ∧ rn }. 3. An individual c  C is given  by a binding between roles and fillers according to the definition of C: c = ∧{ri fi } = {r1 ⊗ f1 ∧ r2 ⊗ f2 ∧ ... ∧ rn ⊗ fn }. 4. Symbolic structures are recursively structured: i.e., each filler fij of a certain role ri is a symbolic structure. This property will be extensively used to develop a context representation structure in Section 2.3.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

64

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

We describe the use of the interface in a working example. Recall the PIRData and SmokeSensorData concepts previously introduced. In order to make an instance within the ABox, we must specify the fillers of the corresponding value roles, which are - respectively - of type integer and float. The former describes the detection of something moving within the sensor scope; the latter the presence of smoke in the nearby. Actual Data are provided by a couple of services, i.e., pir-service or smoke-detector-service, through the messages pir-msg and smoke-detector-msg. Suppose that they are characterized - respectively - by the values 1 and 0 (i.e., something has been detected and there is no smoke in the environment). The resulting instances of PIRData and SmokeSensorData are then given by the following interface descriptions: {value = 1} and {value = 0}, which have a straightforward correspondence within the ABox. In other words, sensor data are mapped onto instances of Data, thus updating specific roles of Device instances. Similar considerations hold for other kind of features as well. In our architecture, these descriptions - which arrange information in patterns relating numerical data to symbols - are thus used to represent instances of concepts within the knowledge base. Therefore, there is a direct mapping between numerical and symbolic information and - because association between sensor data and symbols is designed a priori - instances of Data is not given a semantic meaning at this point. Despite the simple structure, the symbols associated with these data can be immediately used by an automatic reasoner.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

2.2.4. Symbolic Information Fusion The aim of symbolic information fusion is to assess base symbols which refers to the same Entity. This involves the merging of redundant information, the disambigation among conflicting data, etc. In our approach, this is currently achieved using subsumption among symbol descriptions, according to fusion operators (implemented as agents) which cooperate with ontology.is-kb and ontology.content-kb. Symbolic information fusion is achieved using subsumption, which is the main reasoning scheme provided by DLs. Given two concepts, namely C1 and C2 , we say that C1 is subsumed by C2 (and we write C1  C2 ) if C2 is more general than or equivalent to C1 . In the following paragraphs, given a concept C, we will refer to its description (i.e., the set of its constituent roles) using δC, and its instances (i.e., a description on how its roles have been filled) by ξC. Let’s explain how information fusion works by introducing how we deal with user location tracking. Consider the following scenario: • We consider a little apartment, composed by a couple of rooms (actually, a kitchen and a bedroom). Hovewer, we focus on what happens inside the kitchen only. It has been provided with four sensors: one CMPIR, two PIR detectors and one RFID reader. CMPIRs are ceiling mounted PIRs, characterized by a range view of 360 degrees, thus being suitable to detect the presence of someone in closed environments (such as rooms and corridors); PIRs are instead used for short range detection. • We use information originating from CMPIRs to discriminate among the rooms where the user could be (i.e., placing a CMPIR in each room of our set-up), then we infer the user identity through RFIDs, and finally we discriminate among specific areas through the use of PIRs. The details of user association are not really

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

65

relevant: hereafter, we simply assume to be able to infer user identity (which is an interesting research problem in its own right). Therefore, while CMPIRs give a rough location, PIRs provide boolean information about the presence of someone in a specific area (according to the sensor range and position). rfids provides da with information suited to infer user identity (actually, a simple id), which is used to retrieve the proper ξ User from the ontology. • From a multi-agent perspective, these sensors are managed by services (i.e., cmpir-1-service, pir-1-service, pir-2-service and rfid-1-service) and then by three device-agents (i.e., cmpirs, pirs and rfids). • Inside ontology.is-kb.abox and ontology.content-kb.abox, we represent four Services, i.e., cmpir1Service = ξ CMPIRService (which is controlled by cmpir1Agent = ξ CMPIRAgent), pir1Service, pir2Service = ξ PIRService (which are controlled by pir1Agent, pir2Agent = ξ PIRAgent), and – finally – rfid1Service = ξ RFIDService (controlled by rfid1Agent = ξ RFIDAgent). • Data provided by cmpir1Agent, pir1Agent, pir2Agent and rfid1Agent are represented within the ontology through instances of PIRData, RFIDData  IntegerData. Cognitive agents (i.e., da) are represented as well. The Data they provide is represented within ontology.content-kb.abox: in particular, we consider instances of UserName (associated with daAgent).

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Algorithm 1 Compute User Location Require: userName = ξ UserName; pir1Data, pir2Data = ξ PIRData Ensure: user.location = userArea = ξ UserArea 1: cmpirDevice = ; room =  2: = InitialGuess(userName) 3: room = RetrieveRoom(cmpirDevice) 4: userArea = FindArea(room, pir1Data, pir2Data)

In this case, the data fusion process is managed by a proper agent, namely cul (where “cul” is for compute user location), that checks the subsumption between the scope role of each Sensor associated with the involved Services (see Algorithm 1). Despite its apparent simplicity, the Algorithm performs three main operations: 1. InitialGuess (see Algorithm 2). Once the user whose location is to be computed through RFIDs is identified, an initial guess is instantiated in userArea using the scope of the CMPIR which detected the user (e.g., stoveArea  tableArea). 2. RetrieveRoom (see Algorithm 3). The room where the CMPIR is located is computed: this enable us to select other possible sources of information which could be useful for improving the user location (e.g., kitchen). 3. FindArea (see Algorithm 4). For each PIRSensor located in the room where the user has been detected, we check the detection value and we merge the related scope with userArea trough subsumption (e.g., if pir1Data.value = 1 and pir1Data.scope = stoveArea, then userArea becomes stoveArea, because stoveArea  tableArea  stoveArea).

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

66

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Algorithm 2 InitialGuess Procedure Require: userName = ξ UserName Ensure: cmpirDevice, userArea 1: for all user s. t. user = ξ User do 2: if userName  δ user.name then 3: for all daAgent s. t. daAgent = ξ DAAgent do 4: if δ daAgent.attribute.produced  userName then 5: for all cmpirAgent s. t. cmpirAgent = ξ CMPIRAgent do . 6: prodBycmpirAgent = δ cmpirAgent.attributes.produced . reqBydaAgent = δ daAgent.attributes.required 7: if prodBycmpirAgent  requiredByDaAgent then 8: 9: for all cmpirService s. t. cmpirService = ξ CMPIRService do 10: prodBycmpirService = δ cmpirService.attribute.produced reqBycmpirAgent = δ cmpirAgent.attribute.required 11: if prodBycmpirService  reqBycmpirAgent then 12: . 13: cmpirDevice = δ cmpirService.controlling . 14: userArea = δ cmpirDevice.scope break 15: 16: end if 17: end for 18: end if 19: end for 20: end if 21: end for 22: end if 23: end for

Algorithm 3 RetrieveRoom Procedure Require: cmpirDevice Ensure: room 1: for all room s. t. room = ξ Room do 2: for all place s. t. place = ξ Place do 3: if place  δ room.made-of then 4: for all area s. t. area = ξ Area do 5: if area  δ place.made-of then 6: if cmpirDevice  δ area.contains then 7: room = room break 8: 9: end if 10: end if 11: end for 12: end if 13: end for 14: end for

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

67

Algorithm 4 FindArea Procedure Require: room Ensure: userArea 1: for all area s. t. area = ξ Area do 2: for all pirSensor s. t. pirSensor = ξ PIRSensor ∧ pirSensor.location room do 3: for all pirAgent s. t. pirAgent = ξ PIRAgent do 4: if pirSensor  δ pirAgent.controlling then 5: if pirData = δ pirAgent.sensing = true then 6: if userAreaGuess  δ pirSensor.scope then 7: userAreaGuess = δ pirSensor.scope end if 8: 9: end if 10: end if 11: end for 12: end for 13: end for

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

2.3. Level 2: Situation Assessment Up to Level 1, symbolic information is integrated only on the basis of computational algorithms, i.e., without assigning a semantics to symbols to be used in data processing. This is done in Level 2, where relationships among Level 1 objects are established, according to the ontology and the scenario defined in Level 1. Therefore, we provide ontology with another module modeling situations and contexts which we want to monitor, i.e., we introduce ontology.context-kb. Here, symbols are given precise meanings, defined according to the Level 1 objects which are involved in the definitions. Traditionally, symbol grounding has been considered a difficult task because it required to understand complex situations from a small number of complex information sources. Here, the opposite approach is adopted, i.e., we try to understand simple situations from a great number of simple information sources, and then to build complex representations using certain simple situations, which are easier to identify. Specifically, we consider simple the sources of information which associated symbols can be immediately used by computational algorithms, and which error probability can be neglected. To achieve these results, we introduce three concepts, namely Predicate, Situation and Context. 2.3.1. Modeling Predicates and Situations A Predicate is a structure stating facts about the current state of each Entity belonging to an environment. In other words, its purpose is to establish a connection between an Entity and some Data. Each Predicate is characterized by two roles (i.e., begins-at and ends-at) specyfing the validity interval for the predicate: in this way it is possible in principle - to reason about temporal relationships among Predicates. Definitions of different predicative structures vary depending on the particular Predicate. Hovewer, it is possible to identify a number of intermediate base concepts which can be further specified according to incoming data. For instance, ToBeSomewhere relates a User to a location within the environment (e.g., a collection of Areas), TurnOn (referring to a LightDevice) is true when the corresponding light is turned on, Low (referred to the current tempera-

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

68

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

ture) is verified whenever the temperature is below a given threshold, or - if the system can infer it - ToRead models the fact that a User is reading something. These predicative structures can be even more detailed: e.g., ToBeSomewhere inherits such concepts as ToBeInTheBathroom or ToBeNearTheStove according to the classification of the Area where the user is detected through cul. This is very important in that different meanings are assigned to these Predicates, thus being possible to fire ad hoc system behaviors whenever Predicate patterns are considered anomalous. As long as new Data become available at the symbolic level, instances of Predicate are updated within the ABox. Because most of the time Predicates originate from very simple (possibly boolean or at least ordered) Data, the error probability associated with a Predicate is to be considered near zero. This is a consequence of the fact that simple information (though filtered by cognitive agents) is immediately translated into symbols. For instance, the error probability associated with TurnOn does not depend at all on computational algorithms, but only on the physical characteristics and on the error probability associated with the actual device. After all, when one turns on the lights, they are immediately turned on! Similar considerations hold for different Predicates. A Situation is a mapping between an Entity and one or more Predicates associated with that Entity. Two roles (namely, begins-at and ends-at) are used to model the temporal extension of the Situation. In principle, Situations are equivalent to Predicates in that it is always possible to define a Situation using only one Predicate: for this reason, in the following paragraphs, we use indifferently both the terms. For example, BeingSomewhere is the Situation associated with the ToBeSomewhere predicative structure only; on the contrary, to define such a concept as Studying, we use the combination of two Predicates which refer to the same Entity: ToBeAtTheDesk and ToReadABook. The temporal extension is computed by considering the and of the validity period of each Predicate. According to the hierarchical nature of DL-based representations, Situations are implicitly organized as a taxonomy, which is structurally related to the one made up of Predicates. Despite the simple structural organization, for each intermediate Situation branch, this tree-like layer proves to be effective in practice. It is characterized by a number of advantages compared to more complex models: (i) it is easily manageable and extensible: new Situation branches can be added by creating new concepts, given that the system is able to distinguish among different situations through (a combination of) sensor data; (ii) its creation can be automated: current work is focused on creating decision trees from data sets obtained in simulation whose nodes are concepts conveniently defined; (iii) using subsumption, a system exploiting the tree for managing an active classification system can be easily implemented (see Section 2.5). 2.3.2. Classifying Situations Consider the following example, which explains how the information fusion is carried out at this Level. Assume now that we are interested in tracking not only the location, but also the activities of a user = ξ User, inside an environment. Using cul, the system - according to sensor data - infers that user is located inside the kitchen = ξ Kitchen. Therefore, toBeInTheKitchen = ξ ToBeInTheKitchen  ToBeSomewhere  Predicate is instantiated, using base symbols as user and kitchen. At the same time, ToReadABook is detected. Thus, the system only knows that the user is reading a book while being in the kichen.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

69

While cul is running, in the BeingSomewhere branch of the Situation tree, beingInsideTheKitchen is detailed with the knowledge of the actual ξ Area inside which

is located. The possible choices (i.e., child concepts of BeingInsideTheKitchen) are known in advance since each Room is divided in Places, and then in Areas, etc. To achieve this goal, cul queries the status of the the PIRs located in the kitchen (see Algorithm 1). Assume again that stoveArea = ξ StoveArea is the output of cul. If it is the case, a new Predicate, namely toBeNearTheStove = ξ ToBeNearTheStove  ToBeInTheKitchen, becomes valid. At this point, two things happen. First, the original beingInsideTheKitchen is replaced by beingNearTheStove; second, the new concept checkingIngredients = ξ CheckingIngredients  DoingSomething  Situation is created, from the knowledge that toBeNearTheStove and ToReadABook are valid. Basically, updating the Situation for each Entity is a recursive task. In principle, it is possible to derive a new description to be added to a Situation whenever either a new Data is available or it has been updated. Algorithm 5 describes the abstract behavior of this process. Once new Data are available, Situations corresponding to each ξ Entity

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

user

Algorithm 5 Classify Situations Require: data = ξ Data Ensure: classification for each entity such that entity = ξ Entity 1: if data then 2: for all entity such that entity = ξ Entity do 3: for all situation such that situation = ξ Situation do 4: if entity  δ situation.refers-to then 5: for all predicate such that predicate = ξ Predicate do 6: if predicate  δ situation.made-of then 7: if data  δ predicate.states-that then 8: δ predicate.states-that = data δ situation = δ situation  δ predicate 9: if δ situation  ⊥ then 10: 11: unexpected classification: fire an alarm 12: end if 13: end if 14: end if 15: end for 16: end if 17: end for 18: end for 19: end if

are updated. This is first accomplished by updating all the instances of Predicate which depend on Data (lines 1 – 8). Next, the Situation is updated according to the classification assumed by its constituent Predicates (line 9). If the resulting Situation is inconsistent, an alarm is fired (line 10). This may involve either to call a remote operator or caregiver, or - for the system - to plan a course of action in order to recover from the failure.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

70

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

2.3.3. Managing Contexts For each group of entities, a more comprehensive state can be inferred by considering the superimposition of the most recent instances of the Situation concept relating the different instances of Entity to each intermediate base concept introduced so far. By composing Situations originating from different tree branches, it is possible to build more complex - yet partial - representations of the environment. A Context is used to model such a collection of Situations. In other words, it allows to specify interesting groups of Situations which - when considered as a whole - are particularly significant for the system to operate. In our model we consider either relational or temporal Contexts. The first group is aimed at capturing relationships among Situations occuring during the same period of time; the second group is used to detect sequences of ξ Situation, which are then related over time. In the following sections, two examples are introduced to explain relational and temporal Contexts. As a first example, assume the availability of another agent sd (where sd stands for “smoke detection”), modeled as a device-agent, communicating information about the detection of smoke by a specific Sensor, thus producing instances of SmokeData  IntegerData. If such a sensor is located above the stove, i.e., within stoveArea, it is possible to infer interesting facts about cooking activities performed by the user. Whenever ξ SmokeData is updated, a corresponding predicate, i.e., ξ SmokeIn  Predicate is updated as well. In this case, ξ SmokeIn is related to the Area where the sensor is located. A more specific SmokeOverTheStove can be defined in case of StoveArea  Area. A corresponding ThereIsSmokeOverTheStove  Situation can be defined. If we introduce Cooking = δ BeingNearTheStove  δ ThereIsSmokeOverTheStove  Context, once the corresponding Situations are met, we can infer that Cooking is the current Context. From a computational perspective, this mechanism is equivalent to the one presented in Algorithm 5, substituting Predicate and Situation with, respectively, Situation and Context. As a second example, consider the problem of monitoring user health status by checking its visits to the bathroom. This can be modeled using OftenInTheBathroom  Context, characterized by specific roles to monitor the validity of subsequent instances of BeingInsideTheBathroom. The concept OftenInTheBathroom monitors descriptions belonging to either instances of BeingInsideTheBathroom whose temporal extension (i.e., the difference between the roles begins-at and ends-at) exceeds a threshold, or to sequences of δ Situation where the frequency of BeingInsideTheBathroom amounts to a given percentage. The introduction of Context (and, in a sense, of Situation) poses the following, fundamental, problem: how do we relate the representation and the meaning of Contexts to what is really happening within the intelligent space? In other words, does a method exist to quantitatively validate the inferences performed by the system about the currently occurring Situation? In a sense, by anchoring base symbols corresponding to simple information, we delegate the responsibility for the Context acquisition entirely to the ontology, through building predicative structures which are more and more complex as far as base symbols are related to each other. Unfortunately, no formal method exists to verify the correspondence between actual concepts and real events. This is very close to what happens in hand-writing recognition: first, written symbols on paper have to be unambiguously labeled; next, they must be related to each other in order to infer words, sentences, and then the meaning of an entire phrase. According to our approach, simple

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

71

base facts are to be recognized, and then put together to infer situations and context, in order to finally understand what is really happening.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

2.4. Level 3: Impact Assessment Level 3 of the JDL model is responsible for predicting the effects of the system behavior over the entities modeled by the ontology. In this work, we focused mainly on the status of the devices which are distributed throughout the intelligent space. This is very important as the ontology should maintain a coherent model of the system all the time, in order to immediately infer faults and to present reliable data to either decision processes (e.g., planning or rule-based systems) or human supervisors (e.g., remote caregivers). Impact Assessment, in the field of AmI, assumes a particular significance in the sense that the space is designed to be intelligent: in other words, designers and engineers are usually allowed to place sensors and actuators wherever they think it fits their goals and objectives (which, hopefully, should meet user requirements). Classical Artificial Intelligence failed within scenarios where this “trick” was not allowed. Knowledge representation, planning and task execution were confined to work well only within “toy worlds”, i.e., artificial realities - or extreme simplifications of an environment - where noise and uncertainty are banned, and - consequently - the outcome of each action and representation is deterministic. If designers select sensors providing simple information and actuators allowing only simple actions - as in our approach - AmI applications are very close to meet the “toy world requirement”: the error probability associated with measurements or with controls issued to actuators is minimum as far as the domain of measurements and controls is restricted. Thus, it could be reasonable to degrade the possibilities offered by cameras in order to perform, e.g., motion detection, rather than applying complex algorithms for user identification (which are characterized by a higher error probability), as discussed in Section 2.2. The same result is achieved - in our approach - using several simple sensors as sources of information, and merging their data using well-defined structures assigned with semantic meaning. For the same reasons, it is preferable - as in AmI applications - to limit the number of actuators necessary to perform an atomic action within the environment. In fact, in the case of boolean data and controls, error probabilities are strictly related to those of the actual devices. This allows significant simplifications in the control architecture and - ultimately - within the symbolic representation. In our approach, each Device representation is augmented with a description of its behavior modeled as a simple finite state machine (FSM). This is possible due to the fact that error probabilities are not taken explicitly into account within the model: otherwise, Markov Chains would be the obvious choice. The purpose of this FSM representation is twofold: (i) to model each device as a fluent, i.e., an entity whose state changes over time (and which track is kept), thus being able to reason at the temporal level: this information is immediately available due to the Predicates associated with the Device; (ii) to determine expected state values given the current state and inputs, thus detecting device faults and reacting accordingly. The choice to use a FSM to model Device behaviors can be extended to involve a planning system, designed to work with ontology, able to predict outcomes of action sequences (and their estimated effects on devices) which are necessary for the fulfillment of a given goal. Our approach is very similar to the one described in [27], thus it does not deserve further investigation.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

72

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

In the case of Sensors, FSMs are used to infer faults on the basis of lack of updates: if a sensor Data is not updated for a long time (the threshold is specified within the model itself), an alarm is immediately activated. Currently, we perform these controls at the Predicate level. Modeling of Actuators deserves special attention. Actuators are usually provided with a corresponding Sensor used to check the status of the Device as an action is performed on the Device itself (i.e., a DC motor with a corresponding encoder). This approach is usually suited to improve the fault diagnosis capabilities of the system. In real scenarios, it is often unfeasible to place a Sensor for each Actuator. Nonetheless, whenever the “toy world assumption” is met, this can be reasonably avoided, as actions performed within the intelligent space are the consequence of simple, atomic, operations performed by an Actuator. Furthermore, within our representation, we associate a Sensor with an Actuator even if a corresponding physical sensor does not exist. In this case, virtual measurements are taken directly from references and updated according to the timings in the FSM: this is possible, again, because error probabilities are almost null.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

2.5. Level 4: Process Refinement Process Refinement deals with topics ranging from learning to inter level information flow management. As previously discussed, learning will be the focus of future work, so it is not longer discussed here. Inter level information flow management deals with techniques aimed at guiding data acquisition (according to specific goals) and - ultimately - to orchestrate the coordinated behaviour of each service and agent operating within the system. In our architecture, this can be accomplished by introducing new cognitive agents which, on the basis of a number of patterns (explicitly modeled by designers or users, or stated on-the-fly by decision processes), are able to schedule the activities of other agents. Using this technique, we propose a system for the active classification of Situations and Contexts, which cooperates with the representations provided by ontology. It is implemented using a new agent, called asc (where asc stands for “active situation classification”), which exploits a mapping between Data and groups of agents. This information is immediately available as ontology is used to implement an agent-directory-service. The basic idea of the algorithm is that the branching levels of each Situation tree are distinguished according to the information provided by selected agents. Thus, whenever a situation = ξ Situation is subsumed by Situationi  Situation, the system searches - through

child concepts of Situationi . These are subsumption - for possible stored in Σ = Situationji , j = 1, ..., |Σ| . If Σ is null, the classification ends here, and situation  Situationi . On the contrary, if Σ is not empty, the kind of Data necessary to disambiguate among the different Situationji is identified. Next, by quering the agent-directory-service, it is possible to retrieve the set of agents providing the requested Data. Therefore, the selected agent can provide the necessary information ondemand: once the new Data has been introduced within the ontology (and, consequently, the classification of situation is changed) the algorithm can be recursively applied. The algorithm ends when the current classification can not be further specified.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

73

Figure 1. (Left) Light sensor in the Know-House@DIST set-up; (Right) A door passage controlled by a PIR sensor.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

3. Experimental Results and Discussion The experiments we designed are aimed at validating different subsystems of our architecture, and the corresponding interfaces. In particular, we performed experiments in a real set-up which has been furnished with several devices to monitor the status of the environment, as well as simulations aimed at validating the proposed ontology architecture. Usually, simulated experimental results are characterized by several drawbacks: among them, the difficulty to provide the simulated systems with data which are comparable to real ones. In our case inputs and outputs managed by cognitive subsystems are either boolean (as in the case of the information provided by PIRs and smoke detectors, or commands issued to doors or windows) or ordered values (data originating from temperature or light sensors): for this reason - as it has also been verified during experiments - we are confident that simulations can provide useful insights about the behavior of real scenarios. In the following paragraphs, we first describe experiments in a real set-up; next, we introduce results obtained in simulation, discussing the reliability of the symbolic representation; finally, we explore the capabilities offered by the system.

3.1. Experiments in a Real Scenario In our current implementation, services are implemented using different technologies. Some of them exploit the Echelon LonWorks fieldbus for distributed control and programming (as it is the case for home automation, i.e., actuators for doors, windows or lights, or light and temperature sensors, etc.), whereas others use standard Ethernet interfaces (e.g., camera images for caregivers are accessible through common HTTP streams). This is even more true when dealing with agents. These are implemented using ETHNOS, a framework developed for distributed Robotics and AmI applications [31], used here as a middleware for integrating heterogeneous technologies. It provides communication capabilities either among agents or between services and device-agents - in a sense, standardizing the information exchange. Thus, at the low level, it allows communication between devices and cognitive algorithms by providing interfaces with the fieldbus or with the Ethernet network. At the high level, it allows the implementation of

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

74

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

cognitive computational processes (such as ontology, cul or asc) integrating different software systems. ontology is developed embedding in ETHNOS NeoClassic, which is a Description Logic guaranteing sound and complete subsumption inferences [22]. Required planning capabilities are achieved using a PDDL compatible planner: currently, an ETHNOS agent acts as an interface to SGPlan5 [17]. Within our set-up (that we called Know-House@DIST - see Fig. 1 and Fig. 2 on the left) a single user scenario has been tested, implementing the main objectives of the proposed architecture. In particular, we first validated our intuition about the low error probability associated with the anchoring of simple data originating from devices, and then we tested how this information drive the behavior of cognitive agents. Days of continuous work and data gathering can be summarized by saying that the error probability for each anchoring process is almost null, the only error sources being device faults. Consider, for example, a possible application of light sensors to produce corresponding symbolic information within the ontology: in particular, we want to classify lightSituation = ξ LightSituation, according to Dark, Soft, Bright  LightSituation. This information is first anchored to an instance of LightData  IntegerData, and then within a Predicate relating the specific Sensor to ξ LightData. Despite the current arbitrary definition of thresholds to discriminate among Dark, Soft and Bright, we noticed a temporary wrong classification only because of voltage instability of the board controlling the light sensor, which caused a wrong A/D conversion of the actual light level and thus wrong data fed to ontology. Conversely, experiments performed with cameras and algorithms aimed at performing user tracking through bounding box features (see Fig. 3 on the right) are subject to errors due to light conditions and occlusions: for these reasons, we decided not to use them in our approach. From these results, we performed several experiments with the purpose of investigating the reliability of the system in tracking user behavior. In particular, simple Situations and Contexts have been evaluated, according to the specific agents which have been implemented. Currently, we use a couple of CMPIR (the first inside the kitchen, the second in the bedroom), several PIRs (located near the bed, the stove, the doors - see Fig. 1 on the right -, etc.), two light sensors (one for each room), a temperature sensor (see Fig. 1 on the left); furthermore, we are able to control the lights of each room and to open/close the windows. These experiments are related to user location tracking through CMPIRs and PIRs, to position-based activity recognition involving furniture and visited areas, and to dangerous events detection on the basis of simple Contexts. From the experiments - once again - we noticed a strong (positive) dependance on the low level symbol grounding. In general, by correctly defining proper concepts and relationships it is usually possible to model the desired Context. One important aspect of symbolic representation is to present reliable and up-to-date information to either human caregivers or (semi)automated decision processes. Therefore, specific agents are responsible for intelligent information presentation. In our setup we developed and tested web interfaces meant to be used by remote relatives or caregivers of the inhabitant of intelligent spaces (see Fig. 2 on the right). Beside providing basic information regarding camera images and the status of all the sensors/actuators, the interface queries the system using a common uri-based mechanism, exploiting the hierarchical structure of the ontology. For example, the status of the kitchen = ξ Kitchen  Room can be queried by a browser using http:///knowhouse/room/kitchen.xml, and the system returns an xml file describing current values for sensors/actuators lo-

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

75

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 2. (Left) The Know-House@DIST set-up; (Right) A web-interface for caregivers.

cated inside the kitchen itself. The tags used to build device descriptions are automatically generated from the roles of the concept descriptions. The same method is exploited to provide information about the actual user situation. By querying the system with http:///knowhouse/user/user.xml the caregiver receives an xml description where the current instances of Contexts involving the selected user are represented using nested tag descriptions (each level corresponding to a branch node of the Situation tree). At the same time, the caregiver could be provided with information about Situations which are likely to occur. This could be achieved simply by reasoning on the Situation tree: likely Situations correspond to descriptions which differs only for a limited number of concept fillers. The system is currently being tested at Istituto Figlie di N.S. della Misericordia, Savona (Italy), an assisted-living facility for elderly and people with special needs. 3.2. Experiments in Simulation In order to validate the most advanced capabilities of the system, the cognitive agents of our framework have been tested in a simulation environment designed as part of the architecture itself [7]. The simulator consists of several agents implementing simulated devices, furniture and user behaviors, and then visual interfaces (see Fig.3 on the left), etc. Simulated experimental results are carried out by adding specific agents simulating user activities, and providing sensors with data sets recorded through real sensors (data sets have been especially created to simulate fire, gas leaks, etc.). Each base user pattern (see Algorithm 6) is a sequence of parameterized actions, e.g., movements, interactions with the environment, etc. Examples of base patterns are apc (where apc stands for “answer phone calls”), composed by the sequence of actions {answer, talk, hang-up} or tl (where “tl” stands for take lunch), constituted by base actions such as {goto(kitchen), goto(nearStove), cook, wait(1), goto(nearTable), eat}. Actions are taken from a pool of possible actions according to the simulated environment: e.g., each furniture can be operated on during simulations. It is worth noting

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

76

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

Algorithm 6 Base Pattern Require: A = {a1 , ..., an }; E = {e1 , ..., em }; Epdf Ensure: A specific user behavior 1: for all i such that i = 1, ..., n do 2: perform action ai 3: choose j according to Epdf 4: fire the event ej 5: end for

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 3. (Left) The simulation environment; (Right) Bounding Boxes extraction from camera images.

that not all the actions are treated as discrete events: e.g., user movements (i.e., goto(x)) correspond to trajectories in the cartesian space. Moreover, during each iteration in Algorithm 6, an event ej is chosen to possibly introduce a perturbation in the current pattern (possibly a null perturbation). Events can range from waiting a certain amount of time to interacting with appliances, from receiving phone calls to completely changing the current pattern with a new one. Events are selected according to a non-uniform outcome probability distribution Epdf . In all the experiments, beside the device-agents involved in data acquisition, cul and asc are able to track the simulated user with cameras, PIRs and other sensors, maintaining also multiple hypotheses through subsumption. Another agent, tlb (where tlb stands for “too long in bed”), is able to fire alarms whenever the user remains within the bedArea = ξ BedArea for more than a given period, i.e., the corresponding BeingInBed  Situation lasts for too long. Furthermore, OftenInTheBathroom  Context is detected multiple times. Specific experiments are aimed at stressing the active classification agent asc. For example, during the execution of tl, after the user has moved to stoveArea, the event (and, consequently, the corresponding agent) apc is selected. This implies that the current pattern is replaced by the one implemented inside apc. After some time, ontology is not updated with the expected information: in other words, the user is supposed to be cooking, but at the same time he is supposed to talking at the phone. This is detected because the corresponding ξ BeingNearTheStove is not updated. Thus, a specific query to sd is made. If the ξ SmokeData is such that smoke is detected, the system reminds the user about his previous cooking action. On the contrary, if no smoke is detected, it is inferred that the user was simply standing inside stoveArea.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

77

3.3. Expressiveness of the Representation

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

The choices associated with the definition of predicates, situations and contexts are inspired by the design philolophy of the components reuse, which is well known in Software Engineering. Composing several base symbols (i.e., predicates) originating from simple information extends the representation capabilities associated with the system: a number of predicates contributes to build one or more situations and - accordingly - a number of situations is used to build contexts. Obviously, the same predicate or situation can contribute to the definition of different situations and contexts. In the following paragraphs, we make a number of examples easily described using our approach: • During the night, the user is sleeping. This can be inferred because the user location is bedArea. This situation could be enforced either using a BedSensor or by analysing biometric signals, if available. Lights are turned off, which is easily known. At some point during the night, the light sensor reveals that in the kitchen some persistent source of light is present: e.g., the symbolic association to sensor data is classified as Bright. The validity interval of the corresponding predicate is thus used to measure the persistence of this situation. Because lights and other applicances (such as, e.g., the TV) are turned off, and the user is still in bed, the overall situation is considered anomalous. Then, a number of queries are made to specific sensors: e.g., to the smoke and temperature sensors in the kitchen. The sensors are identified because they are in the same room where an unexpected light has been detected. In case of precence of smoke and a high temperature, a dangerous situation is immediately detected (i.e., the presence of fire inside the kitchen): this is notified to the user and to caregivers. • Usually, immediately after lunch, the user is used to spent half an hour reading the newspaper while sitting at the desk and then to sleep for a couple of hours. This is detected using a temporal context, relating three situations: i.e., Eating, ReadingTheNewsPaper and Sleeping. One day, a slightly different pattern is detected: after a quick lunch, the user moves immediately towards the bed, without reading the newspaper. Furthermore, its movements are very slow: this fact is detected by considering the duration of time intervals corresponding to BeingSomewhere situations. As a consequence, it can be inferred that either the user is very tired, or he is not feeling well. A caregiver could be silently notified about this potential dangerous context. • The user is in the kitchen, preparing food for lunch. The system knows that because either CheckingIngredients or Cooking are valid. For example, in the second case, the user is inside the stoveArea while, at the same time, sd is reporting the precence of smoke. Suddenly, the user receives an SMS message, which is automatically read by the system. Immediately, the user goes to the bedroom, taking something from the closet. His movements are easily tracked as being instances of BeingSomewhere. Then he goes out, forgetting the food over the stove. Because of this, the system infer that something potentially dangerous happened somewhere: it turns off the fire of the stove, and then notifies this anomalous situation to an user relative. All the examples describe situations and contexts which - without the use of a well-structured symbolic representation - would be otherwise very difficult to model. Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

78

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

Nonetheless, in our approach, these scenarios can be easily described once the base predicates are available.

4. Conclusion This work presented a symbolic knowledge representation and information fusion system suitable for developing highly integrated AmI applications. Despite its simple design, the architecure can manage heterogeneous information at different levels (i.e., numerical as well as symbolic), “closing the loop” between sensors and actuators. The basic underlying idea is to avoid the use of a small number complex sources of information to infer complex situations and events occurring within an environment, but to adopt the opposite approach, i.e., to use a great number of simple information sources to infer simple situations and events, to be later used by symbolic structures to infer complex situations and contexts.The proposed architecture has been thoroughly described adopting the JDL information fusion model. Experimental results are presented, detailing how the architecture has been applied to a real scenario and discussing the expressive capabilities of the representation language defined within the proposed ontology.

References [1]

[2]

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

[3]

[4] [5] [6]

[7]

[8]

[9] [10]

[11]

A.H.G. Al Dhaher, D. MacKesy. Multi-sensor Data Fusion Architectures. In Proceedings of the 3rd IEEE International Workshop on Haptic, Audio and Visual Environments and their Applications (HAVE’04), Ottawa, Canada, October 2004. J. C. Augusto and C. D. Nugent. A New Architecture for Smart Homes Based on ADB and Temporal Reasoning. In Proceedings of the 3rd International Conference on Smart Homes and Health Telematics (ICOST’05), Canada, July 2005. J.C. Augusto, P. McCullagh, V. McClelland and J.A. Walkden. Enhanced Healthcare Provision Through Assisted Decision-Making in a Smart Home Environment. In Proceedings of the Second Workshop on Artificial Intelligence Techniques for Ambient Intelligence (AITAmI’07), Hyderabad, India, January 2007. T. S. Barger and D. E. Brown and M. Alwan. Health-Status Monitoring Through Analysis of Behavioral Patterns. In IEEE Transactions on Systems, Man, and Cybernetics – Part A, vol. 35, no. 1, January 2005. C. Batini, M. Lenzerini. A Comparative Analysis of Methodologies for Database Schema Integration. In ACM Computing Surveys, vol. 18(4), pp. 323 – 364, 1986. I. Bloch, A. Hunter, A. Appriou, A. Ayoun, S. Benferhat, P. Besnard, L. Cholvy, R. Cooke, F. Cuppens, D. Dubois, H. Fargier, M. Grabisch, R. Kruse, J. Lang, S. Moral, H. Prade, A. Saffiotti, P. Smets, C. Sossai. Fusion: General Concepts and Characteristics. In International Journal of Intelligent Systems, vol. 16(10), pp. 1107 – 1134, 2001. F. Capezio, D. Femia, F. Mastrogiovanni, A. Sgorbissa, R. Zaccaria. AE-Sim 2.0: a Distributed Simulator for Intelligent Spaces and Robots. In Proceedings of the Fourth International Conference on Smart Homes and Health Telematics (ICOST’06), Northern Ireland, UK, June 2006. Shaoua Chen and Hong Bao and Xianyun Zeng and Yimin Yang. A Fire Detecting Method based on Multi-sensor Data Fusion. In Proceedings of the 2003 International Conference on System, Man and Cybernetics (SMC), Washington, October 2003. D.J. Cook, S.K. Das. How Smart are our Environments? An Updated Look at the State of the Art. In Pervasive and Mobile Computing, vol. 3, no. 2, pp. 53 – 73, March 2007. R. Dapoigny and P. Barlatier. Towards a Context Theory for Context-Aware Systems. In Proceedings of the Second Workshop on Artificial Intelligence Techniques for Ambient Intelligence (AITAmI’07), Hyderabad, India, January 2007. A. P. Dempster. A Generalization of Bayesian Inference. In Journal of the Royal Statistical Society, Series B, vol. 30, pp. 205 – 247, 1968.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

[12]

[13] [14]

[15] [16]

[17]

[18] [19] [20]

[21]

[22] [23]

[24]

[25]

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

[26] [27]

[28]

[29]

[30]

[31]

[32]

[33]

79

J. Donoghue and J. Herbert. An Intelligent Data Management Reasoning Model within a Pervasive Medical Environment. In Proceedings of the First Workshop on Artificial Intelligence Techniques for Ambient Intelligence (AITAmI’06), Riva del Garda, Italy, August 2006. IEEE FIPA Standards Committee. FIPA Abstract Architecture Specifications. www.fipa.org H. Hagras, V. Callaghan, M. Colley, C. Graham. A Hierarchical Fuzzy-genetic Multi-agent Architecture for Intelligent Buildings Online Learning, Adaptation and Control. In Journal of Information Sciences – Informatics and Computer Science, vol. 50, no. 1–2, March 2003. S. Harnad. The Symbol Grounding Problem. In Physica D, vol. 42, pp. 335 – 346, 1990. T.L. Hayes, M. Pavel, N. Larimer, I.A. Tsay, J. Nutt, A. Gustavo Adami. Distributed Healthcare: Simultaneous Assessment of Multiple Individuals. In IEEE Transactions on Pervasive Computing, vol. 6, no.1, pp. 36 – 43, January 2007. Chih-Wei Hsu, B.W. Wah, R. Huang, Y. Chen. Constraint Partitioning for Solving Planning Problems with Trajectory Constraints and Goal Preferences. In Proceedings of the Twentieth International Joint Conference on Artificial Intelligence (IJCAI-07), Hyderabad, India, January 2007. A. Hunter, W. Liu. Fusion Rules for merging uncertain information. In International Journal of Information Fusion, vol. 7(1), pp. 97 – 134, March 2006. A. Hunter, Rupert Summerton. A Knowledge-based Approach to Merging Information. In KnowledgeBased Systems, vol. 19, no. 8, pp. 647 – 674, December 2006. Fan Tian Kong and You-Ping Chen and Jing-Ming Xie and Zu-De Zhou. Distributed Temperature Control System Based on Multi-sensor Data Fusion. In Proceedings of the 2005 International Conference on Machine Learning and Cybernetics, China, August 2005. E. Leon, G. Clarke, V. Callaghan, F. Sepulveda. A User-Independent Real-Time Emotion Recognition System for Software Agents in Domestic Environments. In Engineering Applications of Artificial Intelligence, vol. 20, no. 3, pp. 337 – 345, April 2007. H.J. Levesque, R.J. Branchman. Expressiveness and Tractability in Knowledge Representation and Reasoning. In Computational Intelligence, vol. 3, no. 2, pages 78 – 93, 1987. L. Liao, D. Fox and H. Kautz. Location-based Activity Recognition using Relational Markov Networks. In Proceedings of the 19th International Joint Conference on Artificial Intelligence (IJCAI-05), Edimburg, Scotland, 2005. J. Llinas, C. Bowman, G. Rogova, A. Steinberg, E.Waltz, F. White. Revisions and Extensions to the JDL Data Fusion Model II". In P. Svensson, J. Schubert (Eds.), Proceedings of the 7th International Conference on Information Fusion, Stockholm, Sweden, June 2004. R. Lundh, L. Karlsson and A. Saffiotti. Plan-Based Configuration of an Ecology of Robots. In Proceedings of the 2007 IEEE International Conference on Robotics and Automation (ICRA’07), Rome, Italy, April 2007. A. Makarenko, H. Durrant-Whyte. Decentralized Bayesian Algorithms for Active Sensor Networks. In Information Fusion, vol. 7, pages 418 – 433, 2005. F. Mastrogiovanni and A. Sgorbissa and R. Zaccaria. A System for Hierarchical Planning in Service Mobile Robotics. In Proceedings of the Eighth International Conference of Intelligent Autonomous Systems (IAS-8), Amsterdam, The Netherlands, March 9 - 12, 2004. M. Miozzo, A. Sgorbissa, R. Zaccaria. The Artificial Ecosystem: A Multiagent Architecture. In Intelligent Data Engineering and Automated Learning - Lecture Notes in Computer Science, vol. 2690, Springer Verlag, Berlin, August 2003. H.T. Nguyen, J. Qiang, A.W.M. Smeulders. Spatio-Temporal Context for Robust Multitarget Tracking. In IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 1, pp. 52 – 64, January 2007. W. Pentney, H. Kautz, M. Philipose, A. M. Popescu, S. Wang Sensor-Based Understanding of Daily Life via Large-Scale Use of Common Sense. In Proceedings of he 26th Annual Conference of the American Association of Artificial Intelligente (AAAI), Boston, Massachusetts, July 16 - 20, 2006. M. Piaggio, A. Sgorbissa, R. Zaccaria. Pre-emptive versus Non Pre-emptive Real Time Scheduling in Intelligent Mobile Robotics. In Journal of Experimental and Theoretical Artificial Intelligence, vol. 12, no. 2, 2000. N. Richard and S. Yamada. Context-Awareness and User Feedback for an Adaptive Reminding System. In Proceedings of the Second Workshop on Artificial Intelligence Techniques for Ambient Intelligence (AITAmI’07), Hyderabad, India, January 2007. I. Satoh. A Location Model for Smart Environments. In Pervasive and Mobile Computing, vol. 3, no. 2,

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

80

[34]

[35] [36]

pp. 53 – 73, March 2007. P. Smolensky. Computational Levels and Integrated Connectionist/Symbolic Explanation. In P. Smolensky and G. Legendre (Eds.), The Harmonic Mind: From Neural Computation to Optimality-Theoretic Grammars, Vol. 2, Cambridge, MA, MIT Press, 2006. A. Steinberg, C. Bowman. Revisions to the JDL Data Fusion Model. In D. Hall, J. Llinas (Eds.), Handbook of Multisensor Data Fusion, CRC Press LLC, Boca Raton, FL, 2001. F. E. White. Managing Data Fusion Systems in Joint and Coalition Warfare. In Proceedings of the 1998 International Conference on Data Fusion (EuroFusion’98), Great Malvern, UK, October 1998. M. Wozniak. Proposition of Common Classifier Construction for Pattern Recognition With Context Task. In Knowledge-Based Systems, vol. 19, no. 8, pp. 617 – 624, December 2006.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

[37]

F. Mastrogiovanni et al. / An Active Classification System for Context Representation

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Advances in Ambient Intelligence J.C. Augusto and D. Shapiro (Eds.) IOS Press, 2007 © 2007 The authors and IOS Press. All rights reserved.

81

Ambiance: A Mobile Agent Platform for End-User Programmable Ambient Systems

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Reza RAZAVI a,1, Kirill MECHITOV b, Gul AGHA b, Jean-François PERROT c a University of Luxembourg, FSTC, LUXEMBOURG b University of Illinois at Urbana-Champaign, IL, USA c Université Pierre et Marie Curie, LIP6, Paris, FRANCE

Abstract. We are interested in situations where multiple uncoordinated nonprofessional programmer end-users want to exploit the Ambient Intelligence (AmI) infrastructure on their own, without calling in embedded systems programmers, in order to support their daily activities. Our goal is allowing them to achieve this objective through on-the-fly creation and execution of high-level programs that we call uQueries (for user-defined or ubiquitous queries). The key challenge then is to support the cost-effective and stepwise development of uQuery engines---systems for end-user programming and execution of uQueries. We present a meta-level architecture that addresses this issue by leveraging Artificial Intelligence methods that make possible the separation of uQuery representation and reasoning concerns from those of their effective execution through model-tocode transformation. We show that (1) interconnections between ambient devices may be dynamically specified as control flows between high-level descriptions of their primitive functionality, (2) specifications may be elaborated by concurrent, uncoordinated end-users through a Web interface, and (3) they may be automatically distributed and concurrently executed on ambient devices as a system of mobile agents. We have created a prototype of this architecture, the Ambiance Platform, which has allowed experimental validation of the approach using an application scenario proposed in the state-of-the-art of relevant research areas. This experience led us to identify important issues to be explored, including dynamic and seamless integration of sensor and actuator nodes into the system. Furthermore, opportunities exist for significant performance and resource use optimization, for instance by integrating learning mechanisms into uQuery specification, transformation and execution. Keywords. Ambient Intelligence, Artificial Intelligence, Sensor Networks, Macroprogramming, Adaptive Object-Models, Mobile Agents, Actor Systems

Introduction Ambient Intelligence (AmI) envisions the ‘invisible’ incorporation into our surrounding environment and everyday objects of billions of loosely-coupled sensing, actuating, 1

Corresponding Author: Reza Razavi, FSTC, University of Luxembourg, 6, rue Richard Coudenhove-Kalergi, L-1359 Luxembourg, LUXEMBOURG; E-mail: [email protected]. The work communicated in this chapter has been mostly conducted while the corresponding author was acting as principal investigator on the Åmbiance project funded by the University of Luxembourg (2005-2006).

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

82

R. Razavi et al. / Ambiance

computing and communicating components as part of an AmI infrastructure [1, 2]. The aim is to discover new ways of supporting and improving people's lives and enabling new work practices. See Berger et al. [3] for a number of examples. AmI infrastructure components such as Micro-Electro-Mechanical Systems (MEMS), smart materials, ad hoc smart dust networks, and bio-inspired software are being intensively researched and developed. The Cyber Assist project by Nakashima [4] describes a fully developed example. Another key enabling technology is networks of potentially very large numbers of wirelessly connected small, autonomous, distributed, and low-powered computers endowed with limited processing, storage, sensing, actuation, and communication capabilities. Such a system is called a Wireless Sensor Network (WSN), and each sensor node a mote [5, 6]. The problem is now to write programs for the AmI infrastructure. This is no easy task, given the complexity and dynamicity of the structures and the diversity of potential users and their needs. Our work is concerned with situations where multiple uncoordinated end-users need to exploit the AmI infrastructure on their own, in order to solve problems in everyday life and to support their daily activities in different domains. They should be able to achieve this goal by on-the-fly writing and executing high-level programs that we call uQueries (for user-defined or ubiquitous queries/macroprograms), without calling in specialists of embedded systems programming. This is motivated by the diversity of functionalities that end-users expect from the AmI infrastructure, (see Böhlen [7] for a general argument and Richard and Yamada [8] for a typical example), further amplified by the unpredictability of the phenomena being monitored and the potential changes in the ambient computing infrastructure. From the critical standpoint taken by Huuskonen [9] we clearly adopt the “Person-centric” approach. In our view, this environment calls for a number of Artificial Intelligence techniques to be applied, notably knowledge representation and machine learning. We shall also see that multi-agent systems likewise play an important role as observed by Nakashima [4]. We consider an AmI infrastructure that comprises both WSNs and more traditional computing artifacts such as PCs, gateway nodes, and handheld mobile devices (although the issues specific to mobile devices are not dealt with in this chapter). Let us call each hardware component an ambient node. This infrastructure is open in that both ambient nodes and uQueries may enter and leave the computing environment dynamically. Based on resource availability and optimization criteria, available ambient nodes coordinate and determine their mutual application execution responsibilities at runtime. As a motivating example, consider a scenario from Microsoft Research [10], where the ambient infrastructure, installed in a parking garage, comprises break beam sensors and security camera nodes. Two ordinary end-users, namely Liz and Pablo, who work independently, desire to use the ambient system for their own purposes. Liz is a site manager of the garage building and is interested in collecting vehicle arrival time data. Pablo is a security officer in the building who wants to issue tickets to speeding drivers. We assume that the deployed system does not include ready-to-use specific functionalities required by typical end-users such as Liz and Pablo. It should therefore be programmed, deployed and executed by the users themselves. In the following sections we describe an architecture that satisfies the above requirements and provides a ubiquitous and omnipresent interactive Web-enabled environment for programming and executing uQueries, illustrated through application to the above example scenario.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

R. Razavi et al. / Ambiance

83

1. Problem Statement

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

1.1. Inappropriateness and Complexity of Current Programming Techniques Current system development and deployment techniques do not transfer well to programming ambient systems. New computation models and software development methodologies are required. Satoh observes, for instance, that “Ambient intelligence technologies are expected to combine concepts of intelligent systems, perceptual technologies, and ubiquitous computing.” [11]. In particular, effectively programming WSNs is difficult due to their typical resource limitations. Moreover, sensor nodes are prone to failure (for example, if they run out of energy), and communication between them is unreliable. Programming such networks requires addressing those limitations. Unfortunately, current methods for WSN programming lead developers to mix high-level concerns such as quality of service requirements, for instance timeliness, reliability, application logic, adaptivity, with low-level concerns of resource management, synchronization, communication, routing, data filtering and aggregation. This makes developing software for WSNs a costly and error-prone endeavor, even for expert programmers (see a simple illustrative case in the next subsection). Macroprogramming has been proposed as a technique for facilitating programming WSNs. Macroprogramming enables the definition of a given distributed computation as a single global specification that abstracts away low-level details of the distributed implementation. The programming environment first automatically compiles this high-level specification down to the relatively complex low-level operations that are implemented by each sensor node, and then deploys and executes these operations [12]. However, macroprogramming is of interest for specialized embedded systems programmers, not for end-users. On the contrary, as explained above, we are interested in situations where both the users’ requirements and the WSN environment may be dynamic. Thus, the key challenge is to develop uQuery engines---systems that support enduser programming and execution of uQueries. In particular, this requires enabling specifications by multiple concurrent and uncoordinated end-users of queries, which may convey a complex logic (comprising control constructs and hierarchical structures). It also requires deploying and executing such specifications in a concurrent and resource-aware manner. This chapter presents a technique to support the cost-effective and stepwise development of uQuery engines. As we shall see, our approach makes a central use of a two-level multi-agent system, together with a knowledge base about the target application domain. Adaptive learning behavior for the agents remains to be implemented in our system.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

84

R. Razavi et al. / Ambiance

1.2. Illustration Consider a simple WSN application scenario where we want to detect, on demand, an object passing through a break beam sensor. The algorithm is as follows: 1. Wait for a request from the user. 2.

3.

Perform the requested action: 2.1 Execute detectBeamEvent() primitive. 2.2 Keep checking the sensor until a change in status is detected. Send the result of the detection event back to the user.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

The user (programmer) is responsible for choosing the right OS and network components, and assembling them along with the specific application logic into an executable program. The code excerpt below presents an implementation of a simple WSN program for the above algorithm that such a programmer could write in the nesC language for the TinyOS sensor platform. Comment blocks denoted by /* … */ indicate additional code segments not related to application functionality. // Detect break beam event application (code excerpt) configuration Example {} implementation { // list of application components components Main, ExampleM, LedsC, GenericComm, TimerC, Photo, CC1000ControlM; // statically link all components Main.StdControl -> GenericComm; Main.StdControl -> TimerC; Main.StdControl -> Photo; Main.StdControl -> ExampleM; ExampleM.SendMsg -> GenericComm.SendMsg[10]; ExampleM.ReceiveMsg -> GenericComm.ReceiveMsg[10]; ExampleM.CC1000Control -> CC1000ControlM; ExampleM.Timer -> TimerC.Timer[unique("Timer")]; ExampleM.Leds -> LedsC; ExampleM.PADC-> Photo; } module ExampleM { /* … */ } implementation { TOS_Msg msgbuf; uint8_t msglen, sendPending; volatile uint8_t ioPending; uint16_t ioData; /* … */ // primitive function #20: detect beam event (using photo sensor) uint16_t detectBeamEvent(); // I/O: convert split phase non-blocking I/O to blocking I/O uint16_t IO(uint16_t a, uint16_t b) __attribute__((C,spontaneous)) { while (ioPending) yield(); if (a == 20) { call PADC.getData(); ioPending=1; } while (ioPending) yield(); return ioData; } async event result_t PADC.dataReady(uint16_t data) { ioPending=0; ioData=data; return SUCCESS; }

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

R. Razavi et al. / Ambiance

85

// Communication: receive requests for execution and send results void sendPacket(uint8_t *buf, uint8_t n) __attribute__((C,spontaneous)) { memcpy(msgbuf.data, buf, n); msglen = n; if (call SendMsg.send(TOS_BCAST_ADDR, msglen, &msgbuf) == SUCCESS) sendPending = 1; } uint8_t isSendPending() __attribute__((C,spontaneous)) { return sendPending; } event result_t SendMsg.sendDone(TOS_MsgPtr mp, result_t success) { if (!success) call Timer.start(TIMER_ONE_SHOT, 200); else { call Leds.redToggle(); sendPending = 0; } return SUCCESS; }

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

event TOS_MsgPtr ReceiveMsg.receive(TOS_MsgPtr mp) { TOS_Msg m; call Leds.greenToggle(); if ((uint8_t)mp->data[0] == 20) { m.data = deref(detectShadow()); sendPacket((uint8_t *)m.data, strlen(m.data)); } return mp; } event result_t Timer.fired() { return call SendMsg.send(TOS_BCAST_ADDR, msglen, &msgbuf); } /* … */ // Implementation of detectBeamEvent primitive uint16_t detectBeamEvent() { int i; uint16_t data, avg = 0; ledSet(0); for (i = 0; i < 10; i++) avg += IO(2, 0); avg /= 10; while ((data = IO(2, 0)) > avg - 15) yield(); ledSet(7); return list(2, newWord(20), newWord(data)); } }

The majority of the code is not related to the specific application domain (detecting a break beam event) but to managing resources, communication and low-level control flow in the WSN node. As is explained in the following sections, our approach allows an end-user to write the same program by means of simply creating a uQuery, which contains one step: Detect Beam Event. The executable code, corresponding to the one illustrated above is generated by model transformation techniques (see Section 4.3, specifically the last paragraph, and Figure 9). The remainder of the chapter is organized as follows. Section 2 provides an overview of our solution. Sections 3 and 4 describe and illustrate respectively the uQuery representation and execution mechanisms. Section 5 is devoted to our end-user programming interface. Section 6 enumerates some optimization opportunities. Section 7 explores the related work, before concluding in Sections 8 and 9.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

86

R. Razavi et al. / Ambiance

Web Client HTTP request

Web document uQuery Server

Mobile actor code

Platform-specific semantic data AmI Infrastructure

Figure 1. High-level view of the uQuery Engine system architecture

2. Architectural Style of uQuery Engines

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

The key technical requirements of uQuery engines are dynamic end-user programmability by multiple uncoordinated end-users, and automated deployment and execution. To meet these requirements, we propose the architectural style [13] which is illustrated by Figure 1. There are three subsystems as follows. On the top, the Web Client subsystem is responsible for providing uncoordinated concurrent end-users with a domain-specific Web programming interface (for Vehicle Tracking, in our example case). At the bottom, the AmI Infrastructure subsystem provides an interface to hardware and software platforms of the ambient nodes, in our case, mostly WSN nodes. In the middle, the uQuery Server subsystem is responsible for representing uQueries and processing them for execution on the AmI infrastructure. The Web Client subsystem communicates with the uQuery Server subsystem by standard HTTP Requests. This communication serves two main purposes: elaborating the uQuery on the one hand, and executing it on the other hand. Accordingly, the uQuery Server subsystem responds with dynamically generated Web documents, which embody either the current state of the program being built, or its execution results. The uQuery Server subsystem communicates in turn with the AmI Infrastructure subsystem by non-standard mobile actor code bundles in text format (Scheme-like code). It receives semantically meaningful execution results, which belong to the application domain ontology, but in (WSN) platform-specific format. The receiver, uQuery Server subsystem, is responsible for encoding the results into an application-specific format, before embedding them into Web documents. Standard Web 2.0-compliant browsers are used as Web Clients. The AmI Infrastructure subsystem encompasses an open set of heterogeneous ambient nodes. We impose an important constraint on the software interface of these nodes: they must provide dynamic mobile agent code deployment and execution services. In the case of sensor nodes, such interface is provides by platforms like ActorNet [14] and Agilla [15]. As for the main component of this architecture, the uQuery Server subsystem, it is designed and implemented as a meta-level object-oriented application as follows.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

R. Razavi et al. / Ambiance

87

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 2. uQuery Server subsystem architecture

As we said, the uQuery Server has a double purpose of specification and execution. Such a situation has already been studied in Software Engineering, notably by Ralph Johnson and his colleagues under the name of adaptive object-models (AOM) [16]. Accordingly, our Server follows the architectural style of AOMs, which defines a family of architectures for dynamically end-user programmable object-oriented software systems. More particularly, this design belongs to the class of flowindependent AOMs [17], where the expected behavior (here uQueries) is specified at run-time as a high-level description of the control flow between reusable and sharable computational entities that embody the basic application domain algorithms (for example, for vehicle tracking). In the reminder of this chapter, we call the latter services, following the service-oriented computing paradigm, and suggest wrapping and representing the basic functionality provided by ambient nodes also as services. Our design is illustrated in Figure 2, and encompasses four components and several metadata repositories as follows. On the top, the Dynamic Web Application Server Framework component provides the standard functions of a dynamic Web server application, by serving HTTP requests from concurrent clients and by dynamically generating Web documents. We use the Seaside framework (www.seaside.st) for this purpose, which provides a high-level dynamic Web server programming interface by reuse and extension. At the bottom, the Concurrent uQuery Processing Framework component controls uQuery processing (transformation, optimization, and deployment). In the middle, there are two components. The uQuery Execution-History Management Framework component is responsible for storing and providing an extensible interface to access the executions of uQueries, for analyzing, rendering and visualizing results, and also for optimization purposes. This component is not currently mature enough to be communicated here. The Concurrent uQuery Specification Framework component is responsible for uQueries representation. From the functional perspective, this architectural style keeps track of two types of static and dynamic metadata: Static metadata corresponds to the knowledge acquired and represented at built-time concerning the business ontology, and the available services (contracts). They are read- and write-accessible by the Concurrent uQuery Specification Framework component (see meta-repositories in Figure 2). The dynamic

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

88

R. Razavi et al. / Ambiance

metadata corresponds to the knowledge acquired at run-time concerning the userdefined uQueries, their executions (domain objects and uQuery execution history), the available service implementations, and ambient nodes and their resources (see Network Directory Service in Section 6). These are read- and write-accessible by all components, but primarily by the Concurrent uQuery Processing Framework component (see the repositories in Figure 5). Overall, the representation and processing of this metadata is a major role of the architectural style of uQuery engines, specifically its knowledge level. The execution of uQueries per se is the responsibility of the operational level. The knowledge level representation and processing role is shared between the two Concurrent uQuery Programming and Processing Framework components. The processing component provides additionally a hook to the operational level execution role, which is accomplished by AmI infrastructure node platforms (here: ActorNet). The following sections provide more details on the design of these two major components of the uQuery engine architectural style.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

3. Concurrent uQuery Specification Framework We now describe the design of the Concurrent uQuery Specification Framework component for representing uQueries in our system. The processing component will be explained in the Section 4, and the end-user programming interface in Section 5. In [17] we describe Dart, a unique (to the best of our knowledge) method for dynamically specifying control flow in flow-independent AOM architectures that satisfies the requirements of support for dynamicity, end-users accessibility, as well as extendibility and reusability by programmers, in particular for plugging specific execution strategies (such as dynamic multi-agent distributed execution, as it is the case here). We therefore suggest reusing Dart for representing uQueries. Figure 3 illustrates the core design of Dart using the UML notation. A uQuery is represented as a graph, which aggregates tasks where each task is a set of interrelated steps. A step reifies a service request by bridging a construct with a domain object that results from its execution. A construct specifies the computation method for that object. The most common type of construct is the primitive construct, which reifies a low-level primitive service request. By low-level primitive we mean a method or a library function, which implements low-level logic, such as mathematical algorithms, and platform-specific sensing and data processing functionality. A step referring to a primitive construct is called a primitive step (or leaf step). Each construct refers to (or instantiates) a contract which holds metadata about the construct. In the case of primitive constructs, a contract incorporates the type specification, arguments, name and a list of properties required for its execution. In order to execute a construct, we need to make platform-specific decisions, which are delegated by each contract to the associated execution strategy. The service repository for a uQuery engine holds these contracts. The business ontology holds the descriptions of domain concepts, together with their relationships and constraints. Steps hold instances of domain concepts, domain objects, which conform to the ontology. Result and argument specifications refer to domain concepts that define their types. Finally, to each task may be associated one or several organizations, which determines the visual representation of its steps. Different representations, such as list, grid and freeform, are possible since order is irrelevant when executing Dart graphs (see the next Section).

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

R. Razavi et al. / Ambiance

89

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 3. Detailed design of the Concurrent uQuery Specification Framework component in UML

We assume that both the business ontology and the service repository for the business domain are provided. Actually, both may be implemented as part of a single domain ontology written in OWL (with the help of the Protégé editor), and accessed (with the Pellet reasoner) by means of queries written in SPARQL. We further assume that the service repository is comprehensive enough to allow all uQueries of interest to be explicitly represented as a composition of its elements. Of course, such a repository would greatly benefit from the advanced techniques for context representation proposed by Dapoigny and Barlatier [18]. For illustration, consider the parking garage example from Section 1. Pablo wants to make a uQuery to take a photo from a specific camera in a sensor network. We assume a business ontology with one concept, Photo, and a service repository with one contract, Capture Camera, which returns a Photo object, and a library including an implementation of the above service (e.g., low-level nesC code for the Mica2 sensor platform, accessible through an ActorNet interface). The representation of this simple uQuery comprises a single task with a single Capture Camera step, which specifies a camera sensor id as a property. More realistic uQueries, however, require a higher degree of concurrency, control (iterative and conditional), and nesting of tasks. For instance, consider Liz’s uQuery that asks for a vehicle arrival time histogram for a period of two weeks. The additional ontologies required to specify steps for this uQuery are shown in Figure 4. We denote the return value of tasks by an upward arrow, and the “contributes” relation of Figure 3 by a dashed arrow. The entire box represents a uQuery. The boxes 2, 3 and 4 represent tasks. The rest of the boxes represent steps. We skip describing the details of the ontology and services required, which are outlined in the figure.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

90

R. Razavi et al. / Ambiance 1 2 6

5 Histogram Entry Collection

Histogram

For a Period of Do

Plot Histogram

Main 3 16

15 Vehicle

Histogram Entry

Do

Create Histogram Entry T1: Compute Histogram Entry

4 9

8

7

Pulse Pair 3

Mobile Object

Vehicle

Average Pulse Pairs

Estimate Motion

Classify Vehicle

10

11

Pulse Pair 1

Pulse Pair 2

Sort Edges

Sort Edges

12

13

14

Pulse 1

Pulse 2

Pulse 3

Detect Beam Event

Detect Beam Event

Detect Beam Event

T2: Identify Vehicle Liz’s query Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 4. Representation of Liz’s uQuery by three nested and subordinate tasks, and several steps2

The corresponding uQuery, called Liz’s uQuery (1), 3 is represented as three interrelated hierarchical tasks: a main task (2), and two subordinate tasks, called T1: Compute Histogram Entry (3) and T2: Identify Vehicle (4). The main task has two steps: Plot Histogram (5), which returns a Histogram object to the end-user, and the control construct For a Period Do (6) that is in charge of executing T1 for a duration of two weeks. T1 is then subordinate to step (6), which controls its execution. It in turn comprises two steps. The first step instantiates Create Histogram Entry (15), which returns a Histogram Entry object (to the main task). This requires as argument a Vehicle object, provided by the second step of T1, Do (16), which 2

Numbers in ovals are used for annotation, and are not a part of the representation. In this paragraph and the 5th paragraph of Section 5, numbers between parentheses, e.g., (1), refer to the numbered rectangles in respectively Figure 4 and Figure 12. 3

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

91

R. Razavi et al. / Ambiance

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

nests T2 as computation method. T2 comprises several steps (7-14), whose representation is analogous to that of the Capture Camera step described above. The task T2 uses inputs from multiple break beam sensors to detect moving objects and classify them as vehicles. Section 5 describes and illustrates how uQueries like this may be specified using our Web interface. To summarize, this uQuery representation contains occurrences of the following kinds of steps: • Primitive calls: designed as steps that call a primitive operation as their computation method, • Control structures: designed as steps that control the execution of other tasks, such as the For a Period Do step that controls T1, • Nested tasks: designed as steps that use a task as their computation method, e.g., the Do step that uses T2, and thereby simplify the specification of complex uQueries by hierarchically modularizing them. The syntax of Dart is recursive, i.e., steps may hierarchically point to tasks. It is also extensible, i.e., tasks and steps may be used to extend constructs. Arbitrary control constructs can be implemented by uQuery engine programmers using these properties, thus extending the abstract notion of a construct in Dart. We provide by default an abstraction called Control Construct, which has an instance variable called closures (illustrated by the directed arrow from Control Construct to Task in Figure 3), which keeps track of a collection of subordinate tasks whose number and semantics depend on the desired functionality, based on the metadata held by its contract. In the case of the For a Period Do control structure, which iteratively executes a given task for a duration specified as a property, the closures collection holds one task, here a pointer to T1. Its propertyValues instance variable, inherited from its superclass, holds the duration value, i.e., 2 weeks in our example. The representation of nested tasks is close to that of control structures. The closure collection contains only one task, which is executed unconditionally upon each activation of the step.

uQuery meta-repository

uQuery graph

Application-specific semantic data

Domain objects repository

Concurrent uQuery Interpreter Domain ontology meta-repository

Service contract meta-repository

Self-sufficient graph node

Platform-specific semantic data

Concurrent uQuery Node Processor

Service implementation repository uQuery execution-history repository

Figure 5. Concurrent uQuery Processing Framework component architecture

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

92

R. Razavi et al. / Ambiance

4. Concurrent uQueries Processing Framework Now, we explain how uQuery graphs are concurrently executed by describing the design of the Concurrent uQuery Processing Framework component as illustrated in Figure 5. We propose an interpretation mechanism coupled with a truly dynamic deployment of independent code segments, which may interact and migrate, running on distributed sensor nodes. Specifically, in alignment with experimental systems such as Actalk [19], this component is designed as a framework for concurrent execution of uQueries’ object-based representations. Specifically, we use mobile actors as the execution model, which ensures feasibility and improves scalability by splitting computations on appropriate nodes and merging them according to resource availability and control flow. We separate the actor at the operational level, which is a thread responsible for executing primitive steps, from the meta-actor at the knowledge level, which is a thread that represent control flow through the computation, controls the execution on a single uQuery graph node, and carries state. Moreover, our architecture allows meta-level objects to modify their behavior in more fundamental ways through introspection and intercession, for example, if the meta-actors are endowed with learning mechanisms. In our current example, a vehicle identification uQuery is decomposed in terms of a set of meta-actors. These meta-actors control the concurrent and distributed collection of data by actors dynamically deployed on beam and camera sensors nodes to process and analyze the data. Note that the analysis may trigger further data collection, or reuse previously collected data. Subsections 4.1, 4.2, and 4.3 describe the design of three subcomponents that implement these decisions.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

4.1. Concurrent uQuery Interpreter The Concurrent uQuery Interpreter subcomponent is responsible for concurrent uQuery graph traversal, meta-actor creation, encoding execution results into application-specific formats, and storing execution results and uQuery executions. It reads uQuery graphs from the corresponding repository and updates related repositories. In particular, uQuery enactments history and domain objects repositories are respectively updated with uQuey executions and application-specific semantic data (domain objects). The execution results are received in a platform-specific format, and are encoded into application-specific formats using correspondence tables. In general, all domain objects that result from the execution of steps in a uQuery are stored. The final result of the execution of a uQuery corresponds to the result of the execution of the step in the main task that is tagged as the return value. The graph traversal algorithm is associated with meta-actor type hierarchy illustrated in Figure 6 using the UML notation. The algorithm associates a meta-actor to each graph node. Dart node types present in uQuery graphs, i.e., uQuery, Task and Step are respectively mapped to uQuery Meta-Actor, Task Meta-Actor, and Step MetaActor types. Each meta-actor is uniquely responsible for processing its corresponding node. The process is initiated by creating a uQuery meta-actor, which recursively takes charge of the creation of the other meta-actors. The underlying algorithm is described in the next subsection. Each meta-actor is concurrently linked to the whole system only by knowing the address (called “actor name” in Actor terminology) of its parent

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

93

R. Razavi et al. / Ambiance

meta-actor, which is the meta-actor that has created it. Each meta-actor keeps also track of all its child meta-actors. The uQuery meta-actor has access to the whole system of meta-actors, which is structured as a distributed connected graph (since uQuery graphs are connected by construction). The parent of all uQuery meta-actors is the uQuery engine, which is itself designed as a meta-actor type (see Figure 6). The order is irrelevant in creating meta-actors. Concurrent execution of metaactors is allowed by construction, since the execution of Dart graph nodes is only restricted by their data dependencies. In other words, those nodes that do not have direct or indirect data dependencies, or have their required data available, may execute in parallel at any time. The interpretation process for a uQuery reaches its end when all relevant meta-actors have returned from execution (possibly with an error). The occurrence of an error in one node may result in garbage collecting the whole system of corresponding meta-actors and stopping the execution, or spawning a new meta-actor for another try, for example using different resources. Algorithms for nonintrusive, scalable distributed mobile active object garbage collection, proposed by Wei-Jen Wang and Carlos A. Varela [20], may be considered here. The next Subsection describes the design of meta-actors and their processing algorithm per se.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

4.2. Concurrent uQuery Node Processor The Concurrent uQuery Node Processor subcomponent is responsible for processing a single graph node and returning its result, when applicable. Results are provided in a platform-specific format. Autonomous and concurrent processing of nodes is possible since Dart graph nodes are self-sufficient. Self-sufficiency refers to the fact that each individual service request embedded in a graph node is provided, by design, thanks to pointers to the service contract and its argument steps, with the required knowledge about the arguments, resources, context and method required for its execution. Metaactors are designed as active objects [19], executing concurrently in separate threads, communicating and synchronizing through asynchronous message passing, specifically for fetching the required data. They arrive to the end of their lifecycle when the execution is terminated (either successfully or by an exception).

steps

Abstract Meta-Actor

Step Meta-Actor

Primitive Step Meta-Actor

Composite Meta-Actor

uQuery Meta-Actor

Task Meta-Actor Control Meta-Actor

Ambient Node Step Meta-Actor uQuery Engine Figure 6. Detailed design of the Concurrent uQuery Processing Framework component in UML

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

94

R. Razavi et al. / Ambiance

The interface of meta-actors comprises start(), body(), and stop() methods. The body() method specifies the core function of meta-actors, which contains four steps: (1) checking preconditions for executability, (2) initializing data structures, (3) executing the behavior, and (4) sending an asynchronous message to the parent to notify it of the execution result. The behavior execution step (No. 3) is specific to each kind of meta-actor. As we have explained in [21], it may comprise a set of sophisticated actions, including implementation matching, resource matching, decision making and optimization, online scheduling and task allocation, marshaling, and deployment. Specifically, meta-actors screen the runtime environment, allocate resources, spawn other meta-actors and actors, and control the lifecycle of those actors. Such a lifecycle, described in more detail in the next subsection, involves activation and registration, request management, application logic, and result dissemination. Dart graph nodes are also minimally-constrained. Minimally-constrained refers to delaying as long as possible placing constraints necessary to execute a specific instance of the service, in other words, the service instance does not refer to information that can be computed or supplied to it at run-time. This characteristic, together with the openness, resource-limitedness and error-proneness attributes of the AmI infrastructure have motivated that sophisticated processing architecture. At the beginning of their execution, uQuery (root) meta-actors launch a meta-actor for their main task. The main task meta-actor continues the execution by launching step meta-actors. A step meta-actor can only execute in two cases: (1) it is associated to a root step, i.e., a step whose construct requires no arguments, or (2) all its effective arguments are available. The execution strategy associated with each step is in charge of guiding this process. uQuery and task meta-actors are designed as Composite MetaActors, which provides functionality for launching a set of child meta-actors, here respectively tasks and steps, and controlling their execution. If the construct of a step points hierarchically to tasks and subordinate tasks, then the corresponding step metaactor also creates hierarchically subordinate task meta-actors. The above execution algorithm is applied recursively to the latter. For example, a uQuery meta-actor is created and launched for executing Liz’s uQuery. This meta-actor launches a task meta-actor for the main task, which in turn launches a step meta-actor for its unique root step, i.e., For a Period Do. The other step of the main task is not executable since it needs an argument. The For a Period Do meta-actor starts a timer to control the number of iterations. Each iteration consists of launching a subordinate task meta-actor for executing the same task, T1. The execution result of these meta-actors is collected and constitutes the result of the execution of the For a Period Do step. At this point, the main task meta-actor passes this collection to the Plot Histogram meta-actor and resumes it. In the case of T1, there is also a unique root step, which is Do . The execution of the associated T1 meta-actor consists of creating a subordinate T2 task meta-actor. The T1 metaactor sleeps then on a semaphore, waiting for the T2 meta-actor to return a Vehicle or signal an error. The domain objects that are expected as uQuery enactment result are basically obtained from the execution of meta-actors in charge of primitive step nodes. The role of the other meta-actors is essentially to control the execution flow. Primitive step meta-actors may proceed in two ways for computing their results: interpretation or compilation. Interpretation is used when the primitive step embodies a local function call. In that case, the execution of that function is invoked locally, by computing and

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

95

R. Razavi et al. / Ambiance

passing the required arguments. On the fly code generation, deployment and execution is used in the more complex case of ambient node primitive steps, which embody a sensing or actuating service request that must be executed on an ambient node. This process is implemented by a specialization of the current subcomponent and is described in the next Subsection. 4.3. Ambient Node Primitive Processor

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

The Ambient Node Primitive Processor subcomponent is responsible for executing ambient node primitive steps. It collects raw sensor data, transforms them into domain objects encoded into platform-specific representations, and routes them to the uQuery processor subcomponent. This requires access to a platform interface that provides (1) deploying and executing dynamically generated agent code, (2) dynamically discovering and providing access to all sensors in the WSN, and (3) implementing the elements of the service repository. To the best of our knowledge, the only existing system that fully satisfies our operational level requirements is ActorNet [14]. ActorNet agents are based on the actor model of computation [22]. The ActorNet runtime, illustrated in Figure 7, consists of a lightweight interpreter running on each sensor node in the WSN, along with several supporting services. The runtime enables the actors to execute, communicate, migrate and access sensors and actuators. Figure 8 illustrates our approach for integrating a platform such as ActorNet. First, the actor template in Figure 9 is filled by the Code Generator module. Each Ambient Node Primitive Meta-actor having its preconditions satisfied and being signaled, directly or indirectly, by a returning step of the main task, can generate the code for its associated actor, in parallel with other meta-actors. The generated code pieces are deployed to the actor platform by the meta-actors, through the Actor Deployment Interface (ADI). The ADI is a multithreaded server providing socket connections over the Internet for concurrently and remotely deploying and executing actors.

Mobile agent code

Platform-specific semantic data

Actor Deployement Service

Messaging Service

Actor Interpreter Garbage Collector

Communication Driver VM Driver

Application Level Context Switcher OS (TinyOS, ContikiOS, etc.) Hardware (Mica2, iMote, MICAz, etc.) Figure 7. The architecture of ActorNet, a major module of the Ambient Node Primitive Processor subcomponent

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

96

R. Razavi et al. / Ambiance

Marshaled semantic data

Ambient node primitive step

Actor Messaging Interface

Code Generator Actor code

Platform-specific semantic data

Actor Deployment Interface

Mobile Actor Platform Compact abstract syntax tree

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

.

Figure 8. Interactions between modules involved in executing sensor primitive steps

Once deployed, the agent is parsed and transformed from a text string to a more compact abstract syntax tree representation, more suitable to sending across lowbandwidth wireless networks. The id of its generating meta-actor is embedded in the actor code, enabling the actor to communicate with its associated meta-actor through the Actor Messaging Interface. The meta-actor can sleep until it receives the actor’s output object through the messaging interface, after the actor finishes its computation. Arguments and return values are marshaled and unmarshaled using the information kept by the execution strategy. In case a primitive cannot successfully complete its computation, a failure is signaled to the relevant meta-actor, which may trigger garbage collection for the task and its associated actors and meta-actors. The code generator fills the template illustrated in Figure 9 in four locations: metaactor id on lines 5 and 11, application logic and list of arguments on line 14, and execution location and uQuery engine server address on lines 5 and 12, respectively. In Figure 9, semicolons ‘;’ indicate comments. Characters in italics indicate values filled in by the code generator, as an example, for the case of the three Detect Beam Event steps in the T2 meta-actor shown in Figure 4. As Beam Event objects are received as results of these executions, the Extract Edge meta-actors may be signaled concurrently, and so on. Note that this ActorNet code segment embodies a much more compact, higherlevel specification of functionality similar to that of the NesC code in Section 1.2. One actor is launched, and it calls the Detect Beam Event primitive (primitive function with index 1 in line 12 Figure 9), which blocks until an event is detected. Scheme functions are used for communication, migration and control flow, while low-level resource management is left up to the ActorNet interpreter and the underlying services. 4.4. Complex uQueries Now let us consider a uQuery which requires distributed coordination between different sensors. Such a uQuery cannot be executed as a single mobile agent. For example, Pablo wants to take a photo with a camera only when a nearby break beam sensor is triggered. The task that represents this uQuery has two steps: detecting a Pulse object (generated by the beam sensor) and then taking a Photo with the camera.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

R. Razavi et al. / Ambiance

1.( 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21.

(lambda (migrate)

; start of the actor function

(seq

; sequentially execute the following:

97

(migrate 200 111)

; migrate to the destination node ; (destination id 200 and meta-actor id ; 111 are given as parameters)

(par

; create a new actor to reduce code size

(extmsg 111 (migrate 100

; send result back to the meta-actor… ; after migrating back to the source node

(prim 1 nil)))))) ; ; ; ;

execute primitive function with index 1 and with an empty list of arguments Note: the primitive is evaluated first, then migration, then sending the message

(lambda (adrs val) ; implementation of the migrate function (callcc (lambda (cc) ; using “call with current continuation” (send (list adrs cc (list quote val)))))))

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 9. ActorNet mobile agent dynamically created by instantiating and filling a generic agent template

For executing such uQueries in a coordinated manner, we use the argument relation in our meta-model (see Figure 3). Whenever an argument is encountered in the uQuery specification, the knowledge level automatically creates a dependency relation between the meta-actors for these two steps. Meta-actors with dependencies check whether they have all the arguments necessary to generate and execute their associated actors. In our example, the meta-actor for capturing a Photo will be made dependent on the Detect Beam Event meta-actor. It will be deployed and executed only when a Pulse object is returned by the latter. We can now consider a scenario where uncoordinated concurrent uQueries are entered into the system by two different users, Liz and Pablo. Pablo is interested in taking photos of Vehicles when they speed through break beam sensors. We assume that Liz’s uQuery remains the same as in the previous section. Each uQuery is represented independently, i.e., there is no structural relationship between uQuery representations. In this case, the execution for each uQuery is triggered concurrently at the knowledge level, according to the algorithm described above. This execution procedure can be subject to optimization, as discussed below.

5. End-User Programming Web Interface Implementing end-user programming languages that satisfy both technical and cognitive requirements is a challenging task. This may explain why Pane and Myers observe that, when new programming languages are designed, the findings of Psychology of Programming (PoP) and related fields such as the Cognitive Dimensions Framework [23] do not seem to have much impact [24].

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

98

R. Razavi et al. / Ambiance

Tasks Service Repository

Steps

Business Ontology Concepts uQuery

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 10. A snapshot of the Web interface of our uQuery engine prototype

Dart is, on the contrary, designed taking these attributes as primary considerations. We have chosen the spreadsheet as a basis for designing the end-user programming features of Dart, since it is the most widely used end-user programming metaphor [25]. The two main underlying assumptions are that (1) our end-users belong to the large category of domain experts, who are described by Bonnie A. Nardi as “having the detailed task knowledge necessary for creating the knowledge-rich applications they want and the motivation to get their work done quickly, to a high standard of accuracy and completeness” [25], and (2) complex behavior can be modeled by (a) associating operations to cells that compute objects, and (b) relating together those cells as “operation result holders” that provide the argument to new operations. This is how programming takes place through Dart-based end-user programming interfaces. As illustrated by Figure 10, in the Web-enabled end-user programming interface that we have prototyped for the case of uncoordinated end-users Liz and Pablo, a uQuery is specified by direct manipulation techniques as a collection of tasks, where each task comprises a set of steps. Each task frame corresponds to a sheet in the spreadsheet paradigm. Each step plays the role of an “operation result holder”, and is defined by specifying a call to a domain-related service, available from the service repository. This corresponds to the cell-operation association in the spreadsheet

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

R. Razavi et al. / Ambiance

99

paradigm. It should be noted, however, that the interface in Figure 10 does not match the traditional spreadsheet interfaces, which use a grid layout. Nevertheless, the programming method remains consistent with the spreadsheet paradigm sketched out above. As explained in Sections 3 and 4, steps may be graphically organized using different visual representations, such as list (Figure 10), grid, or even freeform (like in Figure 4), without any semantic impact. Step specifications may refer to the expected results of previous calls, in terms of ontology concepts. The data flow in such cases is specified by dragging the step’s visual representation and dropping it into the instruction editing area, situated at the bottom of the programming interface (see Figure 10). That area is then expanded with a set of fill-in widgets, one per required argument (see Figure 11). The end-user may then select from a menu the name of the appropriate step, and validate the modification. For example, the two arguments of the first Sort Edge step (fourth line in the list of steps in Figure 10) are set to Pulse_1 and Pulse_2, which refer to the names assigned automatically to the results of steps in the first two lines of the same task. The same method is used for modifying data flows. A more sophisticated interface may allow specifying data flow by direct drag-and-drops (without using the editing area). Figure 10 illustrates more specifically the specification of T2: Identify Vehicle, which is a subordinate task and part of the Liz’s uQuery, described and illustrated in Section 3 and Figure 4. The other tasks and steps of that uQuery may be defined in the same way. Pablo may define its uQuery by reusing T2: Identify Vehicle (assuming it is already defined by Liz), and creating two new tasks (see Figure 12). A main task (2), and another subordinate task, called T3: Photograph Speedy Vehicle (3). The latter nests a call to T2: Identify Vehicle (5). Each time a vehicle is identified by (5), it is passed to a primitive step called Photograph If Speedy (6), which takes a photo of the vehicle if it is speedy. This photo is then sent back to the caller main task, as the result of the execution of (3), which stores it in a collection and also sends it to Pablo (for example by email). This procedure repeats forever due to the definition of the unique step of the main task. The same uQuery may be defined in alternative ways, depending on how the service repository is designed. For example, instead of Photograph If Speedy (6), the uQuery engine may propose primitives like Check If Speedy and Capture Camera (described also in Section 3), which would allow a more fine-grained definition of (6). Instead of screenshots, we have chosen the “manual” notation of uQueries for illustrating Pablo’s uQuery in Figure 12, since it is more compact.

Figure 11. Example of data-flow editing by the Web interface of our uQuery engine prototype

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

100

R. Razavi et al. / Ambiance 1 2 4 Collection of Photo Forever Do Main 3 5

6 Vehicle

Photo of Speedy Vehicle

Do

Photograph If Speedy T3: Photograph Speedy Vehicle Pablo’s query

Figure 12. Representation of Pablo’s uQuery by three tasks, one reused (T2: Identify Vehicle)

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

It is worthy to note that Dart extends the classic features of spreadsheets by allowing the users to integrate a set of domain and task-specific built-in primitives with relative ease, to associate any type of business object to the spreadsheet cells, and to use that object as an argument for defining new formulas (service calls), to incorporate user-defined functions comparable to those described in [26], and to take advantage of a rich set of general-purpose and domain-related control structures. Dart is designed as an abstract object-oriented framework. It was initially implemented in Cincom VisualWorks Smalltalk [27], and used to refactor and improve the end-user programming language of an ecological simulation system. In this case, the end-users are experts in ecology, whose job is to model ecosystems (individual and social behavior of animals, plants, etc.) and to observe and study the evolution of their behavior through simulation and statistics [28]. Our new implementation of Dart is based upon Squeak 2.8 (www.squeak.org) and Seaside 2.7 systems.

6. Optimization The uQuery engine design presents several opportunities for optimization as follows. Knowledge level optimizations: Our uQuery representation allows composing a computation history for every business object by looking recursively at the constructs and objects used in its computation. The computation history allows access to the metadata associated with each element of the history (see Figure 3). As a result, more complex matching algorithms can be implemented. For example, a uQuery can compare the conditions under which the available business object has been computed, and whether they match the requirements of the current uQuery. Incidentally, computation histories are also useful for auditability, that is identifying precisely all computation steps and their contexts for a given event in the system. The same feature allows satisfying non-repudiability, by computing who has initiated which action. Operational level optimizations: We consider also the functional specification of queries, where the user only specifies the desired functionality without explicitly listing all implementation details, such as the identifiers or location coordinates of sensors.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

R. Razavi et al. / Ambiance

101

The uQuery processing engine can infer the requirements for these resources from the static metadata associated with the uQuery and the contracts of the primitives it invokes. We can then fill in specific sensors and other parameters matching these criteria. To find suitable sensing resources meeting these constraints, we extend ActorNet with the Network Directory Service (NDS), which performs network monitoring and provides an up-to-date listing of sensor and computing resources available in the network. The NDS is searchable by sensor type, location and other attributes, listed as name-value pairs. Thus the uQuery engine can find appropriate sensors for a uQuery by looking at attributes such as location and orientation. This simplifies the uQuery creation process for the user and provides opportunities for optimizing uQuery execution, by enabling the uQuery Engine to pick the best resources (e.g., closest, least congested, or having the most remaining battery power) satisfying the uQuery specification. Further, the mobility of actors allows for load balancing and resource-aware task allocation, even as new actors are added. In the parking garage example, we can choose, based on the current communication traffic and available computing power, whether to move the raw break beam sensor data to a PC for processing, or to move the vehicle detection code, as a system of mobile agents, to the sensor where the data is generated. Also, by exposing available resources (sensors, actuators, data stores, etc.) to the knowledge level, compatible resources may be substituted for one another at runtime to simplify scheduling or reduce congestion. Concurrent uQuery optimizations: Data computed in one uQuery can be reused to satisfy requirements of another. This mechanism is based on exploiting static and dynamic metadata maintained by the uQuery engine (see Section 2). For example, a Vehicle object produced by one of the queries described above is associated with dynamic metadata such as a timestamp and the detecting beam sensor’s id (see Execution Context abstraction in Figure 3). When processing a uQuery with a step requiring a Vehicle object as argument, e.g., Compute Histogram Entry in task T1 above, the already-computed Vehicle may be substituted instead of executing the Identify Vehicle subtask, assuming the dynamic metadata of the object matches the constraints within the static metadata of the uQuery. Such matching algorithms can be implemented using an inference engine, such as NéOpus [29], which provides rulebased reasoning in object-oriented applications. This process eliminates generating, deploying and executing redundant actors at the operational level, with the benefit of saving significant communication, sensing and computation resources in the WSN, where these resources are scarce.

7. Related Work A survey of solutions currently proposed in the literature reveals a variety of approaches to macroprogramming WSNs: a spreadsheet approach [10], EnviroSuite [6], a market-based approach [12], and Semantic Streams [5]. Although many of these approaches are quite powerful, none of them provide the language abstractions required for dynamic macroprogramming by end-users as outlined above. For example, the spreadsheet approach uses an Excel spreadsheet to represent the layout of nodes and insert their functionality in the spreadsheet; queries are resolved by a logic program that generates a composition of services, where a service is a .Net component. The approach satisfies the requirement of usability by non-programmers. However, it is not sufficiently general: it enforces a particular naming grid-based

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

102

R. Razavi et al. / Ambiance

scheme and does not allow for the definition of arbitrary groups of nodes and operations over such groups. EnviroSuite proposes environmentally immersive programming, an object-based programming model in which individual objects represent physical elements in the external environment. In both EnviroSuite and ActorNet, actors or objects must be created explicitly by programmers to provide a service. Behavioral specifications are not in terms of groups of actors. Protocols to support operations over groups of objects and protocols to implement such specifications may not be re-used. Traditionally, WSN application development involved fairly static programming languages, operating systems and reprogramming services, for efficiency reasons. For example in TinyOS, the sensor network application components are written in nesC and compiled together with the operating system code and middleware services into a single application image, which can be uploaded to the sensor nodes using the Deluge protocol [30] prior to program execution. This approach proves successful in achieving its stated goal of highly efficient utilization of sparse computing resources. Unfortunately, it is ill-suited for an open system comprising a dynamic set of diverse, transient tasks that is the expected workload in ambient systems. If we take Deluge as the deployment method for our target systems (where queries from multiple users, all specified at runtime, need to be transformed into executable code, uploaded on the deployed sensor network and executed), this results in unnecessarily transmitting a large volume of code which has not changed (OS components, routing protocol, etc.) along with the newly-generated application code. Other comparable approaches include the Tenet architecture for tiered sensor networks [31], Sensor Webs [32], and Sensor Grids [33]. A major difference from our architecture is that we don’t attribute a priori ‘master’ and ‘mote’ roles to the architecture components, masters being traditional artifacts having the responsibility to control the behavior of motes. In addition, none of these architectures provide the combination of an expressive and easy-to-use end-user programming for uQuery specification (Dart) with a Turing-complete mobile agent language for deployment and execution (ActorNet). Further, network flooding techniques are in general used for dynamic code deployment, instead of fine-grained code deployment available in a mobile agent platform like ActorNet. Finally, P. Levis et al. [34, 35] observe that a fairly complicated action, such as transmitting a message over the radio, could be represented as a single bytecode instruction provided by an application-specific instruction set, and provides a framework for implementing high-level application-specific virtual machines on motes and for disseminating bytecode. Dart behaves, in some manner, as a framework for developing application-specific instruction sets, and thereby allows developing uQuery engines by reuse. Its coupling with a mobile agent language as explained in this chapter provides it with a powerful execution facility on motes, which is more expressive that a virtual machine such as Maté.

8. Conclusion Ambient Intelligence technologies will enable novel applications and new work practices in many fields. AmI will provide for the integration of real-time data into everyday life activities, enabling real-time decision making and workflow process

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

R. Razavi et al. / Ambiance

103

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

definition and modification. Such dynamicity will facilitate responding to situations more efficiently, with a higher degree of quality and end-user satisfaction. In this chapter, we explained how dynamic uQuery programming by end-users can be achieved for ambient systems comprising WSNs and traditional computing artifacts such as PCs, gateway nodes and handheld mobile devices, by extending the architectural style of Adaptive Object-Models. The resulting two-level approach to architecting uQuery engines allows separating uQuery representation and reasoning concerns from those of their effective execution on diverse runtime platforms through model-to-code transformation. The knowledge level comprises the representation of a domain ontology and a service repository, together with the uQuery composition machinery, and implements the rules that govern uQuery transformation, coordination and optimization. Our approach allows for effective end-user uQuery specification and automated execution. The uQuery representation meta-model, designed with extendibility and reusability as primary considerations, allows uQuery engine programmers to add specific constructs via classical object-oriented techniques. Business logic primitives are separated from the core of the mobile agent system, facilitating addition of new domain-specific primitives to the system. Representations of queries are transformed to platform-specific code for ActorNet, dynamically deployed and executed. Using ActorNet as the uQuery execution environment provides the dynamicity of macroprogramming, while enabling load balancing and other optimizations to take place. We thereby combine both expressiveness of a Turing-complete language with the simplicity of a domain-related language. The presented meta-level and mobile-agent architecture for implementing uQuery engines is prototyped by reusing and extending our previous work on ActorNet and its implementation on Mica2 motes [14], and Dart implemented as an object-oriented framework in different dialects of Smalltalk [27, 17]. This prototype is called Ambiance Platform and employs a new implementation of Dart in Squeak. We have further used the Seaside framework for developing the Web-enabled uQuery programming interface of Ambiance Platform.

9. Perspectives A number of important issues remain to be explored. These include dynamic and seamless integration of sensor and actuator nodes, sensitivity to privacy concerns and trustworthiness, and coherence and integrity analysis of uQueries. For example, the computation history discussed in Section 6 allows security enforcement through dynamic decision making about whether to execute the current step or not. Significant optimization may also be possible, for instance by integrating learning mechanisms into uQuery specification, transformation and execution. A first report of our ongoing work on these topics and more particularly on dynamic global resource management appears in [21]. Our longer term plans comprise topics such as a more robust and scalable networking subsystem for disseminating mobile code, and reliably delivering responses, in particular in presence of disconnections caused by mobility. For reliability and fault-tolerance, we would like to explore the integration of techniques for exception handling in presence of asynchronous active objects as proposed by Ch. Dony et al. [36].

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

104

R. Razavi et al. / Ambiance

Furthermore, ambient nodes and more particularly wireless sensors, actuators and Radio Frequency Identification (RFID) tags, allow developing digitally augmented real-life products with better value for both the manufacturers and customers. The distinguishing feature of these ambient products is their aptitude to smoothly incorporate themselves into the larger context of physically augmented business processes, by providing real-time “field” knowledge all along their lifecycle. Smart features of ambient processes such as personalization, predication, and anticipation, allow delivering unprecedented services to end-users and thereby increasing significantly competitive advantages as well as customer comfort, safety, security and satisfaction. Another application perspective of this work consists of developing networks of collaborating ambient products to enable applications such as predictive maintenance, reverse logistics, and ambient traceability (full lifecycle product and process identification, authentication, location and security). Finally, we would like to explore conditions under which domain experts could be replaced by monitoring agents endowed with the same expertise. We plan considering morphology-based techniques elaborated by Campagne & Cardon [37] to simulate emotions in a robot, where so-called analysis agents are expected to control a population of subordinate (and massively multiple) aspectual agents. These analysis agents are supposed to possess cognitive capacities and to modify the behavior of aspectual agents. For this purpose, the structure of aspectual agents is designed to be changeable at runtime. Their behavior is “governed by some sort of augmented transition network (ATN) that can be parameterized by a set of values” [37]. This is precisely where our proposed architecture would be employed.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Acknowledgments This work is partially funded by the University of Luxembourg, in the framework of the Åmbiance project (R1F105K04) and NSF under grant CNS 05-09321, by ONR under DoD MURI award N0014-02-1-0715 to UIUC. The authors would also like to acknowledge the valuable comments and collaboration of J.-C. Augusto, F. Arbab, A. Cardon, N. Bouraqadi, P. Bouvry, Ch. Dony, B. Foote, the late and deeply regretted V. Ginot, R. Karmani, Ph. Krief, R. Johnson, M. Malvetti, S. Nadjm-Tehrani, D. Riehle, D. Shapiro, S. Sundresh, and J. Yoder.

References [1] [2] [3] [4] [5]

[6]

IST Advisory Group. “Ambient Intelligence: from vision to reality - For participation in society & business,” September 2003. Ducatel, K., Bogdanowicz, M., Scapolo, F., Leijten, J. and Burgelman, J-C. (eds.). “ISTAG Scenarios for Ambient Intelligence in 2010,” IPTS-ISTAG, European Commission, 2001. Berger, M., Fuchs, F. and Pirker, M. “Ambient Intelligence – From Personal Assistance to Intelligent Megacities,” this volume, IOS Press, 2007. Nakashima, H. “Cyber Assist Project for Ambient Intelligence,” Section 3.5.1 of this volume, IOS Press, 2007. Whitehouse, K., Zhao, F. and Liu, J. “Semantic Streams: A Framework for Composable Semantic Interpretation of Sensor Data,” Third European Conference on Wireless Sensor Networks, Zurich, Switzerland, pp. 5-20, February 2006. Luo, L., Abdelzaher, F., He, T. and Stankovic, J.A. “EnviroSuite: An Environmentally Immersive Programming Framework for Sensor Networks,” ACM Trans.on Embedded Computing Systems, 2006.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

R. Razavi et al. / Ambiance

[7] [8] [9] [10]

[11]

[12] [13]

[14]

[15]

[16] [17]

[18] [19]

[20]

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

[21] [22] [23] [24]

[25] [26] [27]

[28] [29] [30]

105

Böhlen, M. “Help From Strangers - Media Arts In Ambient Intelligence Research,” this volume, IOS Press, 2007. Richard, N. and Yamada, S. “Two Issues for an Ambient Reminding System: Context-Awareness and User Feedback,” this volume, IOS Press, 2007. Huuskonen, P. “Run to the Hills! Ubiquitous Computing Meltdown,” this volume, IOS Press, 2007. Woo, A., Seth, S., Olson, T., Liu J. and Zhao, F. “A Spreadsheet Approach to Programming and Managing Sensor Networks,” Fifth International Conference on Information Processing in Sensor Networks Track on Sensor Platform, Tools and Design Methods for Networked Embedded Systems (IPSN SPOTS), Nashville, Tennessee, USA, April 2006. Satoh, I. “Mobile Agents for Ambient Intelligence,” Ishida, T., Gasser, L., and Nakashima, H. (eds.), Massively Multi-Agent Systems I, Lecture Notes in Artificial Intelligence, Springer Verlag, vol. 3446, pp. 187-201, 2005. Mainland, G., Kang, L., Lahaie, S., Parkes, D.C., and Welsh, M. “Using Virtual Markets to Program Global Behavior in Sensor Networks,” Eleventh ACM SIGOPS European Workshop, Belgium, 2004. Clements, P., Bachmann, F., Bass, L., Garlan, D., Ivers, J., Little, R., Nord, R. and Stafford, J. “Documenting Software Architectures: Views and Beyond,” Addison-Wesley Professional, ISBN: 0201703726, September 2002. Kwon, Y., Sundresh, S., Mechitov, K. and Agha, G. “ActorNet: An Actor Platform for Wireless Sensor Networks,” Fifth International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS 2006) , Future University, Hakodate, Japan, May 2006. Fok, C.L., Roman, G.-C. and Lu, C. “Mobile Agent Middleware for Sensor Networks: An Application Case Study,” Fourth International Conference on Information Processing in Sensor Networks (IPSN'05), Los Angeles, California, pp. 382-387, April 2005. Yoder, J. W. and Johnson, R. E. “The Adaptive Object-Model Architectural Style,” Third IEEE/IFIP Conference on Software Architecture (WICSA3), pp. 3-27, Montréal, Canada, August 2002. Razavi, R., Perrot, J.-F. and Johnson, R. E. “Dart: A Meta-Level Object-Oriented Framework for TaskSpecific, Artifact-Driven Behavior Modeling,” Sixth OOPSLA Workshop on Domain-Specific Modeling (DSM’06), Gray, J., Tol-vanen, J.-P., Sprinkle, J. (eds.), Computer Science and Information System Reports, Technical Reports, TR-37, University of Jyväskylä, Finland, pp. 43-55, October 2006. Dapoigny R. and Barlatier, P. “Towards a Context Theory for Context-aware systems,” this volume, IOS Press, 2007. Briot, J.-P. “Actalk: A framework for object-oriented concurrent programming - design and experience,” In Bahsoun,J.-P., Baba,T., Briot, J.-P., and Yonezawa, A. (eds.), Object-Oriented Parallel and Distributed Programming, Hermès Science Publications, Paris, France, pages 209-231, 2000. Wang W.-J. and Varela, C. A. “Distributed Garbage Collection for Mobile Actor Systems: The Pseudo Root Approach,” First International Conference on Grid and Pervasive Computing (GPC 2006), Lecture Notes in Computer Science, Springer, vol. 3947, pp. 360-372, Taichung, Taiwan, May 2006. Mechitov, K., Razavi, R. and Agha, G. “Architecture Design Principles to Support Adaptive Service Orchestration in WSN Applications,” ACM SIGBED Review, vol. 4, no. 3, 2007. Agha, G. “Actors: a Model of Concurrent Computation in Distributed Systems,” MIT Press, 1986. Green, T. R. G. and Petre, M. “Usability analysis of visual programming environments: a ‘cognitive dimensions’ framework,” Journal of Visual Languages and Computing 7, 131-174, 1996. Pane, J.F., Myers, B.A. “The Influence of the Psychology of Programming on a Language Design: Project Status Report,” Proceedings of the 12th Annual Meeting of the Psychology of Programmers Interest Group, A.F. Blackwell and E. Bilotta (eds.), Italy: Edizioni Memoria, p. 193-205, 2000. Nardi, B.A. “A Small Matter of Programming: Perspectives on End User Computing,” MIT Press, 1993. Peyton-Jones, S., Blackwell, A. and Burnett, M. “A User-Centred Approach to Functions in Excel,” International Conference on Functional Programming, ACM, Uppsala, Sweden, p. 165-176, 2003. Razavi, R. “Tools for Adaptive Object-Models – Adaptation, Refactoring and Reflection,” (in French: “Outils pour les Langages d'Experts – Adaptation, Refactoring et Réflexivité”) Université Pierre et Marie Curie, Department of Computer Science Technical Report (based on doctoral dissertation), vol. LIP6 2002/014, 285 pages, Paris, France, November 2001. Ginot, V., Le Page, C., and Souissi, S. “A multi-agents architecture to enhance end-user individualbased modeling,” Ecological Modeling 157, pp. 23-41, 2002. Pachet, F., and Perrot, J.-F. “Rule firing with metarules,” Sixth International Conference on Software Engineering and Knowledge Engineering (SEKE’94), pp. 322–29, 1994. Hui, J.W. and Culler, D. “The Dynamic Behavior of a Data Dissemination Protocol for Network Programming at Scale,” Second ACM Conference on Embedded Networked Sensor Systems (SenSys’04), pp. 81-94, Baltimore, Maryland, November 2004.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

106

R. Razavi et al. / Ambiance

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

[31] Gnawali, O., Greenstein, B., Jang, K.-Y., Joki, A., Paek, J., Vieira, M., Estrin, D., Govindan, R. and Kohler, E. “The TENET Architecture for Tiered Sensor Networks,” Fourth ACM Conference on Embedded Networked Sensor Systems (SenSys’06), Boulder, Colorado, November 2006. [32] Delin, K. A., Jackson, S. P., Johnson, D. W., Burleigh, S. C., Woodrow, R. R., McAuley, J. M., Dohm, J. M., Ip, F., Ferré, T. P.A., Rucker, D. F. and Baker, V. R. “Environmental Studies with the Sensor Web: Principles and Practice,” Sensors 5, pp. 103-117, 2005. [33] Lim, H. B., Teo, Y. M., Mukherjee, P., Lam, V. T., Wong, W. F. and See, S. “Sensor Grid: Integration of Wireless Sensor Networks and the Grid,” Local Computer Networks, 2005. [34] Levis, P. and Culler, D. “Maté: A tiny virtual machine for sensor networks,” Tenth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOSX), pp. 85-95, October 2002. [35] Levis, P., Gay, D. and Culler, D. “Active sensor networks,” Second Symposium on Networked Systems Design & Implementation (NSDI '05), Boston, MA, USA, May 2005. [36] Dony, Ch., Urtado, Ch. and Vauttier, S. “Exception Handling and Asynchronous Active Objects: Issues and Proposal,” Advanced Topics in Exception Handling Techniques, Lecture Notes in Computer Science, Springer, vol. 4119, pp. 81-100, September 2006. [37] Campagne, J. C. and Cardon, A. “Artificial emotions for robots using massive multi-agent systems,” Social Intelligence Design International Conference (SID2003), London, 2003.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Advances in Ambient Intelligence J.C. Augusto and D. Shapiro (Eds.) IOS Press, 2007 © 2007 The authors and IOS Press. All rights reserved.

107

A Human Activity Aware Learning Mobile Music Player Sandor DORNBUSH a,1 , Anupam JOSHI a and Zary SEGALL a and Tim OATES a a Computer Science Department, University of Maryland Baltimore County, USA Abstract. The XPod system aims to integrate awareness of human activity and multimedia preferences to produce an adaptive system that plays contextually appropriate media. The XPod project introduces a “smart” music player that learns its user’s preferences and activity, and tailors its music selections accordingly. We have experimented with various physiological sensing platforms to measure a user’s physiological state. These devices are able to monitor a number of variables to determine the user’s levels of activity and motion to predict what music is appropriate at that time. The XPod user trains the player to understand what music is preferred and under what conditions. After training, XPod can predict the desirability of a song given the user’s physical state. XPod learns a users listening preferences from the interaction with the user and from collaborative filtering Keywords. Context aware, machine learning, music

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Introduction In this paper we analyze the problem of creating a music player that plays contextually appropriate music by using a system of sensors to monitor a user’s physiological state. We propose using statistical techniques to estimate a user’s preference for songs in various situations. Several machine learning algorithms were used to model the complex relationship between an individual’s musical preferences and his or her activities. We studied several machine learning systems on a modified version of the existing mobile MP3 player, XPod. This player attempts to select the song best suited to the current activity of the user. XPod was previously reported on in [1], wherein the system was designed to use a database and a neural network to suggest music to users and [2] where different machine learning techniques where investigated. We have proposed a new type of recommendation system, one that incorporates community and context in the recommendation process. While there have been several context aware music players and many collaborative filtering music recommendation systems, no group has previously investigated using a collaborative filtering system to help choose contextually appropriate music. The large and diverse community of music fans on Last.FM makes an ideal data source for collaborative filtering. We show that the best music recommendation comes from a combination of collaborative filtering and context sensing. 1 Corresponding Author: Sandor Dornbush, University of Maryland Baltimore County, 1000 Hilltop Circle,

ITE 325, Baltimore, MD, USA; E-mail: [email protected] Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

108

S. Dornbush et al. / A Human Activity Aware Learning Mobile Music Player

1. Related Work There are several areas of study that are relevant to building a context aware media delivery system. We built on the large body of work that studies context aware computer systems. We used statistical techniques from the fields of machine learning, information retrieval and recommendation systems. There are several context aware music systems relevant to this field. 1.1. Context Aware Context aware systems can greatly improve a user’s experience with computer systems. For example several systems relate user activity to mobile phones [3,4]. Much of the work in context aware systems relates to determining the context of the user or object. The are a couple of approaches to this problem. Many researchers use a formal model such as an ontology [5] to describe the context [6,7]. An approach closer to our own is to use an unsupervised learning system to determine the context [8]. We take the view that no formal model of context is needed. Our system maps directly from sensory inputs to musical ratings. This approach is modeled on the work of Brooks [9].

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

1.2. Recommendation Systems There is a large body of work studying media recommendation systems. A good survey of available techniques is presented in [10]. There are several ways to recommend media to users including, collaborative filtering [11] and content filtering. Collaborative filtering systems match users with users who have similar tastes. They will recommend items to a user that a group of similar users appreciated. Last.FM is a collaborative filtering system where the users preferences are made public. Collaborative filtering can be seen in successful mainstream companies such as Amazon and Netflix. Content filtering uses features extracted from media to recommend media to a user. Those features can either be extraced by humans or computers [12]. In fields where computer annotation is difficult a common approach is to build a system that allows groups of people to annotate items the media library. One example of this type of system in the video domain can be found in [13]. Pandora [14] uses human experts to create a music content filtering system. Pandora is based on the music genome project where each song is manually annotated on 400 features. In [10] the authors conclude that future recommendation systems will be context aware. In the widely read report [15] the authors argue that the value of content is decreasing, however the value in the future will be aggregation, and providing contextually correct recommendations. The authors use Youtube as an example of a highly valuable media aggregator and recommender, where the content providers often get no compensation. The aggregation of videos on Youtube has value, the individual videos have little value. 1.3. Context Aware Media In recent years many researchers have studied the use of contextual information in the capture and delivery of media. For example in [16] the authors develop a framework for using context to improve media retrieval as an information retrieval task. In particular Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

S. Dornbush et al. / A Human Activity Aware Learning Mobile Music Player

109

they show improved recall on a library of images. Using context to better access media content has been discussed in [17,18]. Davis et al [19] demonstrate a system that annotates pictures with information about the context at the time of capture. The concept of a music player that is aware of the user’s activity has made it into the mainstream market with industry leaders Nike and Apple teaming up to deliver an iPod that wirelessly communicates with a sensor in a Nike running shoe [20]. That system has had limited market success so far, but has shown that there is consumer interest. The primary use case for this system is to record and analyze a runners athletic performance. That system is built to facilitate the user’s athletic training, not to allow the user to train the system. We uniquely address the problem of a context aware music player that learns a user’s preferences. Other researchers have studied the relationship between a user’s activity and the music selection played for them. In [21] the authors explore the idea of using Bayesian networks to predict the proper music for the current situation. The authors used a fairly small set of attributes to describe the context. The authors only used attributes from the large area context such as weather, user gender, time and season. A detailed explanation of multidimensional recommendation can be found in [22]. The authors present a strong theoretical case for multidimensional recommendation systems. This work focuses on OLAP query type functionality over machine learning techniques. The authors present a website that incorporates a concept of context, but do not show a mobile context aware recommendation system. Our initial system reported in [1] used this technique for music recommendation. In that system the array of user state information was converted to one of three states (active, passive and resting) using a decision tree. The state was used as a key into the database of state dependant ratings. It is our view that machine learning techniques are more appropriate for mobile devices where OLAP functionality may not be available. Personal Soundtrack [23] is a system that correlates the song played to a users pace. The users pace is determined by accelerometers that count the users steps per minute. This system used machine learning to determine which song to play, matching the song’s beats per minute (BPM) to the users pace. While this naive approach provides a level of context aware music it is not sufficient. First, this does not address the question of what music to play when a user is not active or when participating in an activity such as bicycling where the concept of pace is not well defined. Second, in our experiments BPM is not sufficient to characterize a piece of music to predict when it would be preferred. We discovered that our user found some pieces of music with high BPM suitable to low activity levels, and pieces of music with low BPM that were suitable for high activity levels. In [24] the authors describe a system of context aware media recommendations based on a combination of multidimensional representations, Naïve Bayes and a rule based system. This mixture of approaches was tested as a movie recommendation system. The authors demonstrated simple rules which would recommend the richest media item the mobile device could support. For example a video on a smart phone, and text with pictures on a less capable phone. This falls well short of a true context aware recommendation system.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

110

S. Dornbush et al. / A Human Activity Aware Learning Mobile Music Player

Figure 1. An author collecting training data. The author is running on a treadmill with a BodyMedia device. The device can be seen on his upper arm. The tablet PC can be seen on the treadmill stand.

2. Motivation The XPod concept is based on the idea of automating much of the interaction between the music player and its user. A large part of the interaction between users and music players involves choosing the right music to play. We seek to create a system that learns its user’s musical preferences for different activities, and tailors its music selections accordingly. The device is able to monitor a number of variables to determine its user’s levels of activity, motion, and physical states at the current moment and predict what music would be appropriate. The XPod user trains the player to predict the preference of music under various conditions. After an initial training period, XPod is able to use its internal algorithms to make an educated selection of the song that would best fit its user’s emotion state and level of activity. Before playing a song the internal algorithm is used to predict the user’s preference to that song based on their current emotional and physical state. That prediction is used to weigh the chance that the current song will be played. A song with a low expected

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

S. Dornbush et al. / A Human Activity Aware Learning Mobile Music Player

Genre

Artists

Tracks

Alternative Rock

Red Hot Chili Peppers

17

Blues

Louie Armstrong

20

Electronic

M.I.A.

12

Funk

Mofofunka

15

Hip-Hop

Black Eyed Peas Digable Planets Kanye West

14 11 20

Art Blakey

8

Jazz

John Coltrane Miles Davis

13 10

Reggae

Bob Marley

10

Beatles Phish Rolling Stones Grateful Dead

11 16 10 8

Jimi Hendrix Santana

6 9

Rock

111

Ska Tokyo Ska Paradise Orchestra 27 Table 1. A 2 GB music library with 239 songs used to evaluate the early versions of XPod.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

preference may be skipped in the current state. Every song has a chance of being played at any time. This is done so that the XPod explores the feature space and does not get stuck playing a few songs. If a song is skipped once, it should not be interpreted that the user never desires listening to that song. In a different state the user might prefer that song over all others. Our goal would be to model the level of knowledge for every song, then use that model to weight the trust we have in the rating.

3. Experiments 3.1. BodyMedia We developed several experimental systems for testing the XPod system. Our first system used a BodyMedia [25] arm band to measure the users physiological state. This system is shown in figure 1. The BodyMedia device measures a large number of physiological attributes and in real time broadcasts its readings wirelessly to a nearby tablet PC, which records the tracks played or skipped and associated BodyMedia data. The BodyMedia device is capable of monitoring a user’s physiological and emotional state [26]. In this paper we focus on the physiological state; however this system should be able to adapt musical preference to the emotional state. The list of variables monitored by the BodyMedia device is shown in figure 2. These measurements include skin temperature, heat flow, two dimensions of acceleration, cadence, and galvanic skin response, which is a measure of how much sweat is on the user’s skin. We processed the stream of values from the sensor to calculate a running average and variance. We also counted peaks in the acceleration to count steps so that we could calculate the users cadence. We presented these derived values to the machine learning algorithms. This device provided very ac-

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

112

S. Dornbush et al. / A Human Activity Aware Learning Mobile Music Player

Figure 2. The learning task using a BodyMedia device. The attributes at the top of the diagram are used to estimate the users current preference on a scale from 1-5.

curate measurement of the user’s state but was very bulky and difficult to use while exercising. The BodyMedia device can provide very accurate measurements due to the fact that it is worn directly against the skin. However, it is our view that many people would not be willing to always attach their music player directly to their arm. A device that mounts directly on the skin might be successful if it was marketed to just the athletic market. While this platform did have some form problems it was able to perform all of the learning techniques in realtime. In these experiments we used a 2 GB music library, the contents of which are summarized in table 1. We analyzed the 239 songs to find the beats per minute.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

3.2. Mobile Phone Handsets In subsequent work we have investigated using mobile phone handsets as the XPod platform. Dedicated mobile music players are quickly being replaced by music capable mobile phones. In technically advanced Asian countries such as South Korea and Japan, handsets are expected to contain many multimedia capabilities and many people use phones as music players. Gartner expects the number of music capable phones to exceed the number of dedicated music players in the near future. Many companies, including Nokia [27], Sony Ericsson and Apple [28], are releasing mobile phones with accelerometers which can provide a view of how the device is being moved. Nokia research has generously granted our research group several 5500 Sport phones [27], on which we are currently developing the next generation of XPod. The variables monitored by the Nokia Sport phone are illustrated in figure 3. We performed the same preprocessing on this sensor data stream as the previous experiments, where we calculated a moving average, variance and cadence. While these devices do not provide as much physiological data as the BodyMedia device, their small form factor makes it easier to use the system. So with these devices we do not collect as much information at every timestep, but we can build much larger datasets because it is easier to run experiments. The limited computation and storage capabilities of these phones lend themselves to a dumb terminal architecture where the server does most of the work and the phone is principally responsible only for providing a user interface. The architecture that we use is

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

S. Dornbush et al. / A Human Activity Aware Learning Mobile Music Player

113

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 3. The classifier task when using a motion aware phone. The classifier learns to whether a user is likely to skip or listen to a given song based on the sensed location and activity.

illustrated in figure 4. We use the devices to collect information about how the user listens to music. The device then sends log files of music listening behavior to a central server. This model is already well established for the social music website Last.FM where users voluntarily upload their music listening, so others can see what they are listening too. This model of uploading context information from phones was also demonstrated in the Reality Mining project [29] were mobile phones where used to study real world social networks. Once the music listening patterns are transferred and archived on the server, the more powerful server builds a model of the users listening patterns. The model is then transferred back to the mobile handset. From that point on the the handset can use the model of the user’s listening pattern to choose music the user is likely to enjoy for their current context. This arrangement means that the mobile phone only has to store a short reccord of a user’s recently played tracks, and a fairly compact model of the user’s listening preferences. The server is tasked with the more difficult jobs of keeping an archive of all tracks played or skipped by the user, and building new models whenever new data is reported. In these experiments we sampled from a music library of more than 55 GB of music from which we drew 2 GB samples.

4. XPod Dataset The XPod system consists of a standard MP3 playing device and a human body sensor. The device tracks and stores a record of each song’s meta-data as that song is played, including artist, album, genre, title, and beats-per-minute. In addition, the system records the time of day, a user’s rating (from 0 to 4 stars), and a full range of physical responses from the user’s body. The set of attributes recorded are different in the two different hardware platforms (refer to the previous section for a list of attributes). Some of the numerical algorithms, such as Support Vector Machines(SVMs) and neural networks, cannot handle symbolic values. For these algorithms each of the symbolic attributes,

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

114

S. Dornbush et al. / A Human Activity Aware Learning Mobile Music Player

Figure 4. The dumb terminal architecture of the learning system using a mobile handset and servers. The phone collects a log of a users listened and skipped songs and the associated context information. That is transferred to a server who performs machine learning on a large database of that users past listening behavior.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

artist, album, title, and genre, are all expanded into a large number of binary attributes. The results listed in this paper were calculated from the BodyMedia system, using a 2 GB library of music. In our experiments the total number of attributes, including the user state, is 289. Typically each instance is a sparse array with most attributes set to 0 or false. In the latter experiments over the larger 55 GB music library we have thousands of attributes.

5. Machine Learning Algorithms To design the best context aware music recommendation system, we trained several independent learning algorithms on our test data. A researcher collected training information using a prototype system. The prototype shown in Figure 1 involved a tablet computer and a BodyMedia device. A researcher on the XPod team proceeded to record training instances in a variety of physical situations (exercise, mild activity, rest, etc.). A training instance, or data point, includes a value for each field in figure 2. A total of 565 training instances were recorded. For each instance the XPod music player would rate a song and play that song. If the rating matched the researcher’s preference he took no action. If the rating did not match the preference the researcher gave the song a rating from zero to four, reflecting how appropriate the researcher felt the song was at that time. If the user skipped a song it was given a rating of zero. Each classification algorithm was trained on some or all of the training instances. That training was used to predict how a user would rate a song in the future. It is our goal to show that a music player can make better music recommendations if it uses information about the user’s physical context. To prove this theory we created

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

S. Dornbush et al. / A Human Activity Aware Learning Mobile Music Player

115

(a) % Accuracy of various learning algorithms.

(b) RMSE of various learning algorithms.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 5. Performance of learning algorithms

two sets of machine learning systems, those trained with user state information, and those without user state information. “State” refers to the array of information gained from the BodyMedia device, as well as any other outside information, such as date and time. Due to our small sample size, activity can be inferred from the date and time. For this reason, those variables were included in state information. We will consider our experiment successful if the system is more accurate when it has access to the state information than when it does not have access to state information. We used 10 fold cross-validation to measure the accuracy of the machine learning algorithms. We also experimented with Leave-One-Out-Cross-Validation(LOOCV). We found very similar results between the two methods. Since LOOCV is much more computationally expensive we have reported the results of 10 fold cross-validation. We used classifiers from the open source Weka library [30] and neural networks from the open source Joone library [31]. In the following experiments each training instance could be classified into one of five classes. The expected performance of a random algorithm would be 20%. All of the algorithms performed significantly better than random. 5.1. Decision Trees The first classifier used was the decision tree algorithm (J48)[32]. When learning without state, the decision tree was able to properly classify the training data 39.47% of the time. However, when using state, the decision tree was able to properly classify the training data 41.06% of the time. The accuracy of decision trees was not the best in the survey, however they do show a slight (2%) improvement in percent accurate by using state information. The poor generalization of J48 can be seen in the counter intuitive fact that, unlike the other algorithms J48 trained without state information had a smaller RMSE

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

116

S. Dornbush et al. / A Human Activity Aware Learning Mobile Music Player

than J48 that did use state information. This can be described as a performance inversion where the RMSE is better when state information is withheld. 5.2. AdaBoost The performance of the J48 classifier improved when it was boosted with AdaBoost (AdaBoostM1)[33]. This classifier was correct against the training data 39.47% without state, and 46.55% with state. AdaBoost suffered from the same generalization performance inversion as the original J48 classifier. This showed that AdaBoost is somewhat effective at increasing the performance of the J48 algorithm. 5.3. Support Vector Machine (SVM) The third classifier that we experimented with was support vector machines (SMO)[34, 35]. SVM’s generalized well and had a little improvement when using state (43.19%) over not using state (40.89%). In this case the SVM was almost able to divide the dataset into the researcher’s preference based solely on the musical data. When adding in state, the dimension space changed minimally, adjusting enough to shuffle a few incorrectly classified instances to the proper area. The small difference between the SVM trained with state information and without state information (2%) is likely a result of the relatively large feature space and small training set. The expressiveness of the second order polynomial kernel allows the SVM to identify the user’s preference without the state information.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

5.4. K-Nearest Neighbors(KNN) We had surprisingly positive results from the lazy classifier: k-nearest neighbors (IBK)[36]. We allowed Weka to choose the optimal number of neighbors. The best number of neighbors was found to be 9. Results showed a 7% increase in accuracy when using state (46.72%) over not using state (39.82%). More importantly KNN had a low root mean squared error (RMSE) (0.3753). 5.5. Neural Networks We had very promising results from a neural network trained on this data-set. We created a three layer network with 288, or 276 inputs depending on whether state information was used. A small hidden layer and a single neuron output was used. We experimented with a variety of different size hidden layers from 1 to 50. The results of these experiments are shown in Figures 6(a) and 6(b) We found very similar results with a small number of hidden nodes, 3, as compared to a large number of hidden nodes, 50. As the size of the hidden layers grows, the accuracy of the network using state information does not increase much. However the accuracy of the network not using state information does increase. We believe that the more complex networks are better at memorizing noise in the dataset. We had difficulties with over training. Neural networks which over train will memorize noise in the dataset. This is undesirable as the generalization performace of the network will decrease in an over trained network. The network would find the best val-

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

S. Dornbush et al. / A Human Activity Aware Learning Mobile Music Player

117

(a) % Accurately identified using different size hidden layers.

(b) RMSE using different size hidden layers.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 6. Performance of different size networks.

idation error in the first 100 training epochs. We used early stopping to keep the best network on the validation data. We are investigating ways to avoid this problem. We were able to achieve respectable performance with a network given state information. That network classified instances with 43.54% accuracy much better than the 31.87% accuracy without state information. This is not the best percent accuracy, however it did get the best results in terms of RMSE (0.17). The neural network had a fraction of the RMSE of the other methods.

6. Incorporating Collaborative Filtering and User Tags After we demonstrated that a music player can choose the contextually correct music if it uses information about a user’s physiological state we decided to try to improve the accuracy of the music selections. We incorporated information from traditional collaborative filters for slightly better results. Figure 7 lists the fields that we used in this set of experiments. In Table 2, ID3 tags refer to the standard tags embedded in MP3 files. Tag refers to the user generated tags from the Last.FM website. Users of this website can tag music with arbitrary strings to help organize their music library. Similarity refers to the similarity of two artist using the Last.FM webservice. Last.FM uses association rule mining [37] to calculate the similarity of two artists. The similarity of artist α to β is the percent of people who have artist α in their library who also have β in their library.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

118

S. Dornbush et al. / A Human Activity Aware Learning Mobile Music Player

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Figure 7. The machine learning task using tags and collaborative filtering from Last.FM

Meta-Data

RMSE

% Correct

ID3

0.1878

33.29%

Tag

0.1900

34.39%

Similarity

0.1869

33.10%

Tags/Similarity

0.1889

33.56%

State

0.1714

47.20%

State/Tag

0.1708

48.03%

State/Similarity

0.1645

49.63%

State/Tags/Similarity 0.1679 49.03% Table 2. Performance of the different networks.

Association rule mining is a common technique used in collaborative filtering systems for everything from books to beer. Our goal in this work was to show an improvement in prediction accuracy using information obtained from Last.FM. We only tested neural networks on this dataset as they showed the best performance in the earlier experiments. To incorporate collaborative filtering we added a new attribute for every artist in the library. We then populate those arrays using the similarity of the current artist to all other artists in the collection. This helped the neural network generalize about music. Two training instances with two artists that are very similar to one another will appear similar to one another to the neural network. We also added an attribute for every tag that was assigned to the tracks in the dataset. The value of each of those attributes was given by the frequency that the given tag was assigned to the song in question by Last.FM users. Due to the unstructured nature of the tags we had to create a large number of attributes to represent all of the tags in our small library.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

S. Dornbush et al. / A Human Activity Aware Learning Mobile Music Player

119

(a) % Accurately identified using different meta-data.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

(b) RMSE using different meta-data. Figure 8. Performance of the neural network when it was supplied with various song meta-data. There is a small but clear improvement with social information on songs.

7. Conclusion and Future Work Our goal is to show that a music player trained with a user’s physical activity and preference can choose the contextually correct music. All of the systems evaluated performed significantly better than random (20% accuracy). As shown in Figure 5(a), given state information every system chooses the exactly correct label more often than the same algorithm without state information. For each testing instance there are five possible categories. Figure 5(b) shows that tree based algorithms tend to generalize poorly, evidenced by the fact that systems using state information had greater RMSE than the systems not using state information. These algorithms had comperable accuracy, however the mistakes in estimation tended to be larger. The other algorithms were able to generalize well and achieved high accuracy and low RMSE. We believe that if we collected still more training instances the difference between the performance of the statefull and the stateless system would grow. Presumably if we collected enough training instances we would find a pair of instances that are identical

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

120

S. Dornbush et al. / A Human Activity Aware Learning Mobile Music Player

on all non-state attributes but have different ratings. Then any classifier without state information would have to give both instances the same label. However only one could possibly be correct. A classifier given the same instance including the state information has a chance to classify both instances correctly. Therefore if we collected more training instances the difference between stateful and stateless systems should increase. Although the lazy classifiers IBK tended to perform well. We believe that these results will not scale to larger data sets, or perform satisfactory on a portable music device that is not likely to have much processing power. Given an active user of such a device, listening for multiple hours a day over the course of one or two years, the device would search an instance space of tens of thousands of data points. Performing a calculation like this may be more than inefficient, it could be wholly impractical. Support vector machines may be well suited to the task as they can begin to classify new instances having very little training data to build on. From the end-user’s perspective, this is a desirable feature, as the user would need to spend very little time setting up the system, and more time enjoying the benefits. Further, SVMs are capable of classifying in a very high dimensional space while only performing calculations in a much smaller number of dimensions. Decision trees would likely be the most computational feasible classifier, as they can be converted into a rule set, which can be evaluated very rapidly. As we’ve shown in this application, decision trees perform better with boosting. Our view is that the neural network is the most promising result. Although it did not get the highest exact accuracy it tended to get very close to the right answer, reflected in the small RMSE. Since the result was used to influence pseudo-random choice of music it is actually more important to be close than to be exactly accurate. For example, if the correct rating should be 4 but the system responds with 3, that will result in a low RMSE, but will not count towards the percent correct. A rating of 0 would result in the same percent accurate, however a much higher RMSE. Since a song rated 0 would be skipped, but a song rated three or four would likely be played, a close answer is almost as good as the correct answer. For that reason we feel that the low RMSE of the neural network indicates that it would be the most useful algorithm. Many embedded devices such as mobile phones already employ neural networks, therefore it should be possible to use neural networks in mobile music playing devices. We have used relatively simple music analyzing software to find the beats per minute, however it is possible to find much more by analyzing the music [38]. We have investigated using additional meta-data beyond the standard ID3 tags associated with music files. It has been shown that expert humans can produce very useful annotations to music such as the Pandora Project [14]. We found that the amateur tagging found on Last.FM was not of much use to us. However the collaborative filtering was usefull, providing a small improvement in performance. There is a significant cost to learning over the larger input set created using collaborative filtering. We will investigate use principle component analysis [39] to reduce the total number of attributes generate from collaborative filtering. We will continue to look for new meta-data regarding songs or the user’s context that could be usefull as additional inputs into the machine learning algorithms. Our goal will be to see if there is a significant increase in performance given new information. While the BodyMedia device provides many different attributes a satisfactory system could likely be built with a selection of those attributes. An inspection of the deci-

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

S. Dornbush et al. / A Human Activity Aware Learning Mobile Music Player

121

sion tree built by J48 shows 20 decisions based on acceleration, almost four times more than the sum of all decisions based on other state variables. Since we can derive this information from a phone, which is much easier to wear we are experimenting with motion sensing phones. We have shown the relative advantages of different machine learning systems at choosing the contextually correct music. Many people have shown an interest in this type of system. We will continue or work to further refine this system. Our goal is to release a version of this system to the public for further experimentation.

Acknowledge We would like to thank Nokia Research for the donation of the 5500 Sport phones.

References [1]

[2]

[3] [4] [5] [6]

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

[7]

[8] [9] [10] [11] [12]

[13]

[14] [15] [16]

Sandor Dornbush, Kevin Fisher, Kyle McKay, Alex Prikhodko, and Zary Segall. XPod a human activity and emotion aware mobile music player. In Proceedings of the International Conference on Mobile Technology, Applications and Systems, November 2005. Sandor Dornbush, Jesse English, Tim Oates, Zary Segall, and Anupam Joshi. XPod: A Human Activity Aware Learning Mobile Music Player. In Proceedings of the Workshop on Ambient Intelligence, 20th International Joint Conference on Artificial Intelligence (IJCAI-2007), January 2007. M. Bylund and Z. Segall. Towards seamless mobility with personal servers. INFO Journal, May 2004. Daniel Siewiorek, Asim Smailagic, Junichi Furukawa, Neema Moraveji, Kathryn Reiger, and Jeremy Shaffer. Sensay: A context-aware mobile phone. In ISWC, 2003. S Staab. Handbook on Ontologies. Springer, 2004. Harry Chen, Tim Finin, and Anupam Joshi. An Ontology for Context-Aware Pervasive Computing Environments. Special Issue on Ontologies for Distributed Systems, Knowledge Engineering Review, 18(3):197–207, May 2004. X Wang, D. Q. Zhang, T Gu, and H.K. Pung. Ontology-based context modeling and reasoning using owl. In Workshop on Context Modeling and Reasoning at IEEE International Conference on Pervasive Computing and Communication, Orlando, Florida, USA, 2004. A. Flanagan. An unsupervised learning paradigm for peer-to-peer labeling and naming of locations and ˘ S221, contexts. In LoCA, page 204âA ¸ Dublin, Ireland, 2006. Rodney A. Brooks. Intelligence without representation. Number 47 in Artificial Intelligence, pages 139–159. 1991. G. Adomavicius and A. Tuzhilin. Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. 17(6):734– 749, JUNE 2005. C. Hayes, P. Massa, P. Avesani, and P. Cunningham. An on-line evaluation framework for recommender systems. Workshop on Personalization and Recommendation in E-Commerce, 2002. J. Kuper, H. Saggion, H. Cunningham, T. Declerck, F. de Jong, D. Reidsma, Y. Wilks, and P. Wittenburg. Intelligent multimedia indexing and retrieval through multi-source information extraction and merging. In INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE, volume 18, pages 409–414. LAWRENCE ERLBAUM ASSOCIATES LTD, 2003. Arnon Amir, Marco Berg, Shih-Fu Chang, Winston Hsu, Giridharan Iyengar, CY Lin, BL Tseng, JR Smith, Milind Naphade, Apostol Natsev, Chalapathy Neti, Harriet Nock, John R. Smith, Yi Wu, and Donqing Zhang. Ibm research trecvid-2003 video retrieval system. TRECVID 2003 Workshop Notebook Papers, November 2003. Tim Westergren. Music genome project, 2006. http://www.pandora.com/. Spencer Wang. The long tail: Why aggregation and context and not (necessarily) content are king in entertainment. Bear Stearns Media Research Presents, 2006. Ankur Mani1 and Hari Sundaram. Modeling user context with applications to media retrieval. Multimedia Systems, 12:339–353, March 2007.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

122

S. Dornbush et al. / A Human Activity Aware Learning Mobile Music Player

[17] Pertti Huuskonen. Context and content: A perfect marriage. In J Augusto, editor, Proceedings of Artificial Intelligence Techniques for Ambient Intelligence, page 5, 2006. [18] Nati Herrasti, Antonio López, and Alfonso Gárate. A human home interaction application. In J Augusto, editor, Proceedings of Artificial Intelligence Techniques for Ambient Intelligence, pages 22–26, 2006. [19] Marc Davis, Nathan Good, and Risto Sarvas. From context to content: Leveraging context for mobile media metadata. In MobiSys Workshop on Context Awareness, 2004. [20] Nike and Apple. http://www.apple.com/ipod/nike/, 2006. [21] H. S. Park, J.O. Yoo, and S. B. Cho. A context-aware music recommendation system using fuzzy bayesian networks with utility theory. In International Conference on Fuzzy Sytems and Knowledge Discovery (FSKD’06), pages 970–979, 2006. [22] Gediminas Adomavicius, Ramesh Sankaranarayanan, Shahana Sen, and Alexander Tuzhilin. Incorporating contextual information in recommender systems using a multidimensional approach. ACM Trans. Inf. Syst., 23(1):103–145, 2005. [23] Greg T. Elliott and Bill Tomlinson. Personalsoundtrack: context-aware playlists that adapt to user pace. In CHI ’06: CHI ’06 extended abstracts on Human factors in computing systems, pages 736–741, New York, NY, USA, 2006. ACM Press. [24] Daqing Zhang and Zhiwen Yu. Vehicular technology conference, 2007. vtc2007-spring. ieee 65th. pages 267–271, Dublin, Ireland, April 2007. [25] BodyMedia. http://www.bodymedia.com/, 2006. [26] Fatma Nasoz, Kaye Alvarez, Christine L. Lisetti, and Neal Finkelstein. Emotion recognition from physiological signals for presence technologies. International Journal of Cognition, Technology and Work Special Issue on Presence, 6, 2003. [27] Nokia. 5500 sport phone, 2006. [28] Apple. http://www.apple.com/iphone/technology/sensors.html, 2007. [29] Nathan Eagle and Alex (Sandy) Pentland. Reality mining: sensing complex social systems. Personal Ubiquitous Comput., 10(4):255–268, 2006. [30] Ian H. Witten and Eibe Frank. Data Mining: Practical machine learning tools and techniques, volume 2nd Edition. Morgan Kaufmann, San Francisco, 2005. [31] Paolo Marrone and Joone Team. Joone, 2006. http://www.jooneworld.com/. [32] Ross Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers, San Mateo, CA, 1993. [33] Yoav Freund and Robert E. Schapire. Experiments with a new boosting algorithm. pages 148–156, San Francisco, 1996. Morgan Kaufmann. [34] J. Platt. Fast Training of Support Vector Machines using Sequential Minimal Optimization. Advances in Kernel Methods - Support Vector Learning. MIT Press, 1998. [35] S.S. Keerthi, S.K. Shevade, C. Bhattacharyya, and K.R.K. Murthy. Improvements to platt’s smo algorithm for svm classifier design. Neural Computation, pages 637–649, 2001. [36] D. Aha and D. Kibler. Instance-based learning algorithms. Machine Learning, 6:37–66, 1991. [37] Rakesh Agrawal, Tomasz Imielinski, and Arun Swami. Mining association rules between sets of items in large databases. SIGMOD Rec., 22(2):207–216, 1993. [38] B. Logan and A. Salomon. A music similarity function based on signal analysis. In IEEE International Conference on Multimedia and Expo, ICME, 2001. [39] I. T. Jolliffe. Principal component analysis. Springer Series in Statistics, Berlin: Springer, 1986, 1986.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

Advances in Ambient Intelligence J.C. Augusto and D. Shapiro (Eds.) IOS Press, 2007 © 2007 The authors and IOS Press. All rights reserved.

123

Two Issues for an Ambient Reminding System: Context-Awareness and User Feedback Nadine RICHARD 1 and Seiji YAMADA National Institute of Informatics, Japan Abstract. Busy people would benefit from ambient, adaptive tools for reminding them what they have to do, depending on various contextual parameters. We are developing an adaptive and expressive agent, which learns when and how to notify users about self-assigned tasks and events. In this paper, we focus on two crucial issues for such a system: the selection of the input data set, and the design of an appropriate mechanism to get user feedback without being too intrusive. We describe in particular the inputs that encapsulate the current context of the agent: relative temporal distances, historical information about reminders and categories, and the context of both the user and the hosting device. Keywords. Personal time management, Ambient Intelligence, interface agent, context-awareness, user feedback.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Introduction Intelligent personal assistants help reducing the ever-growing load of information, events and various commitments people need to handle, for instance by learning how to organize and keep track of relevant items. We are developing an adaptive and expressive assistant for reminding self-assigned tasks and events. Our interface agent, called TAMAC OACH, adapts to the organizational skills and preferences of a user, by learning when and how to present notifications, instead of requiring the user to set explicit alarms. Figure 1 presents a typical view of the TAMAC OACH GUI. When designing an interactive learning system, two major issues are: • Selecting an appropriate set of input values; in the case of an ambient system, this should include information about the current context of the application. • Choosing a feedback mechanism, to let the user reward or punish the adaptive components after they respectively lead to correct or incorrect actions. As a result of our preliminary study of available tools and data, this paper summarizes the requirements for our adaptive reminding system, in terms of inputs, outputs and user feedback. Because users might not interact very often with their calendar and todo managers, the design of the feedback mechanism is particularly important. 1 Corresponding author, [email protected]. Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

124

N. Richard and S. Yamada / Two Issues for an Ambient Reminding System

Figure 1. The main window of the TamaCoach GUI.

Overview of TamaCoach

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Within TAMAC OACH, the triggering of a notification basically depends on the following data, which are extracted or abstracted from I C ALENDAR files: • The relative temporal distance to the event starting date or to the task due date. • Various descriptive attributes that help to classify user habits and preferences, e.g. contact information, priority or categories. A reminder can be presented under combinable forms: a non-modal pop-up dialogue, an e-mail message, a mobile e-mail message, or a sound. Figure 2 shows an example of pop-up task reminder. The selected forms of reminder mostly depend on the current context: • The user status, gathered through the agent GUI: availability, kind of activity, global location, and mood. • The execution environment of the agent, collected from the operating system of the hosting device: device name, connectivity, and capability to play a sound. Whatever forms are selected by TAMAC OACH, the new reminder is also appended to the appropriate list of pending events or pending tasks. Those lists are displayed in the TAMAC OACH GUI, as presented in figure 3; selecting an item in a list will make the corresponding pop-up reminder appear. Presently, only a few reminder forms are available, but other kinds of notifications could be added after the learning system has proven to be effective.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

N. Richard and S. Yamada / Two Issues for an Ambient Reminding System

125

Figure 2. A task reminder with three predefined replies and the update button.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

TAMAC OACH interacts with its user both explicitly and implicitly, respectively through the available user interfaces (GUI and e-mails), and through the successive modifications of the calendar files. The I C ALENDAR data are manipulated externally, from the user’s favourite calendaring application. As shown in figure 2, notifications are accompanied by a set of user replies (e.g. accept or later), which are necessary to get explicit feedback about the usefulness of the system. We are currently evaluating the first prototype of our adaptive agent, on three different platforms: a desktop PC under GNU/L INUX, a laptop under MS-W INDOWS, and a Z AURUS PDA under a native distribution of GNU/L INUX. In parallel, we are investigating how to simulate various user profiles for testing and tuning the learning system. Organization of the Paper In order to situate the purpose and the specificities of our reminding system, we first present some related work on personal assistants. Then, in section 2, we describe what we believe should constitute the inputs of an adaptive, context-aware reminding system; we especially detail relative temporal distances, historical data about categories and pre-

Figure 3. The list of events within the main panel. Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

126

N. Richard and S. Yamada / Two Issues for an Ambient Reminding System

vious reminders, and both user and device contexts. Section 3 outlines the learning system of TAMAC OACH, composed of two cascaded XCS-based modules. In section 4, we introduce the light-weight mechanism we propose to get meaningful feedback from the user. Before concluding, we discuss the limitations of the available input data, and we propose interaction strategies to incite the user to provide further information and feedback. In particular, we present our hypothesis about the role of an expressive agent in the case of an adaptive reminding system.

1. Assistants for Personal Time Management The management of personal calendars and todo-lists is difficult, in both personal and professional situations: it requires remembering on-going activities, and continuously rearranging priorities. People tend to remember quite well the main tasks and events they are involved in, but they now have to face a growing volume and variety of information and commitments, as well as more and more sources of interruptions. In the following section, we enumerate the main issues to be investigated when developing adaptive assistants, in the particular context of personal time management. After presenting noteworthy related work, we introduce the specificities of our own project.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

1.1. Main Issues As pointed out by P. Berry et al. [1], human time management has an intensely personal nature. People are usually reluctant to delegate this specific task to others, and even more to software. They also have different preferences and practices regarding how they schedule their time. Moreover, people tend to use various media, from paper post-its to mobile e-mails, to keep track of things they intend to do, and often do not record all the tasks and events they should remember [2]. When dealing with time management, categories of issues concerning personal assistants include machine-learning, interaction modalities, context-awareness, and interruption strategies. In particular, S. Shiaffino and A. Amandi state, in two interesting surveys [3,4], the main issues for building a virtual secretary: the need for personalization, as each user would like to interact with a different kind of assistant, and the reaction of users towards interruptions, errors and explicit requests for feedback. Interruption strategies are essential, as the task performance of a human may dramatically decrease when carelessly disrupted [4,5,6,7,8]. Therefore, a good personal time manager should keep track of various sources of information and events through different time scales, but should also be considerate: it should be aware of the current context of its user in terms of activity, availability, location or even emotional state, in order to notify her appropriately. 1.2. Non-Adaptive Tools Calendars and todo-lists are used to collect, maintain and organize self-assigned tasks and events, with different natures, durations, regularities and frequencies. Electronic calendars and todo managers are different but closely related applications, often part of PIM (Personal Information Management) software suites. Within such tools, users can set alarms for the most important events, such as meetings or deadlines.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

N. Richard and S. Yamada / Two Issues for an Ambient Reminding System

127

Many desktop or PDA (Personal Digital Assistant) applications, like S UNBIRD or E VOLUTION, propose traditional ways of organizing todo-lists and calendars, but do not take into account the multiple sources of task-oriented information. To overcome this lack, V. Belloti et al. have developed TASK V ISTA [2] and TASK M ASTER [9]. TASK V ISTA is a light-weight task list manager, which reduces the cognitive overload of information and events coming from different sources (e.g. e-mails, todo manager and Web browsing). TASK M ASTER enables users to keep track of threads of activity and discussions, manage deadlines and reminders, and mark-up tasks directly within a single email application. However, those applications do not automatically adjust to the specific organizational habits of their users. 1.3. Intelligent Assistants

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Most of the recent research projects on personal assistants focus on information recommenders, health advisors and virtual secretaries. Such assistants are generally cognitive agents, which learn and reason about tasks, user behaviour and their own behaviour, in order to justify their actions, answer questions and give advice. For example, as part of CALO (Cognitive Assistant that Learns and Organizes), the PTIME component learns about user scheduling habits and preferences, mostly in order to autonomously negotiate meetings among potential participants [1,10,11]. TASK P REDICTOR [12] detects the current task of the user, among pre-labeled activities, according to the resources that are being accessed (documents, folders, address book, etc.). This adaptive system reduces the cost of explicit task switching when using the TASK T RACER assistant. The B ESTCOM project [8] investigates various systems that learn about the identity and the goals of the contactor, and about the communication preferences and the current and predicted availability of the contactee. For instance, one of their prototypes performs smart routing of voice-calls. 1.4. Adaptive Reminding Systems AUTOMINDER is a reminding system intended for cognitively impaired elders, who prefer to live at home [13]. It learns about routine activities, and monitors their daily performance in order to issue reminders whenever an essential task is not executed on time. The system maintains a set of soft constraints representing the expected user’s agenda for the day, and compares it to the actual activities to detect relevant discrepancies. Shankar et al. have been experimenting on a context-aware reminding system, which can choose among four actions: a pop-up window, a voice reminder, both, or none [14]. The authors first gathered data about when and how a subject preferred to be interrupted by a reminding system when performing a task. After an off-line mining of those data, their S YCOPHANT system was able to predict which reminder would be suitable for an appointment, given the current user context (keyboard/mouse activity and state of selected processes) and user’s vicinity context (motion and speech detection). This system focuses on the automatic detection of the level of interruptibility of the user.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

128

N. Richard and S. Yamada / Two Issues for an Ambient Reminding System

Figure 4. The overall architecture of TamaCoach.

1.5. TamaCoach The goal of TAMAC OACH is not to help people in maintaining todo-lists or to automatically detect the user-context, but to learn when and how to remind users about what they have explicitly planned to do. Our system decides when to interrupt the user, depending on a categorization of tasks and events, and what notification form is appropriate, depending on the given context (user status and device capabilities). However, such a tool is a complement to more complex assistants like CALO, TASK V ISTA or TASK M ASTER, and could be extended with context detection tools like S YCOPHANT or TASK P REDIC TOR.

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

2. Significant Input Data In order to trigger a reminder at the right time and to choose the best way to present this reminder to the user, TAMAC OACH has to process the available information about the task or the event, and be aware of the current context of both the user and the host device. Some other values can be abstracted or computed from I C ALENDAR data, and from past observations. Figure 4 presents an overview of our architecture, while a summary of the 28 values processed by our learning system is given in table 1. 2.1. Available Calendar and Todo Data I C ALENDAR

files may contain the following information about each task or event:

• Absolute dates, including time information. In particular: creation date, starting date, event ending date, and task due date. • Durations. • Recurrence rules. • User-defined categories. • Explicit alarms.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

N. Richard and S. Yamada / Two Issues for an Ambient Reminding System

129

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

Table 1. This table summarizes the names and value domains of the 28 attributes provided as inputs to the learning system of TamaCoach. Domains that can be extended by the user are differentiated with an ellipsis. Names in bold correspond to attributes whose value is always available. Name

Domain of values

is_event?

true, false

has_alarm?

true, false

is_exception? is_transparent?

true, false true, false

calendar

personal, work, national holidays...

duration priority

very short, short, medium, long low, medium, high

access progress_status

public, private, confidential cancelled, in progress 0-49%, in progress 50-99%, completed

location

office, home, room B4...

category contact

meeting, business trip, birthday, write report... contact1, contact2...

task_load

low, medium, high

opaque_load

low, medium, high

temporal_granularity temporal_distance

very short-term, short-term, mid-term, long-term late, very close, close, quite close, quite far, far, very far

category_duration

very short, short, medium, long

category_delay

very short, short, medium, long

category_is_early?

true, false

nb_reminders last_reminder

0, 1, 2, 3, less than 6, less than 10, more than 15 very close, close, far, very far

user_busy_state

available, busy, very busy, away, disconnected

user_mood user_activity

very good, good, average, bad, very bad work, leisure, vacation, commuting, sick, conference, meeting...

user_location

office, transportation, home, business trip...

device

desktop, laptop, pda...

has_network? has_sound?

true, false true, false

• The progress status of a task. • Information about involved contacts (name, role, e-mail, etc.). • Various other characteristics: summary (title), long description, priority, transparency, location, etc. As addressed later in section 5, we cannot expect the user to give all the useful information for an ideal learning. In particular, she might not provide relevant categories to facilitate the automatic classification. However, the following data is always available: • The name of the calendar file. If the user keeps hold of more than one calendar (e.g. professional, personal and national holidays), the calendar name can be a hint for categorizing tasks and events. • The kind of item, i.e. event or task. • The summary of the item. • The creation date of the item.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

130

N. Richard and S. Yamada / Two Issues for an Ambient Reminding System

Among those fields, only the calendar filename and the kind of item are directly used as inputs for the learning system of TAMAC OACH. Depending on the kind of item, the following information is also always available: • The starting date of an event, and its ending date or its duration2. • The deadline of a task. If not given, we assume that the user has defined a background task. From those mandatory I C ALENDAR dates, we can compute brute durations if they have not been specified by the user, and translate them into symbolic values in order to narrow down the learning space: • The event duration is calculated from the starting and ending dates. • The task duration is calculated from the starting and due dates. If the starting date is not available, we replace it by the creation date, which means that the user is supposed to start working on the task as soon as it has been recorded. If the deadline is not specified, the duration is set to the maximum value possible. In addition, default values can be assigned to some attributes, if they are not provided in the I C ALENDAR files:

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

• The user-defined priority, which states the subjective importance of the item, and which is set to medium by default. • The transparency, which specifies whether an event is time-consuming (opaque). By default, events are transparent. • The presence of at least one alarm explicitly set by the user for an event. • The fact that the item corresponds to an exception date within a recurrence rule (cf. section 2.2). Because they concern only specific occurrences, absolute dates are not relevant for the learning process: more abstract data is required to enable the system to learn a generalization of the best time to notify the user about a certain type of task or event. We thus compute the relative temporal distance, which is the duration between the current date and the starting date of an event or the due date of a task (cf. section 2.3 for more details). To be relevant for the learning mechanism of our agent, the computed distance is translated into a symbolic value expressing approximately how close the current date is to the deadline or the starting date. With such a decomposition, a calendar item is easily categorized as happening soon or in a long time. Furthermore, we can always compute two significant values, which indicate the activity load of the user for the computed relative temporal distance: • The average task load is computed from the number and the priority of tasks that overlap with the given period. • The average load of opaque events is computed from the brute duration and the priority of time-consuming events that overlap with the given period. For all the attributes described above, values can always be provided as inputs to the learning system; in table 1, their names appear in bold characters. When available, the following categorical information is also extracted from the raw I C ALENDAR data: 2 In most calendaring applications, if the duration of an event is not specified, then the ending date is mandatory, and vice-versa.

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

N. Richard and S. Yamada / Two Issues for an Ambient Reminding System

131

• User-defined categories, i.e. character strings that shortly describe the kind of task or event. • Involved contacts. The gathered information about involved contacts are stored in the database, so that the contact attribute corresponds to a single person in the user’s address book. • The progress status of the task, given as a percentage and translated into a more general symbolic value. • The level of confidentiality, i.e. if the item is accessible by other users. • The location of an event. All the values presented in table 1 are stored in the agent database and used as inputs. Some other I C ALENDAR fields, like the summary, the long description or the links to attached documents, are not significant for the learning process, unless they are analysed to infer higher-level information about dependencies or implicit categories. For the moment, their values are only stored in the database to be presented within the reminders. 2.2. Recurrence Rules

Copyright © 2007. IOS Press, Incorporated. All rights reserved.

For a few regular events like weekly meetings or birthdays, users often find it worth to provide a recurrence rule instead of repeating the edition of each occurrence. Within TAMAC OACH, recurrence rules are first translated into distinct occurrences, with absolute dates and durations, in order to be processed as separate items with different temporal distances. As stated in the previous subsection, we added an attribute to specify whether the processed occurrence corresponds to an exception date of a recurrence rule. We believe this information to be relevant, especially when the recurrence rule describes a frequent event: it seems more important to remind the user about the exceptional absence of a recurrent event than about its usual presence. 2.3. Computing Relative Temporal Distances Relative temporal distances are computed from the available dates and durations of occurrences, including the occurrences extracted from recurrence rules. The difficulty of calculating temporal distances in our application comes from the heterogeneity of the dates and times involved, as well as the various meanings of the urgency of a reminder. The temporal information to be manipulated by TAMAC OACH is by nature multigranular [15,16]. For instance, an event that starts in five minutes is obviously close to happen, but a task that is expected to take 8 weeks to be achieved can also be considered as close in a weeks time scale if the current date is less than 9 weeks before the deadline. Those two simple examples show that the urgency of a notification does not only relate to the immediate proximity to a given date. Keeping track of the granularity is thus necessary, in order to consider relative symbolic values like very close or far in an appropriate time scale. For clarity and efficiency purposes, we chose a set of significant thresholds in order to categorize both the granularity and the approximate value of a distance, instead of proposing functions for computing this categorization. Granularities are represented by durations, corresponding to different time scales: very short-term (0-24 hours), short-

Advances in Ambient Intelligence, edited by J. C. Augusto, and D. Shapiro, IOS Press, Incorporated, 2007. ProQuest Ebook Central,

132

N. Richard and S. Yamada / Two Issues for an Ambient Reminding System

term (1-10 days), mid-term (10-42 days) and long-term (more than 6 weeks). The temporal distance thresholds chosen for each granularity are given in table 2. Table 2. This table presents the thresholds of temporal distances (columns) for each temporal granularity (rows). Very close

Close

Quite close

Quite far

Far

Very far

Very short-term (hours) Short-term (days)

< 0.16