Middleware 2003: ACM/IFIP/USENIX International Middleware Conference, Rio de Janeiro, Brazil, June 16-20, 2003, Proceedings (Lecture Notes in Computer Science, 2672) 3540403175, 9783540403173

Table of contents :
Middleware 2003
Preface
Organization
Table of Contents
Approximate Object Location and Spam Filtering on Peer-to-Peer Systems
1 Introduction
2 Background: Structured P2P Overlays
2.1 Routing
2.2 Data Location
3 Approximate DOLR
3.1 Approximate DOLR Design
3.2 A Basic ADOLR Prototype on Tapestry
3.3 Optimizing ADOLR Location
3.4 Concurrent Publication
4 Approximate Text Addressing
4.1 Finding Text Similarity
4.2 Trade-offs
5 Decentralized Spam Filtering
5.1 Basic Operation
5.2 Enhancements and Optimizations
6 Evaluation
6.1 Fingerprint on Random Text
6.2 Fingerprint on Real Email
6.3 Efficient Fingerprint Routing w/ TTLs
7 Related Work
8 Ongoing and Future Work
References
1 Analysis of Robustness of Text Fingerprinting
Efficient Peer-to-Peer Keyword Searching
1 Introduction
1.1 Non-goals
1.2 Overview
2 System Model
2.1 Partitioning
2.2 Centralized or Distributed Organization
2.3 Ranking of Results
2.4 Update Discovery
2.5 Placement
3 Efficient Support for Peer-to-Peer Search
3.1 Bloom Filters
3.2 Caches
3.3 Incremental Results
3.4 Virtual Hosts
3.5 Discussion
3.6 Ranking of Results
3.7 Load Balancing
4 Simulation Infrastructure
4.1 Goals
4.2 Design
4.3 Validation
5 Experimental Results
5.1 Scalability and Virtual Hosts
5.2 Bloom Filters and Caching
5.3 Putting It All Together
6 Related Work
7 Conclusions
References
NaradaBrokering: A Distributed Middleware Framework and Architecture for Enabling Durable Peer-to-Peer Grids
1 Introduction
2 NaradaBrokering
2.1 Broker Organization
2.2 Dissemination of Events
2.3 Failures and Recovery
2.4 JMS Compliance
2.4.1 JMS Performance Data
3 Support for P2P Interactions in NaradaBrokering
3.1 JXTA
3.2 JXTA & NaradaBrokering
3.2.1 The Interaction Model
3.2.2 Interaction Disseminations
3.3 Performance Measurements
4 NaradaBrokering’s Transport Framework
4.1 Some Performance Measurements
5 Performance Monitoring and Aggregation
6 Security Framework
7 Conclusions and Future Work
Bibliography
A Framework for Event Composition in Distributed Systems
1 Introduction
2 Motivation
2.1 Application Scenario: The Active O.ce
3 Related Work
4 Design and Architecture
4.1 Publish/Subscribe Infrastructure Support
4.2 Composite Event Detection Framework
5 Composite Event Detection
5.1 Composite Event Detectors
5.2 Core Composite Event Language
6 Distributed Detection
6.1 Mobile Composite Event Detectors
6.2 Distribution Policy
6.3 Detection Policy
7 Implementation Using JMS
7.1 Evaluation and Results
8 Further Work: Higher-Level Specification Languages
9 Conclusions
References
A Appendix: Formalising the Time and Event Models
A.1 Definition of Interval Timestamps
A.2 Formalising Describable Events and Input Sequences
B Appendix: Generating Composite Event Detectors
Content Distribution for Publish/Subscribe Services
1 Introduction
2 A Publish/Subscribe System with Content Distribution Engine
3 Content Delivery Strategies
3.1 Access-Time Strategy Based on Access Pattern
3.2 Push-Time Strategy Based on Subscription Information and Matching
3.3 Push-Time and Access-Time Schemes Based on Subscription Information and Access Pattern
3.4 Summary of Strategies
4 Workload and Simulator
4.1 Publishing Stream Generation
4.2 Request Stream Generation
4.3 Subscription Generation
5 Experiments
5.1 Metrics and Experiment Setup
5.2 Comparing Dual-Methods and Dual-Caches
5.3 Overall Hit Ratio with Perfect Subscriptions
5.4 Influence of Subscription Quality
5.5 Hit Ratio versus Time
5.6 Traffic Overhead
6 Related Work
7 Conclusion
References
Supporting Mobility in Content-Based Publish/Subscribe Middleware
1 Introduction
2 Content-Based Publish/Subscribe
2.1 Publish/Subscribe Systems
2.2 Content-Based Routing
3 Publish/Subscribe Systems and Mobility
3.1 Mobility Issues in Publish/Subscribe Middleware
3.2 Physical Mobility
3.3 Logical Mobility
4 Notification Delivery with Roaming Clients
4.1 Main Idea
4.2 Discussion
5 Location-Dependent Filters for Logical Mobility
5.1 Main Idea
5.2 Example
5.3 Adaptivity
5.4 Discussion
6 Conclusions
References
Fine-Grained Dynamic Adaptation of Distributed Components
1 Introduction
2 The Comet Middleware
2.1 Principles
2.2 Example: Distributed Multimedia
2.3 Component Internals
3 Dynamic Adaptation in Comet
3.1 Protocols and Roles
3.2 Other Role Categories
3.3 Put It All Together: Dynamic Distributed Debugging
4 Quantitative Evaluation
5 Verification Contracts
5.1 Typing Contracts
5.2 Access Contracts
5.3 Security Contracts
6 Related Work
7 Conclusion
References
A Middleware for Context-Aware Agents in Ubiquitous Computing Environments
1 Introduction
2 Why a Middleware for Context-Awareness?
3 Context Model
3.1 The Context Predicate
3.2 Ontologies to Describe Context Predicates
4 Gaia
5 Enabling Context-Awareness
5.1 Overview of Context Infrastructure
5.2 Use of Different Reasoning Mechanisms by Different Agents
5.3 Context Providers
5.4 Context Synthesizers
5.5 Context Consumers (or Context-Aware Applications)
5.6 Context Provider Lookup Service
5.7 Context History
6 Ontologies for Semantic Interoperability
6.1 Ontologies in Gaia
6.2 The Ontology Server
6.3 Ontologies for Smoother Interaction between Agents
7 Implementation
8 Related Work
9 Future Work
10 Conclusion
References
Adaptable Architectural Middleware for Programming-in-the-Small-and-Many
1 Introduction
2 Middleware Objectives
3 Middleware Core
3.1 Architectural Abstractions
3.2 Efficiency
3.3 Scalability
4 Middleware Extensions
4.1 Connector Extensions
2672b162.pdf
4 Middleware Extensions
4.1 Connector Extensions
4.2 Component Extensions
4.3 Event Extensions
4.4 Other Extensions
5 Tool Support
5.1 Modeling and Analysis
5.2 Deployment and Run-Time Monitoring
6 Further Evaluation and Experience
6.1 Dynamic Service Discovery
6.2 Military Deployment
6.3 Airborne System
7 Related Work
8 Conclusions and Future Work
References
Opportunistic Channels: Mobility-Aware Event Delivery
1 Introduction
2 System Architecture
2.1 Overview of JECho
2.2 Broker Adaptation Architecture
3 Opportunistic Channels in Mobile Environments
3.1 Mobility Adaptation
3.2 Modulator Handoff
3.3 Experimental Evaluation
4 Load Balancing in Opportunistic Channels
4.1 Adaptations for Load Balancing
4.2 Experimental Evaluation
5 Related Work
5.1 Publish/Subscribe Systems
5.2 Mobile IP and Its Location Management
5.3 Adaptations in Mobile Environments
6 Conclusions and Future Work
References
Congestion Control in a Reliable Scalable Message-Oriented Middleware
1 Introduction
1.1 System Model
2 The Congestion Control Problem
3 Congestion Control Protocols
3.1 PHB-Driven Congestion Control
3.2 SHB-Driven Congestion Control
4 Implementation in the Gryphon Broker
5 Experimental Results
6 Related Work
7 Conclusion
References
On Shouting "Fire!": Regulating Decoupled Communication in Distributed Systems
1 Introduction
2 An Institutional Alarm Policy -- A Motivating Example
2.1 An Alarm Policy for a Hospital
2.2 On the Communal Nature of Policy $AP$
3 Law-Governed Interaction (LGI) -- An Overview
3.1 On the Nature of LGI Lawsh and Their Decentralized Enforcement
3.2 The Deployment of LGI
4 Implementation of the Alarm Policy
4.1 Law $cal AP$ to Regulate Alarms
5 On the Performance of the Proposed Mechanism
5.1 Load on the Mediator
5.2 Congestion Caused by Unruly Informers
5.3 Throughput of the Mediation and End-to-End Latency
6 Conclusion
References
Performance Comparison of Middleware Architectures for Generating Dynamic Web Content
1 Introduction
2 Background
2.1 PHP (Hypertext Preprocessor)
2.2 Java HTTP Servlets
2.3 Enterprise Java Beans
2.4 Summary
3 Benchmarks
3.1 Online Bookstore Benchmark
3.2 Auction Site Benchmark
4 Hardware and Software Environment
4.1 Client Emulation Implementation
4.2 Application Logic Implementation
4.3 Software Environment
4.4 Hardware Platform
4.5 Measurement Methodology
4.6 Configurations
5 Experimental Results for the Online Bookstore
5.1 Shopping Mix
5.2 Browsing Mix
5.3 Ordering Mix
5.4 Summary
6 Experimental Results for the Auction Site
6.1 Bidding Mix
6.2 Browsing Mix
6.3 Summary
7 Related Work
8 Conclusions
References
Prefetching Based on Web Usage Mining
1 Introduction
2 Related Work
2.1 Sequential Prediction
2.2 Web Usage Mining
2.3 Prefetching
3 Predicting User Access Patterns
3.1 The Fuzzy LZ Learning Technique
3.2 Evaluation of Fuzzy-LZ
4 System Deployment on Experimental Test Bed
4.1 Experimental Design
4.2 Measurements and Results
5 Concluding Remarks
References
Distributed Versioning: Consistent Replication for Scaling Back-End Databases of Dynamic Content Web Sites
1 Introduction
2 Environment
2.1 Programming Model
2.2 Consistency Model
2.3 Cluster Architecture
3 Distributed Versioning
3.1 Lazy Read-One, Write-All Replication
3.2 Assigning and Using Version Numbers
3.3 1-Copy Serializability and Absence of Deadlock
3.4 Limiting the Number of Conflicts
3.5 Reducing Conflict Duration
3.6 Rationale
4 Implementation
4.1 Overview
4.2 Transaction Start
4.3 Read and Write
4.4 Completion of Reads and Writes
4.5 Early Version Releases
4.6 Commit/Abort
4.7 1-Copy-Serializability
4.8 Load Balancing
4.9 Fault Tolerance and Data Availability
5 Loose Consistency Models
5.1 Definition
5.2 Implementation of Loose Consistency Methods
6 Benchmarks
6.1 TPC-W
6.2 Auction Site Benchmark
6.3 Bulletin Board
6.4 Client Emulation
7 Experimental Environment and Methodology
7.1 Hardware
7.2 Software
7.3 Simulation Methodology
8 Results
8.1 Comparison of Distributed Versioning and Conservative 2PL
8.2 Comparison of Distributed Versioning and Loose Consistency Methods
9 Related Work
10 Conclusions
References
Abstraction of Transaction Demarcation in Component-Oriented Platforms
1 Introduction
2 What Transaction Demarcation Is
2.1 The Policies
2.2 The Domains
3 Challenges for Transaction Demarcation
3.1 Transaction Demarcation Abstraction
3.2 Transaction Monitor Abstraction
3.3 Organization of TD Policies
3.4 Integration of New TD Policies
4 Open Transaction Demarcation Framework
4.1 Overview
4.2 Abstraction of Transactional Monitor
4.3 Abstraction of Transaction Demarcation
4.4 Integration of New TD Policies
5 Java Open Transaction Demarcation Framework
5.1 Transaction Monitor Wrapper
5.2 Transaction Domain
5.3 Transaction Policy
5.4 Platform Usage
6 Experimentations
6.1 Context and Scenario
6.2 The Memory Evaluation
6.3 The CPU Evaluation
6.4 Example of Heterogeneity in Middleware: EJB over OTS
7 Lessons
8 Conclusion and Perspectives
References
Optimising Java RMI Programs by Communication Restructuring
1 Introduction
1.1 Contributions
1.2 Structure
2 Related Work
3 The Veneer Framework
4 Optimisations
4.1 Call Aggregation
4.2 Server Forwarding
4.3 Plan Caching
5 Maintaining Semantics
5.1 Differences between Local and Remote Calls
5.2 Call-Backs
6 Experimental Evaluations
6.1 Vector Arithmetic
6.2 The MUD Example
7 Future Work
8 Conclusion
References
The JBoss Extensible Server
1 Introduction
1.1 The Foundation
1.2 The Building
1.3 Organization of This Paper
2 JMX Foundation
2.1 Dynamic MBeans and Standard MBeans
2.2 The MBean Server
2.3 Reflection in JMX
3 Service Components
3.1 Service Lifecycle
3.2 Service Descriptors
3.3 Dependency Management
3.4 Deployment and Undeployment
3.5 Class Loading Issues
3.6 Unified Class Loaders
3.7 Dynamic Proxy Usage
4 Meta-level Architecture for Generalized EJBs
4.1 Reified Method Invocations
4.2 Remote Invocation Architecture
4.3 Containers
4.4 Interceptors as Pluggable Aspects
5 Project History
6 Ongoing and Future Work
7 Related Work
8 Concluding Remarks
References
Flexible and Adaptive QoS Control for Distributed Real-Time and Embedded Middleware
1 Introduction and Background
2 Related Work
2.1 Our Earlier DRE Middleware Efforts
2.2 Other Adaptive DRE Middleware Efforts
3 Managing End-to-End Real-Time QoS via Middleware-Mediated Resource Management Mechanisms
3.1 Priority-Based OS Resource Management
3.2 Priority-Based Network Resource Management
3.3 Reservation-Based OS Resource Management
3.4 Reservation-Based Network Resource Management
4 Applying Managed QoS in DRE Applications
5 Empirical Results
5.1 Priority-Based End-to-End Adaptive QoS
5.2 Reservation-Based End-to-End Adaptive QoS
6 Concluding Remarks
References
Large-Scale Service Overlay Networking with Distance-Based Clustering
1 Introduction
2 Assumptions
2.1 Service Model
2.2 Overlay Networks
3 HFC Topology Construction
3.1 Distance Map Obtainment
3.2 Clustering by Graph Theory
3.3 Selection of Border Proxies
4 Service Routing Information Distribution
5 Hierarchical Service Path Finding
5.1 Inter-cluster Service Path Finding
5.2 Intra-cluster Service Path Finding
6 Performance Studies
6.1 State Information Maintenance Overhead
6.2 Service Path Efficiency
7 Conclusions
References
A Step Towards a New Generation of Group Communication Systems
1 Introduction
1.1 Context
1.3 Contribution: A New Architecture
1.2 Traditional Group Communication Architecture
2 Existing Group Communication Architectures
2.1 Monolithic Systems
2.2 Modular Protocol Stacks
2.3 Discussion
3 The New Architecture
3.1 Overview of the New Architecture
3.2 Augmented Version of the New Architecture
3.3 Full Version of the New Architecture
4 Assessment of the New Architecture
4.1 Less Complex Stack
4.2 More Powerful Stack (Provides More Functionalities)
4.3 Higher Responsiveness
4.4 Minor Efficiency Issue
5 Conclusion
References
A Middleware-Based Application Framework for Active Space Applications
1 Introduction
2 Active Space Applications’ Key Issues
2.1 Resource-Awareness
2.2 Multi-device
2.3 User-Centrism
2.4 Run-Time Adaptation
2.5 Mobility
2.6 Context-Sensitivity
2.7 Active Space Independence
3 Application Model
3.1 Model
3.2 Presentation
3.3 Controller
3.4 Adapter
3.5 Coordinator
4 Application Mapping
5 Application Management
5.1 Application Instantiation
5.2 Application Termination
5.3 Application Suspension and Resumption
5.4 Application Reliability
5.5 Application Mobility
6 Addressing the Active Space Application Development Key Issues
7 Music Player Example
7.1 Implementation Details
7.2 Instantiating and Using the Application
8 Performance Evaluation
9 Related Work
10 Conclusions and Future Work
References
A Proactive Middleware Platform for Mobile Computing
1 Introduction
2 Motivation
2.1 Requirements
2.2 An Infrastructure for Proactive Adaptation
2.3 Related Work
3 System Architecture
3.1 Step 1: Generic Support for Run-Time Extensions with PROSE
3.2 Step 2: Extension Management with MIDAS
3.3 Example of MIDAS
4 Application Development for Proactive Environments
4.1 Basic Design
4.2 Application
4.3 Adaptation Example
4.4 Applications of the Adaptation Example
4.5 Discussion
5 Conclusion
References
A Flexible Middleware System for Wireless Sensor Networks
1 Introduction
2 Background
2.1 Wireless Sensor Networks (WSN)
2.2 Middleware Technology
2.3 WSN Middleware Requirements
2.4 The Web Services Technology
3 Proposed Middleware Service
3.1 Sensor Network Physical Components
3.2 System Components According to the Service-Oriented Architecture Pattern
3.3 Interoperability Stacks
4 System Operation
4.1 Step 1 – Initial Set Up
4.2 Step 2 – Interest Advertisement
4.3 Step 3 – Data Advertisement
4.4 Step 4 – Cluster Formation
5 Discussion
6 Related Work
7 Conclusions and Future Works
References
A Middleware Service for Mobile Ad Hoc Data Sharing, Enhancing Data Availability
1 Introduction
2 Middleware Service for Data Sharing within Collaborative Mobile Ad Hoc Groups
2.1 Secure Group Management
2.2 Data Management
3 Adaptive Data Replication for Enhanced Availability
3.1 Peers Profile
3.2 Work and Preventive Replicas
4 Assessment
5 Conclusion
References
Author Index

Citation preview

Lecture Notes in Computer Science Edited by G. Goos, J. Hartmanis, and J. van Leeuwen

2672

3

Berlin Heidelberg New York Barcelona Hong Kong London Milan Paris Tokyo

Markus Endler Douglas Schmidt (Eds.)

Middleware 2003 ACM/IFIP/USENIX International Middleware Conference Rio de Janeiro, Brazil, June 16-20, 2003 Proceedings

13

Series Editors Gerhard Goos, Karlsruhe University, Germany Juris Hartmanis, Cornell University, NY, USA Jan van Leeuwen, Utrecht University, The Netherlands Volume Editors Markus Endler PUC-Rio Departamento de Informática Rua Marquês de S˜ao Vicente 225, 22453-900 Rio de Janeiro, Brazil E-mail: [email protected] Douglas Schmidt Vanderbilt University Department of Electrical Engineering and Computer Science Box 1679,Station B Nashville 37235, TN, USA E-mail: [email protected] Cataloging-in-Publication Data applied for A catalog record for this book is available from the Library of Congress. Bibliographic information published by Die Deutsche Bibliothek Die Deutsche Bibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data is available in the Internet at .

CR Subject Classification (1998): C.2.4, D.4, C.2, D.1.3, D.3.2, D.2 ISSN 0302-9743 ISBN 3-540-40317-5 Springer-Verlag Berlin Heidelberg New York This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag. Violations are liable for prosecution under the German Copyright Law. Springer-Verlag Berlin Heidelberg New York a member of BertelsmannSpringer Science+Business Media GmbH http://www.springer.de ©IFIP International Federation for Information Processing, Hofstraße 3, A-2361 Laxenburg, Austria 2003 Printed in Germany Typesetting: Camera-ready by author, data conversion by Olgun Computergrafik Printed on acid-free paper SPIN: 10927472 06/3142 543210

Preface

Next-generation distributed applications and systems are increasingly developed using middleware. This dependency poses hard R&D challenges, including latency hiding, masking partial failure, information assurance and security, legacy integration, dynamic service partitioning and load balancing, and end-to-end quality of service specification and enforcement. To address these challenges, researchers and practitioners must discover and validate techniques, patterns, and optimizations for middleware frameworks, multi-level distributed resource management, and adaptive and reflective middleware architectures. Following the success of the past IFIP/ACM Middleware conferences (Lake District/UK, Palisades/USA, and Heidelberg/Germany) and building upon the success of past USENIX COOTS conferences, the Middleware 2003 conference is the premier international event for middleware research and technology. The scope of the conference is the design, implementation, deployment, and evaluation of distributed system platforms, architectures, and applications for future computing and communication environments. This year, we had a record of 158 submissions, among which the top 25 papers were selected for inclusion in the technical program of the conference. All papers were evaluated by at least three reviewers with respect to their originality, technical merit, presentation quality, and relevance to the conference themes. The selected papers present the latest results and breakthroughs on middleware research in areas including peer-to-peer computing, publish-subscriber architectures, component- and Web-based middleware, mobile systems, and adaptive computing. We would like to express our thanks to the authors of the submitted papers and to all the reviewers and program committee members for their efforts in reviewing a large number of papers in a relatively short time. We would also like to thank ACM, IFIP, USENIX, and the corporate sponsors for their technical sponsorship and financial support, respectively. Finally, special thanks go to Alexandre Sztajnberg, Renato Cerqueira, Fabio Kon, and Fabio M. Costa and all the other organizing committee members for their hard work and efforts to bring Middleware 2003 to Brazil and make it a successful conference.

June 2003

Markus Endler and Douglas Schmidt

Organization

Middleware 2003 was organized under the auspices of IFIP TC6 WG6.1 (International Federation for Information Processing, Technical Committee 6 [Communication Systems], Working Group 6.1 [Architecture and Protocols for Computer Networks]).

Steering Committee Gordon Blair (Lancaster University, UK) Jan de Meer (condat AG, Germany) Peter Honeyman (CITI, University of Michigan, USA) Guy LeDuc (University of Li`ege, Belgium) Kerry Raymond (DSTC, Australia) Alexander Schill (TU Dresden, Germany) Jacob Slonim (Dalhousie University, Canada)

Sponsoring Institutions ACM (Association for Computing Machinery) www.acm.org

IFIP (International Federation for Information Processing) www.ifip.or.at

The Advanced Computing System Association www.usenix.org

VIII

Organization

Supporting Companies IBM www.ibm.com

EA Industry

Sony www.sony.com

BBN Technologies

Sun Microsystems www.sun.com

Hewlett-Packart

Boeing www.boeing.com

Petrobr´ as

Organizing Committee General Chair: Program Co-chairs: Work-in-Progress and Posters Chair: Advanced Workshops Chair: Tutorials Chair: Local Arrangements Co-chairs:

Carlos Jos´e Pereira de Lucena (PUC-Rio, Brazil) Markus Endler (PUC-Rio, Brazil) Douglas Schmidt (Vanderbilt University, USA) Guruduth S. Banavar (IBM T.J. Watson, USA) Gordon Blair (Lancaster University, UK) Frank Buschmann (Siemens AG, Germany)

Alexandre Sztajnberg (UERJ, Brazil) Renato Cerqueira (PUC-Rio, Brazil) Student Travel Grant Chair: Hans-Arno Jacobsen (U. of Toronto, Canada) Student Volunteer Program Chair: Bruno Schulze (LNCC, Brazil) Publicity Co-chairs: Fabio M. Costa (UF Goi´ as, Brazil) Fabio Kon (University of S˜ ao Paulo, Brazil)

Organization

Technical Program Committee Gul Agha (University of Illinois, Urbana Champaign, USA) Jean Bacon (Cambridge University, UK) Gordon Blair (University of Lancaster, UK) Don Box (Microsoft, USA) Roy Campbell (University of Illinois, Urbana Champaign, USA) Andrew Campbell (Columbia University, USA) Geoff Coulson (Lancaster University, UK) Naranker Dulay (Imperial College, UK) Svend Frolund (HP Labs, USA) Chris Gill (Washington University, St. Louis, USA) Andy Gokhale (Vanderbilt University, USA) Rashid Guerraoui (EPF Lausanne, Switzerland) Arno Jacobsen (University of Toronto, Canada) Peter Honeyman (CITI, University of Michigan, USA) Fabio Kon (University of S˜ ao Paulo, Brazil) Doug Lea (SUNY Oswego, USA) Guy LeDuc (University of Li`ege, Belgium) Orlando Loques (UFF, Brazil) Joe Loyall (BBN Technologies, USA) Raimundo J. de Araujo Macedo (Federal University of Bahia, Brazil) Edmundo R. Mauro Madeira (University of Campinas, Brazil) Jan de Meer (condat AG, Germany) Klara Nahrstedt (University of Illinois, Urbana Champaign, USA) Priya Narasimhan (Carnegie Mellon University, USA) Carlos Pereira (UFRGS, Brazil) Vijay Raghavan (DARPA, USA) Kerry Raymond (DSTC, Australia) Luis Rodrigues (University of Lisboa, Portugal) Isabelle Rouvellou (IBM, USA) Bill Sanders (University of Illinois, Urbana Champaign, USA) Rick Schantz (BBN Technologies, USA) Alexander Schill (Technical University of Dresden, Germany) David Sharp (The Boeing Company, USA) Jacob Slonim (Dalhousie University, Canada) Jean-Bernard Stefani (INRIA, Grenoble, France) Joe Sventek (Agilent Labs, UK) Janos Sztipanovits (Vanderbilt University, USA) Nalini Venkatasubramanian (University of California, Irvine, USA) Steve Vinoski (IONA Technologies, USA) Werner Vogels (Cornell University, USA) Martina Zitterbart (University of Karlsruhe, Germany)

IX

X

Organization

Additional Reviewers Filipe Araujo Michael Atighetchi Luciano Porto Barreto Roberto Speicys Cardoso Isidro Castineyra Dan Cerys Po-Hao Chang Liping Chen Joshua Chia Renato Cerqueira Nuno Correia Fabio M. Costa Lou Degenaro Christo Devaraj Gary Duzan Paulo Ferreira Marcelo Finger Islene Calciolari Garcia Jeff Gray Andrei Goldchleger Chris Jones Richard King Nirman Kumar Youngmin Kwon Soham Mazumdar Kirill Mechitov

Thomas Mikalsen Hugo Miranda Gail Mitchell Balachandran Natarajan Dennis Noll Partha Pal Jeff Parsons Irfan Pyarali Smitha Reddy Craig Rodrigues Wendy Roll Paul Rubel Bruno R. Schulze Koushik Sen Rich Shapiro Praveen Sharma Flavio Assis Silva Francisco J. Silva e Silva Irineu Sotoma Sameer Sundresh Alexandre Sztajnberg Stefan Tai Maria Beatriz Felgar de Toledo Nanbor Wang Franklin Webber John Zinky

Table of Contents

Peer-to-Peer Computing Approximate Object Location and Spam Filtering on Peer-to-Peer Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Feng Zhou, Li Zhuang, Ben Y. Zhao, Ling Huang, Anthony D. Joseph, and John Kubiatowicz

1

Efficient Peer-to-Peer Keyword Searching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Patrick Reynolds and Amin Vahdat NaradaBrokering: A Distributed Middleware Framework and Architecture for Enabling Durable Peer-to-Peer Grids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Shrideep Pallickara and Geoffrey Fox

Publish-Subscribe Middleware I A Framework for Event Composition in Distributed Systems . . . . . . . . . . . . 62 Peter R. Pietzuch, Brian Shand, and Jean Bacon Content Distribution for Publish/Subscribe Services . . . . . . . . . . . . . . . . . . . . 83 Mao Chen, Andrea LaPaugh, and Jaswinder Pal Singh Supporting Mobility in Content-Based Publish/Subscribe Middleware . . . . 103 Ludger Fiege, Felix C. G¨ artner, Oliver Kasten, and Andreas Zeidler

Adaptability and Context-Awareness Fine-Grained Dynamic Adaptation of Distributed Components . . . . . . . . . . 123 Fr´ed´eric Peschanski, Jean-Pierre Briot, and Akinori Yonezawa A Middleware for Context-Aware Agents in Ubiquitous Computing Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Anand Ranganathan and Roy H. Campbell Adaptable Architectural Middleware for Programming-in-the-Small-and-Many . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 Marija Mikic-Rakic and Nenad Medvidovic

Publish-Subscribe Middleware II Opportunistic Channels: Mobility-Aware Event Delivery . . . . . . . . . . . . . . . . 182 Yuan Chen, Karsten Schwan, and Dong Zhou

XII

Table of Contents

Congestion Control in a Reliable Scalable Message-Oriented Middleware . . 202 Peter R. Pietzuch and Sumeer Bhola On Shouting “Fire!”: Regulating Decoupled Communication in Distributed Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 Takahiro Murata and Naftaly H. Minsky

Web-Based Middleware Performance Comparison of Middleware Architectures for Generating Dynamic Web Content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 Emmanuel Cecchet, Anupam Chanda, Sameh Elnikety, Julie Marguerite, and Willy Zwaenepoel Prefetching Based on Web Usage Mining . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262 Daby M. Sow, David P. Olshefski, Mandis Beigi, and Guruduth Banavar Distributed Versioning: Consistent Replication for Scaling Back-End Databases of Dynamic Content Web Sites . . . . . . . . . . 282 Cristiana Amza, Alan L. Cox, and Willy Zwaenepoel

Component-Based Middleware Abstraction of Transaction Demarcation in Component-Oriented Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 Romain Rouvoy and Philippe Merle Optimising Java RMI Programs by Communication Restructuring . . . . . . . 324 Kwok Cheung Yeung and Paul H. J. Kelly The JBoss Extensible Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344 Marc Fleury and Francisco Reverbel

Next Generation Middleware Flexible and Adaptive QoS Control for Distributed Real-Time and Embedded Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374 Richard E. Schantz, Joseph P. Loyall, Craig Rodrigues, Douglas C. Schmidt, Yamuna Krishnamurthy, and Irfan Pyarali Large-Scale Service Overlay Networking with Distance-Based Clustering . . 394 Jingwen Jin and Klara Nahrstedt A Step Towards a New Generation of Group Communication Systems . . . . 414 Sergio Mena, Andr´e Schiper, and Pawel Wojciechowski

Table of Contents

XIII

Mobile and Ubiquitous Computing A Middleware-Based Application Framework for Active Space Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433 Manuel Rom´ an and Roy H. Campbell A Proactive Middleware Platform for Mobile Computing . . . . . . . . . . . . . . . . 455 Andrei Popovici, Andreas Frei, and Gustavo Alonso A Flexible Middleware System for Wireless Sensor Networks . . . . . . . . . . . . 474 Fl´ avia Coimbra Delicato, Paulo F. Pires, Luci Pirmez, and Luiz Fernando Rust da Costa Carmo A Middleware Service for Mobile Ad Hoc Data Sharing, Enhancing Data Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 493 Malika Boulkenafed and Val´erie Issarny

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513

Approximate Object Location and Spam Filtering on Peer-to-Peer Systems Feng Zhou, Li Zhuang, Ben Y. Zhao, Ling Huang, Anthony D. Joseph, and John Kubiatowicz Computer Science Division, U. C. Berkeley {zf,zl,ravenben,hling,adj,kubitron}@cs.berkeley.edu

Abstract. Recent work in P2P overlay networks allow for decentralized object location and routing (DOLR) across networks based on unique IDs. In this paper, we propose an extension to DOLR systems to publish objects using generic feature vectors instead of content-hashed GUIDs, which enables the systems to locate similar objects. We discuss the design of a distributed text similarity engine, named Approximate Text Addressing (ATA), built on top of this extension that locates objects by their text descriptions. We then outline the design and implementation of a motivating application on ATA, a decentralized spam-“ltering service. We evaluate this system with 30,000 real spam email messages and 10,000 non-spam messages, and “nd a spam identi“cation ratio of over 97% with zero false positives. Keywords: Peer-to-peer, DOLR, Tapestry, spam “ltering, approximate text matching

1 Introduction Recent work on structured P2P overlay networks ([5,18], [15], [11], [10]) utilize scalable routing tables to map unique identi“ers to network locations, providing interfaces such as Decentralized Object Location and Routing (DOLR) and Distributed Hashtables (DHT). They allow network applications such as distributed “le systems and distributed web caches to ef“ciently locate and manage object replicas across a wide-area network. While these systems excel at locating objects and object replicas, they rely on known Globally Unique IDenti“ers (GUID) for each object, commonly generated by applying a secure hash function to the object content. This provides a highly speci“c naming scheme, however, and does not lend itself to object location and management based on semantic features. To address this problem, we propose an approximate location extension to DOLR systems to publish and locate objects using generic feature vectors composed of a number of values generated from its description or content. Any object can be addressed by a feature vector matching a minimal threshold number of entries with its original feature vector. Based on this extension, we propose an Approximate Text Addressing (ATA) facility, which instantiates the approximate location extension by using block text “ngerprints as features to “nd matches between highly similar text documents. To validate the ATA design as well as the approximate object location extension, we design a decentralized spam-“ltering application that leverages ATA to accurately identify junk M. Endler and D. Schmidt (Eds.): Middleware 2003, LNCS 2672, pp. 1…20,2003. c IFIP International Federation for Information Processing 2003


2

Feng Zhou et al.

email messages despite formatting differences and evasion efforts by spammers. We evaluate the accuracy of our “ngerprint vector scheme via simulation and analysis on real email data, and explore the trade-offs between resource consumption and search accuracy. The rest of this paper is as follows: Section 2 brie”y describes existing work in P2P overlays. Section 3 presents our approximation extension to DOLR systems and a prototype implementation. Section 4 describes the design of ATA and Section 5 discusses the design of the decentralized spam “lter. Section 6 presents simulation and experimental results, followed by a discussion of related work in Section 7 and status and future work in Section 8. Finally, we provide a mathematical analysis of the robustness of text-based “ngerprinting in Appendix A.

2 Background: Structured P2P Overlays In this section, we “rst present background material on structured P2P overlays. Different protocols differ in semantics details and performance objectives. While we present our work in the context of Tapestry for performance reasons, our design is general, and our results can be generalized to most structured P2P protocols. 2.1

Routing

Tapestry is an overlay location and routing layer “rst presented in [18], with a rigorous treatment of dynamic algorithms presented in [5]. Like other structured P2P protocols, object and node IDs are pseudo-randomly chosen from the namespace of “xed-length bit sequences with a common base (e.g. Hex). Tapestry uses local routing tables at each node to route messages incrementally to the destination ID digit by digit (e.g., 4*** =⇒ 45** =⇒ 459* =⇒ 4598 where *•s represent wildcards). A node N has a neighbor map with multiple levels, where each level represents a matching pre“x up to a digit position in the ID. Each level of the neighbor map contains a number of entries equal to the base of the ID, where the ith entry in the j th level is the location of the node closest in network latency that begins with prefixj−1 (N ) + i. To forward on a message from its nth hop router, Tapestry examines its n + 1th level routing table and forwards the message to the link corresponding to the n + 1th digit in the destination ID. This routing substrate provides ef“cient location-independent routing within a logarithmic number of hops and using compact routing tables. Figure 1 shows a Tapestry routing mesh. 2.2

Data Location

In Tapestry, a server S makes a local object O available to others by routing a •publishŽ message to the object•s •root node,Ž the live node O•s identi“er maps to. At each hop along the path, a location mapping from O to S is stored. Mappings for multiple replicas are stored sorted according to distance from the local node. See Figure 2 for an example of object publication. Here two replicas of the same object are published. A client routes a query message towards the root node. The message queries each hop router along the

Approximate Object Location and Spam Filtering on Peer-to-Peer Systems 8F4B

3

L1

89E3

5230

L2

L3

8909

L2

8900

8BB2

8951

L3

L3

L4

8954 8957

L4

L4

895D

L1

AC78

8112

L2

Fig. 1. Tapestry routing example. Path taken by a message from node 5230 for node 8954 in Tapestry using hexadecimal digits of length 4 (65536 nodes in namespace). 4377

4377

437A

437A

43FE

43FE

4228

4228 4664

(4378)

4361

(4378)

4BF4 197E

4A6D CE75

39AA

4664 4361

4BF4 (4378)

Fig. 2. Publication in Tapestry. To publish object 4378, server 39AA sends publication request towards root, leaving a pointer at each hop. Server 4228 publishes its replica similarly. Since no 4378 node exists, object 4378 is rooted at node 4377.

197E

4A6D CE75

39AA

(4378)

Fig. 3. Object Location in Tapestry: Three different location requests. For instance, to locate GUID 4378, query source 197E routes towards the root, checking for a pointer at each step. At node 4361, it encounters a pointer to server 39AA.

way, and routes towards S when it nds the O to S location mapping. Note that for nearby objects, query messages quickly intersect the path taken by publish messages, resulting in quick search results that exploit locality [18]. See Figure 3 for an example of object location. Notice how locality is exploited by directing location requests to nearby replicas.

3 Approximate DOLR DOLR systems like Tapestry provide deterministic, scalable, and ef cient location and routing services, making them attractive platforms for deploying wide-area network

4

Feng Zhou et al.

applications. Files, in particular, can be located ef“ciently if their canonical name is known. Previous approaches, however, generate Globally Unique IDenti“ers (GUID) by a secure hash (e.g. SHA-1) of the content. This approach signi“cantly limits the usability of the system in scenarios where users do not known exact names of objects, but rather perform searches based on general characteristics of the system. In particular, these scenarios might include searches for data that closely approximates, or is similar to known data with certain properties. Examples might include searching for audio or video that matches existing works in content features, or searching or lightly modi“ed replicas of existing data. 3.1 Approximate DOLR Design Here we propose an extension to DOLR, Approximate DOLR, as a generic framework to address some of the needs of these applications. In an ADOLR system, we apply application-speci“c analysis to given objects to generate feature vectors that describe its distinctive features, and provide a translation mechanism between these applicationdriven features and a traditional GUID obtained from a secure content hash of the object contents. This query ability on features applies to a variety of contexts. In the world of multimedia search and retrieval, we can extract application-speci“c characteristics, and hash those values to generate feature vectors. Any combination of “eld to value mappings can be mapped to a feature vector, given a canonical ordering of those “elds. For example, this can be applied to searching for printer drivers given printer features such as location, manufacturer, and speed. If features are canonically ordered as [location, manufacturer, speed], then an example feature vector might be [hash(443 Soda), hash(HP), hash(12ppm)]. Each member of the vector, a feature, is an application-speci“c feature encoded as a hashed identi“er. For each feature f, an object (feature object) is stored within the network. The feature object is a simple object that stores the list of GUIDs of all objects whose feature vectors include f. Clients searching for objects with a given feature set “nds a set of feature objects in the network, each associated with a single feature, and selects the GUIDs which appear in at least T feature objects, where T is a tunable threshold parameter used to avoid false positives while maintaining the desired generality of matches. The •publicationŽ of an object O in an ADOLR system proceeds as follows. First, its content-hash derived GUID is “rst published using the underlying P2P DOLR layer. This assures that any client can route messages to the object given its GUID. Next, we generate a feature vector for O. For each feature in the vector, we try to locate its associated feature object. If such an object is already available in the system, we append the current GUID to that object. Otherwise, we create a new feature object identi“ed by the feature, and announce its availability into the overlay. To locate an object in anADOLR system, we “rst retrieve the feature object associated with each entry of the feature vector. We count the number of distinct feature objects each unique GUID appears in, and select the GUID(s) that appear in a number greater than some preset threshold. The GUID(s) are then used to route messages to the desired object.

Approximate Object Location and Spam Filtering on Peer-to-Peer Systems

5

The ADOLR API is as follows: – PublishApproxObject (FV, GUID). This publishes the mapping between the feature vector and the GUID in the system. A feature vector is a set of feature values of the object, whose de“nition is application speci“c. Later, one can use the feature vector instead of the GUID to search for the object. Notice that PublishApproxObject only publishes the mapping from FV to GUID. It does not publish the object itself, which should be done already using publish primitive of Tapestry when PublishApproxObject is called. – UnpublishApproxObject (FV, GUID). This removes the mapping from the FV to the GUID if this mapping exists in the network, which is the reverse of PublishApproxObject. – RouteToApproxObject (FV, THRES, MSG). This primitive routes a message to the location of all objects which overlap with our queried feature vector FV on more than THRES entries. The basic operations involve for each feature, retrieving a list of GUIDs that share that feature, doing a frequency count to “lter out GUIDs that match at least THRES of those features, and “nally routing the payload message MSG to them. For each object in the system with feature vector F V ∗ , the selection criterion is:
|F V ∗ F V | ≥ T HRES AN D 0 < T HRES ≤ |F V | The location operation is deterministic, which means all existing object IDs matching the criterion will be located and be sent the payload message. However, it is important to notice that this does not mean every matching object in the system will receive the message, because each object ID may correspond to multiple replicas, depending on the underlying DOLR system. The message will be sent to one replica of each matching object ID, hopefully a nearby replica if the DOLR utilizes locality. With this interface, we reduce the problem of locating approximate objects on P2P systems to “nding a mapping from objects and search criteria to feature vectors. The mapping should maintain similarity relationships, such that similar objects are mapped to feature vectors sharing some common entries. We show one example of such a mapping for text documents in Section 4. 3.2 A Basic ADOLR Prototype on Tapestry Here we describe an Approximate DOLR prototype that we have implemented on top of the Tapestry API. The prototype serves as a proof of concept, and is optimized for simplicity. The prototype also allows us to gain experience into possible optimizations for performance, robustness and functionality. The prototype leverages the DOLR interface for publishing and locating objects, given an associated identi“er. When PublishApproxObject is called on an object O, it begins by publishing O•s content-hashed object GUID using Tapestry. Then the client node uses Tapestry to send messages to all feature objects involved. Tapestry routes these messages to the nodes where these feature objects are stored. These nodes then add the new object GUID to the list of GUIDs inside the feature object. If any feature object

6

Feng Zhou et al. return GUID obje

ct set {

Look up approximate object with FV = {fv1, fv2, fv3} Client Node

Object Node X

(1) look up (1) look up (1) lo

ok u

(2) send m sg to "guid1

guid1}

fv2

p fv3

1111 0000 0000 1111 fv1

fv1

DOLR Layer "LocationFailu

{guid1}

Object Node Y

re" msg

"

fv4

{guid2, guid3}

fv3

{guid1, guid4}

4}

1, guid

{guid ject set UID ob

1111 0000 0000 1111 0000 1111 0000 1111

G return

Fig. 4. Location of an approximate object. Client node wants to send a message to all objects with at least 2 feature in {fv1, fv2, fv3}. It “rst sends lookup message to feature fv1, fv2 and fv3. fv2 does not exists. A Location Failure message is sent back. fv1 is managed by object node X. It sends back a list of IDs of all objects having feature fv1, which is {guid1}. Similar operation is done for feature fv3, whose IDs list {guid1, guid4}. Client node counts the occurrence of all IDs in all lists and “nds out guid1 to be the ID it is looking for. It then sends the payload message to object guid1 using Tapestry location message.

is not found in the network, the client node receives a LocationFailure message, creates a new feature object containing the new object, and publishes it. For the RouteToApproxObject call, the client node “rst uses Tapestry location to send messages to all feature objects, asking for a list of IDs associated with each feature value. Nodes where these feature objects reside receive these messages, do the lookup in their maps and send back the result. LocationFailure messages are sent back for nonexistent feature objects, and are counted as an empty ID list. The client node counts the occurrence of each GUID in the resulting lists. GUIDs with less than the threshold number of counts are removed. Finally, the message in this call is sent to the remaining object GUIDs An example of executing a RouteToApproxObject call is shown in Figure 4. Note that an analogous system can be implemented on top of a distributed hash table (DHT) abstraction on P2P systems. Instead of routing messages to previously published feature objects, one would retrieve each feature object by doing a get operation, appending the new GUID, and putting the object back using put. 3.3

Optimizing ADOLR Location

Our initial description of the RouteToApproxObject operation involves several roundtrips from the client node to nodes where the feature objects are stored. We propose two optimizations here that eliminates a network round-trip, reducing overall latency to that of a normal RouteToObject in a DOLR system at the cost of keeping a small amount of state on overlay nodes. The “rst optimization involves a client node caching the result of translating a feature vector to a GUID. now all future messages to the same feature vector are routing to the cached GUID.

Approximate Object Location and Spam Filtering on Peer-to-Peer Systems Send a message to approximate object with FV={fv1, fv2, fv3, fv4} Client Node

1111 0000 0000 1111 0000 1111 0000 1111 000 111 000 111

7

Object Node X

DOLR Layer

To fv1: (F

V, msg)

To fv2: (F

V, msg)

To fv3:

(FV, m

To fv4: (F

sg)

V, msg)

fv3

look up and match FV {guid1, guid3}

fv4

{guid1, guid2}

fv1

{guid1}

guid1{fv1, fv3, fv4, fv5} guid2{fv4, fv6, fv7, fv8} guid3{fv3, fv9, fv10, fv11}

11111 00000 00000 11111 00000 11111

msg

guid1

Fig. 5. Optimized ADOLR location. Client node wants to route a message to a feature vector {fv1, fv2, fv3, fv4}. It sends message to each identi“er fv1, fv2, fv3, fv4. fv2 doesn•t exist, so no object node receives this message. When object node X receives the messages to fv1, fv3 and fv4, it scans its local storage for all IDs matching fv1, fv3 and fv4, which is guid1. Then, object node X sends msg to guid1.

The second optimization is more complex, and illustrated in Figure 5. Normally, the client node retrieves a set of feature objects, counts GUID occurrences locally, then routes a message to the resulting GUID(s). The intuition here is that if features are identi“ed as hashed keys with reasonably low collision rates, each feature will likely only identify a small number (one or two) of objects with that feature. Furthermore, multiple feature objects are likely to be colocated together along with the object they identify, because new feature objects are created by the same node where the object is stored. Another way to look at this is that the feature object is in most cases published at the same time with the object itself by the same node. This implies we can route the application-level message to each feature in the feature vector, and expect it to arrive at the node where the desired object is stored. The key change here is that any node that is storing a feature object, (a “le providing a mapping from a feature to all GUIDs that share that feature), also stores the feature vectors of each of those GUIDs. Routing a message to a feature vector {X, Y, Z} means sending the message to each identi“er X, Y , and Z. Each message also includes the entire feature vector we•re querying for. When a node receives such a message, it immediately scans its local storage for all feature objects matching X, Y , or Z. For each GUID in these feature objects, the node determines the amount of overlap between its feature vector and the queried feature vector. If the overlap sati“es the query threshold, the message is delivered to that GUID•s location. This implies that any of the query messages contains enough information for a node to completely evaluate the ADOLR search on local information. If any locally stored feature objects contain references to matching objects, they can be evaluated immediately to determine if it satis“es the query. Because each message contains all necessary information to deliver the payload to the desired GUID, the set of messages sent to X, Y , and Z provide a level of fault-resilience against message loss. Finally, the determination of the desired GUID can occur when the “rst message is received, instead of waiting for all messages to arrive. The translation from the feature vector to one or more GUIDs occurs in the network, not the client node. This provides signi“cant communication savings.

8

Feng Zhou et al.

Nodes need to keep more state to support this optimization, however. In addition to storing feature objects (that keep the mapping between feature values and GUIDs), they also need to keep track of previously resolved feature vectors in order to drop additional requests for the same feature vector. This state can be stored on a temporary basis, and removed after a reasonable period of time (during which any other requests for the same feature vector should have arrived). 3.4

Concurrent Publication

There is one problem with the PublishApproxObject implementation described above. The lookup of feature objects and publication of new feature objects are not atomic. This can result in multiple feature objects for the same feature value being published if more than one node tries to publish an object with this feature value concurrently. We propose two solutions. First, we can exploit the fact that every object is mapped to a unique root node and serialize the publication on the root node. Every node is required to send a message to the root node of the feature value to obtain a leased lock before publishing the feature object. After the lock is acquired by the “rst node, other nodes trying to obtain it will fail, restart the whole process, and “nd the newly published feature object. This incurs another round-trip communication to the root node. In a more ef“cient •optimisticŽ way to solve this problem, the client node always assumes the feature object does not exist in the network. It tries to publish the object without doing a lookup beforehand. When the publication message travels through the network, each node checks whether it knows about an already published feature object with the same feature value. If such an object does exist, some node or at least the root will know about this. The node who detects this then cancels this publication and sends an message to the existing feature object to •mergeŽthe new information. This process is potentially more ef“cient since con”icts should be rare. In general, the operation is accomplished with a single one-way publication message. This optimistic approach can easily be implemented on top of DOLRs such as Tapestry using the recently proposed common upcall interface for peer to peer (P2P) overlays [2]. This proposed upcall interface allows P2P applications to override local routing decisions. Speci“cally, a node can •interceptŽ the publication message and handle con”icts as speci“ed above.

4 Approximate Text Addressing In this section, we present the design for the Approximate Text Addressing facility built on the Approximate DOLR extension, and discuss design decisions for exploring tradeoffs between computational and bandwidth overhead and accuracy. 4.1

Finding Text Similarity

Our goal is to ef“ciently match documents distributed throughout the network that share strong similarities in their content. We focus here on highly similar “les, such as modi“ed email messages, edited documents, or news article published on different web sites.

Approximate Object Location and Spam Filtering on Peer-to-Peer Systems

9

...

Fingerprint Vector 1303805399 8814255728

.........

R E V E R S E

S E L E C T

S O R T

n

7518253209 N

Substring Checksum

...

L

Fig. 6. Fingerprint Vector. A “ngerprint vector is generated from the set of checksums of all substrings of length L, post-processed with sort, selection and reverse operations.

The algorithm is as follows. Given a text document, we use a variant of block text “ngerprinting “rst introduced in [7] to generate a set of “ngerprints. The “ngerprint vector of a document is used as its feature vector in publication and location, using the Approximate DOLR layer. To calculate a block text “ngerprint vector of size N for a text document, we divide the document into all possible consecutive substrings of length L. A document of length n characters will have (n − L + 1) such strings. Calculating checksums of all such substrings is a fast operation which scales with n. We sort the set of all checksums by value, select a size N subset with the highest values, and reverse each checksum by digit (i.e. 123 ⇒ 321). This deterministically selects a random set without biasing the ID for pre“x or numerical routing. L is a parameterized constant chosen for each application to tune the granularity of similarity matches. For example, a size L of 50 might work well for email, where complete sentences might account for one substring; but less well for source code, where code fragments are often much longer in length. Figure 6 illustrates the “ngerprint process. The calculation is not expensive. Our Java prototype has a processing throughput of > 13M B/s for L = 50 on a 1Ghz PIII laptop. 4.2 Trade-offs There are obvious trade-offs between network bandwidth used and the accuracy of the search. First, the greater the number of entries N in a vector, the more accurate the match (less false-positives), and also the greater number of parallel lookup requests for each document. Next, the distance each lookup requests travels directly impacts bandwidth consumption on the overall network. ATA-enabled applications1 can bene“t from exploiting network-locality by matching against similar documents nearby in the network via a DOLR/DHT with object location locality such as Tapestry. Finally, a trade-off exists 1

Some example applications include spam “lters, plagiarism detection and news article clustering.

10

Feng Zhou et al.

between the number of publishers (those who indicate they have a particular document), and the resources required for a client to “nd a match in their query. Bandwidth and accuracy can be tuned by placing a Time-to-Live (TTL) “eld on the lookup query, constraining the scope of query messages. Clients who fail to “nd a match may publish their own documents, improving lookup performance for other clients. These are explored in detail in Section 6.

5 Decentralized Spam Filtering Spam, or unsolicited email, wastes time and valuable network resources, causing headaches for network administrators and home users alike. Currently the most widelydeployed spam “ltering systems scale to a university- or company- wide network, and use keyword matching or source address matching [13]. Although easy to deploy and manage, these systems often walk a “ne line between letting spam through and blocking legitimate emails. Our observation is that human recognition is the only fool-proof spam identi“cation tool. Therefore, we propose a decentralized spam “lter that pools the collective spam recognition results of all readers across a network. There already exist centralized collaborative spam “ltering systems, such as SpamNet [14], which claims to be peer-to-peer but actually uses a Napster-like architecture. To our knowledge ours is the “rst attempt to build a truly decentralized collaborative spam “ltering system. Compared to alternative university-wide centralized collaborated designs, the most important bene“t of our wide-area decentralized design lies in the fact that the effectiveness of the system grows with the number of its users. In such a system with huge number of users world-wide, it is highly probable that every spam email you receive has been received and identi“ed by somebody else before because of the large number of users. The deterministic behavior of DOLR systems will prove useful, because when any single peer publishes information about a speci“c email, that piece of information can be deterministically found by all clients. Therefore we can expect this system to be more responsive to new spam than systems in which different nodes publish/exchange spam information at certain intervals, such as [3]. Additionally, decentralized systems provide higher availability and resilience to failures and attacks than similar centralized solutions such as SpamNet. 5.1

Basic Operation

The decentralized spam “ltering system consists of two kinds of nodes, user agents and peers. User agents are extended email client programs that users use. They query peers when new emails are received and also send user•s feedback regarding whether a certain email is or is not spam to peers. A peer is a piece of long-running software that is installed typically on a university, department or company server that speaks to other peers worldwide and forms a global P2P network. When an email client receives a message from the server, the user agent extracts the body of the mail, drops format artifacts like extra spaces and HTML tags, generates a “ngerprint vector, and sends it to a peer in the DOLR system. The peer in turn queries the network using the Approximate DOLR API to see if information on the email has

Approximate Object Location and Spam Filtering on Peer-to-Peer Systems

11

been published. If a match is found, and it indicates the email is spam, the email will be “led separately or discarded depending on user preference. Otherwise, the message is delivered normally. If the user marks a new message as spam, the user agent marks the document, and tells the peer to publish this information into the network. 5.2

Enhancements and Optimizations

The basic design above allows human identi“cation of spam to quickly propagate across the network, which allows all users of the system to bene“t from the feedback of a few. There are several design choices and optimizations which will augment functionality and reduce resource consumption. Our “ngerprint vectors make reverse engineering and blocking of unknown emails very dif“cult. With the basic system, however, attackers can block well known messages (such as those from group mailing lists). We propose to add a voting scheme on top of the publish/search model. A count of positive and negative votes is kept by the system, and each user can set a threshold value for discarding or “ling spam using the count as a con“dence measure. A central authority controls the assignment and authentication of user identities. A user agent is required to authenticate itself before being able to vote for or against an email. Thus we can restrict the number of votes a certain user agent can perform on a certain email. Another type of attack is for spammers to “nd arbitrary text segments with checksum values more likely to be selected by the “ngerprint selection algorithm. By appending such •preferredŽ segments to their spam emails, spammers can “x the resulting email “ngerprint vectors to attempt to avoid detection. Note that this attack can only succeed if a continuous stream of unique text segments are generated and an unique segment is appended to each spam message. This places a signi“cant computational overhead on the spammer that scales with the number of spam messages sent. Additionally, mail clients can choose randomly from a small set of “ngerprint calculation algorithms. Different “ngerprinting methods can include transforming the text before calculating the checksums, changing the checksum method, or changing the “ngerprint selection method. To circumvent this, the spammer would need to “rst determine the set of “ngerprint algorithms, and then append a set of preferred segments, each segment overcoming a known selection algorithm. While different “ngerprint algorithms generate distinct spam signatures for the same spam, partitioning the user population and reducing the likelihood of a match, it also requires signi“cantly more computational overhead to overcome. Optimizations can be made for centralized mail servers to compute “ngerprint vectors for all incoming messages. These vectors can be compared locally to identify •popularŽ messages, and lookups performed to determine if they are spam. Additionally, the server can attach precomputed “ngerprint vectors and/or spam “ltering results as custom headers to messages, reducing local computation, especially for thin mail clients such as PDAs.

6 Evaluation In this section, we use a combination of analysis, experimentation on random documents and real emails to validate the effectiveness of our design. We look at two aspects

12

Feng Zhou et al.

of “ngerprinting, robustness to changes in content and false positive rates. We also evaluate “ngerprint routing constrained with time-to-live (TTL) “elds, tuning the tradeoff between accuracy and network bandwidth consumption. 6.1 Fingerprint on Random Text We begin our evaluation by examining the properties of text “ngerprinting on randomly generated text. In particular, we examine the effectiveness of “ngerprinting at matching text after small modi“cations to their originals, and the likelihood of matching unrelated documents (false positive rate). Robustness to Changes in Content. We begin by examining the robustness of the “ngerprint vector scheme against small changes in a document, by measuring the probability a “ngerprint vector stays constant when we modify small portions of the document. We “x the “ngerprint vector size, and want to measure the robustness against small changes under different threshold constants (THRES). In experiments, we take 2 sets of random text documents of size 1KB and 5KB, which match small- and large-sized spam messages respectively, and calculate their “ngerprint vectors before and after modifying 10 consecutive bytes. This is similar to text replacement or mail merge schemes often used to generate differentiated spam. We measure the probability of at least T HRES out of |F V | “ngerprints matching after modi“cation as a function of threshold (T HRES) and the size of the document (1KB or 5KB). Here, “ngerprint vector size is 10, |F V | = 10. We repeat that experiment with a modi“cation of 50 consecutive bytes, simulating the replacement of phrases or sentences and “nally modifying 5 randomly placed words each 5 characters long. In addition to the simulated experiments, we also developed a simple analytical model for these changes based on basic combinatorics. We present this model in detail in Appendix A. For each experiment, we plot analytical results predicted by our model in addition to the experimental results. In Figure 7, we show for each scenario experimental results gathered on randomized text “les, by comparing “ngerprint vectors before and after modi“cations. From Figure 7, we can see the model in Appendix A predicts our simulation data almost exactly under all three patterns of modi“cation. More speci“cally, modifying 10 characters in the text only impacts 1 or 2 “ngerprints out of 10 with a small probability. This means setting any matching threshold below 8 will guarantee near 100% matching rate. When we increase the length of the change to 50 characters, the results do not change signi“cantly, and still guarantee near perfect matching with thresholds below 7. Finally, we note that multiple small changes (in the third experiment) have the most impact on changing “ngerprint vectors. Even in this case, setting a threshold value around 5 or less provides a near perfect matching rate. Avoiding False Positives. In addition to being robust under modi“cations, we also want “ngerprint vectors to provide a low rate of false positives (where unrelated documents generate matching entries in their vectors). In this section, we evaluate “ngerprint vectors against this metric with simulation on random text documents. In Section 6.2, we present similar tests on real email messages.

Approximate Object Location and Spam Filtering on Peer-to-Peer Systems

Probability of complete match for FP vector size x

1

1

0.9

0.9

0.8

0.8

0.7

0.7

Probability

Probability

Probability of complete match for FP vector size x

0.6 0.5 0.4

modify 10 consecutive characters: 1K sized, analytical 1K sized, simulation 5K sized, analytical 5K sized, simulation

0.3 0.2 0.1 0 0

1

2

3

4

0.6 0.5

modify 50 consecutive characters:

0.4

1K sized, analytical 1K sized, simulation 5K sized, analytical 5K sized, simulation

0.3 0.2 0.1

5

6

7

8

9

13

0

10

0

1

2

Number of Matching Fingerprints

3

4

5

6

7

8

9

10

Number of Matching Fingerprints

Probability of complete match for FP vector size x 1 0.9 0.8

Probability

0.7 0.6 0.5 0.4

modify 25 characters with 5 a group 1K sized, analytical 1K sized, simulation 5K sized, analytical 5K sized, simulation

0.3 0.2 0.1 0 0

1

2

3

4

5

6

7

8

9

10

Number of Matching Fingerprints

Fig. 7. Robustness Test (Experimental and Analytical). The probability of correctly recognizing a document after modi“cation, as a function of threshold. |F V | = 10.

First, we generate 100,000 random text “les and “nd document pairs that match 1 out of 10 “ngerprint entries. This experiment is done for different “le sizes ranging from 1KB to 64KB. Figure 8 shows the resulting false positive rate versus the “le size. While the results for one “ngerprint match are already low, they can be made statistically insigni“cant by increasing the “ngerprint matches threshold (T HRESH) for a •document match.Ž Out of all our tests (5 × 109 pairs for each “le size), less than 25 pairs of “les (“le size > 32K) matched 2 “ngerprints, no pairs of “les matched more than 2 “ngerprints. This result, combined with the robustness result, tells us that on randomized documents, a threshold from 2 to 5 “ngerprints gives us a matching mechanism that is both near-perfect in terms of robustness against small changes and absence of false positives. 6.2

Fingerprint on Real Email

We also repeat the experiments in Section 6.1 on real emails. We collected 29996 total spam email messages from http://www.spamarchive.org. Histogram and CDF representations of their size distribution are shown in Figure 9.

Feng Zhou et al. 10

3

10

4

10

5

10

6

10

7

10

8

10

9

False Positive Test

Distribution of Junk Email Size

% of Emails

1 Matching Fingerprint 2 Matching Fingerprint

Probability of # Matching Fingerprint between a Pair of Documents (* >2 Matching Fingerprint = 0)

3

10

4

10

5

10

35%

100%

28%

80%

21%

60%

14%

40%

7%

20%

0% 0

Document Size (Byte)

2

3

4

5

6

7

8

9

10

11

12

13

14

15 >15

Email Size (KByte)

Fig. 8. False Positives. The probability of two random text “les matching i (i = 1, 2) out of 10 “ngerprint vectors, as a function of “le size. Table 1. Robustness Test on Real Spam Emails. Tested on 3440 modi“ed copies of 39 emails, 5629 copies each. |F V | = 10. THRES Detected Failed Total 3 3356 84 3440 4 3172 268 3440 5 2967 473 3440

1

Cumulative %

Probability

14

Fig. 9. Spam Mail Sizes. Size distribution of the 29996 spam email messages used in our experiments, using both histogram and CDF representations. Table 2. False Positive Test on Real Spam Emails. Tested on 9589(normal) × 14925(spam) pairs. |F V | = 10.

Succ. % 97.56 92.21 86.25

Match FP # of Pairs Probability 1 270 1.89e-6 2 4 2.79e-8 >2 0 0

In order to get an idea of whether small modi“cations on spam email is a common practice of spammers, we used a variant of our “ngerprint techniques to fully categorize the email set for uniqueness. We personally con“rmed the results. We found that, out of all these 29996 junk emails, there are: – 14925 unique junk emails. – 9076 modi“ed copies of 4585 unique ones. – 5630 exact copies of the unique ones. From statistics above, we can see that about 1/3 junk emails have modi“ed version(s), despite that we believe the collectors of the archive have already strive to eliminate duplicates. This means changing each email they sent is really a common technique used by spammers, either to prevent detection or to misdirect the end user. We did the robustness test on 3440 modi“ed copies of 39 most •popularŽ junk emails in the archive, which have 5 − 629 copies each. The standard result is human processed and made accurate. The “ngerprint vector size is set to 10, |F V | = 10. We vary threshold of matching “ngerprint from 3 to 5, and collect the detected and failed number. Table 1 shows the successful detection rate with T HRES = 3, 4, 5 are satisfying. For the false positive test, we collect 9589 normal emails, which is compose of about half from newsgroup posts and half from personal emails of project members. Before doing the experiment, we expect collisions to be more common, due to the use of common words and phrases in objects such as emails. We do a full pair-wise “ngerprint

Approximate Object Location and Spam Filtering on Peer-to-Peer Systems

15

match (vector size 10) between these 14925 unique spam emails and 9589 legitimate email messages. Table 2 shows that only 270 non-spam email messages matched some spam message with 1 out of 10 “ngerprints. If we raise the match threshold T to 2 out of 10 “ngerprints, only 4 matches are found. For match threshold more than 2, no matches are found. We conclude that false positives for threshold value T > 1 are very rare (∼ 10−8 ) even for real text samples. 6.3

Efficient Fingerprint Routing w/ TTLs

We want to explore our “ngerprint routing algorithms in a more realistic context. Specifically, we now consider the additional factor mark rate, which is the portion of all users in the network that actively report a particular spam. A user who •marksŽ a spam message actives publishes this fact, thereby registering that opinion with the network. For example, a 10% mark rate means that 10% of the user population actively marked the same message as spam. To simulate the trade-off between bandwidth usage, •markŽ rate, and search success rate, we simulate the searching of randomly generated “ngerprints on transit-stub networks, and vary the required number of overlay hops to “nd a match, as well as the mark rate. We assume users marking the spam are randomly distributed. With an ef“cient DOLR layer, the more users who mark a document as spam, the fewer number of hops we expect a query to travel before “nding a match. We can set a TTL value on queries to conserve bandwidth while maintaining a reasonably high search success rate. We performed experiments on 8 transit stub topologies of 5000 nodes, latency calibrated such that the network diameter is 400ms. Each Tapestry network has 4096 nodes, and each experiment was repeated with 3 randomized overlay node placements. By aggregating the data from all placements and all topologies, we reduced the standard deviation below 0.02 (0.01 for most data points). The results in Figure 10 show the expected latency and success probability for queries as a function of the number of hops allowed per query (TTL). Since there is a high correlation between the TTL value and the network distance traveled in ms, we plot both the TTL used and the associated network distance. For example, we see that queries with TTL of 2 on these topologies travel a distance of approx. 60ms. Further, at 10% publication rate, we expect those queries to be successful 75% of the time. We note that a Time-to-Live value of 3 overlay hops results in a high probability of “nding an existing document even if it has only been reported by a small portion of the participating nodes (2-5%).

7 Related Work There has been a large amount of recent work on structured peer to peer overlays [18,5,11,15,10,8,4]. Recent work [2] has tried to clarify the interfaces these protocols export to applications, including distributed hash tables (DHTs) and decentralized object location and routing (DOLRs) layers. While our proposal is designed for DOLR systems, it can also be implemented on top of DHTs with minor modi“cations. Furthermore, protocols like Tapestry that use network proximity metrics to constrain network traf“c will bene“t the most from our performance optimizations.

16

Feng Zhou et al. Success Probability vs. Expected Latency and TTL 0

TTL=

1

3

2

Success Probability

0.9 0.8 0.7

Mark Rate 0.5% 1% 2% 3% 4% 5% 10%

0.6 0.5 0.4 0.3 0.2 0.1 0

0

50

100

150

200

250

300

350

400

450

Expected Latency (ms)

Fig. 10. Finding an Ideal TTL. A graph that shows, for a •markedŽ document, the correlation between TTL values on queries, probability of a successful search, and percentage of nodes in the network who •markedŽit.

Recent work [6] discusses the feasibility of doing keyword-based web search in structured P2P networks, which can be thought of as an instantiation of our ADOLR proposal applied to text documents with keywords used as features. Both their scheme and our work use inverted indices of keywords/features assigned to different nodes and maintained using structured overlay location and routing primitives. Finally, this work tries to gauge feasibility, rather than to propose any speci“c implementation. In the context of approximate text addressing, centralized text similarity search is a well-studied problem. Comprehensive discussion can be found in [17]. It includes discussion about using "n-grams" to do similarity search using exact search facility. One speci“c technique within this category [7] forms the basis of our approach of using checksum based “ngerprints. In [1], Broder examined the probability of two different strings colliding to an identical single “ngerprint. In contrast, we focus on the collision probability of entire “ngerprint vectors. In Appendix A, we also consider the probability of changes in a “ngerprint vector under different document modi“cation patterns. Many spam “ltering schemes have been proposed and some deployed. Schemes based on hashing and fuzzy hashes [16,14,3], including our proposal, are collaborative and utilize community consensus to “lter messages. These systems include two main components: one or more hash functions to generate digests of email messages, and a repository of all known digests and whether the corresponding emails are spam. Our system differs from others in this group in that the digest repository is fully decentralized, and queries are deterministic by default (i.e. all existing results will be found no matter where it is). This ensures both scalability and accuracy. Another big family of spam “ltering schemes are machine learning-based [12,9]. These schemes “lter incoming messages based on symptoms or trails of spam emails identi“ed explicitly or implicitly by the training process. They can be personalized

Approximate Object Location and Spam Filtering on Peer-to-Peer Systems

17

according to user preferences and email content and therefore perform well on client machines. However, because the “lters these systems use are only based on per-user local information and do not allow cross-user collaboration, they have dif“culty in identifying new spam emails that are very different from those seen before by the local user.

8 Ongoing and Future Work We have implemented the basic Approximate DOLR and Approximate Text Addressing prototype on a Java implementation of Tapestry, and are exploring additional optimizations and extensions. A prototype of the proposed P2P spam “ltering system, SpamWatch, is implemented and available, including a per-node component implemented as a Tapestry application and the user interface implemented as a Microsoft Outlook plug-in2 . One direction for future work is to deploy SpamWatch as a longrunning service, both to provide a valuable service and also to collect valuable trace data. We are also considering extending the system to handle predicate queries. In conclusion, we proposed the design of an approximate location extension to DOLR systems and described an Approximate Text Addressing facility for text-based objects. We discuss issues of data consistency and performance optimizations in the system design, and present a decentralized spam “ltering system as a key application. We validate our designs via simulation and real data, and show how to tune the “ngerprint vector size and query TTL to improve accuracy, reduce bandwidth usage and query latency, all while keeping a low false positive rate.

References 1. Broder, A. Z. Some applications of rabin•s “ngerprint method. In Sequences II: Methods in Communications, Security, and Computer Science, R. Capocelli, A. D. Santis, and U. Vaccaro, Eds. Springer Verlag, 1993, pp. 143…152. 2. Dabek, F., Zhao, B. Y., Druschel, P., Kubiatowicz, J., and Stoica, I. Towards a common API for structured P2P overlays. In Proceedings of IPTPS (Berkeley, CA, February 2003). 3. Distributed checksum clearinghouse. http://www.rhyolite.com/anti-spam/dcc/. 4. Harvey, N. J. A., Jones, M. B., Saroiu, S., Theimer, M., and Wolman, A. Skipnet: A scalable overlay network with practical locality properties. In Proceedings of USITS (Seattle, WA, March 2003), USENIX. 5. Hildrum, K., Kubiatowicz, J. D., Rao, S., and Zhao, B. Y. Distributed object location in a dynamic network. In Proceedings of ACM SPAA (Winnipeg, Canada, August 2002). 6. Li, J., Loo, B. T., Hellerstein, J., Kaashoek, F., Karger, D. R., and Morris, R. On the feasibility of peer-to-peer web indexing and search. In 2nd International Workshop on Peer-to-Peer Systems (Berkeley, California, 2003). 7. Manber, U. Finding similar “les in a large “le system. In Proceedings of Winter USENIX Conference (1994). 8. Maymounkov, P., and Mazieres, D. Kademlia: A peer-to-peer information system based on the XOR metric. In Proceedings of 1st International Workshop on Peer-to-Peer Systems (IPTPS) (Cambridge, MA, March 2002). 2

Fully functional prototypes of the ATA layer and spam “lter are available for download at http://www.cs.berkeley.edu/˜zf/spamwatch

18

Feng Zhou et al.

9. Mozilla spam “ltering. http://www.mozilla.org/mailnews/spam.html. 10. Ratnasamy, S., Francis, P., Handley, M., Karp, R., and Schenker, S. A scalable contentaddressable network. In Proceedings of SIGCOMM (August 2001). 11. Rowstron, A., and Druschel, P. Pastry: Scalable, distributed object location and routing for large-scale peer-to-peer systems. In Proceedings of IFIP/ACM Middleware 2001 (November 2001). 12. Sahami, M., Dumais, S., Heckerman, D., and Horvitz, E. A bayesian approach to “ltering junk email. In AAAI Workshop on Learning for Text Categorization (Madison, Wisconsin, July 1998). 13. Spamassassin. http://spamassassin.org. 14. Spamnet. http://www.cloudmark.com. 15. Stoica, I., Morris, R., Karger, D., Kaashoek, M. F., and Balakrishnan, H. Chord: A scalable peer-to-peer lookup service for internet applications. In Proceedings of SIGCOMM (August 2001). 16. Vipul•s razor. http://razor.sourceforge.net/. 17. Witten, I. H., Moffat, A., and Bell, T. C. Managing Gigabytes: Compressing and Indexing Documents and Images, second ed. Morgan Kaufmann Publishing, 1999. 18. Zhao, B. Y., Kubiatowicz, J. D., and Joseph, A. D. Tapestry: An infrastructure for faulttolerant wide-area location and routing. Tech. Rep. UCB/CSD-01-1141, U.C. Berkeley, April 2001.

A Analysis of Robustness of Text Fingerprinting Here we give mathematical analysis of how to compute the probability distribution of number of unchanged “ngerprints of a text document after small modi“cations. We de“ne: D : the original document D : the original document after modi“cations L : the document is divided in consecutive substrings of length L characters A : the set of checksums calculated from all substrings in D B : the set of checksums calculated from all substrings in D X : A − B, checksums from D which are not present in checksums of D Y : B − A, checksums from D not present in original checksums of D F P (A) : the “ngerprint vector generated from checksums of D, such that F P (A) ⊆ A, |F P (A)| = N F P (B) : the “ngerprint vector generated from checksums of D , such that F P (B) ⊆ B, |F P (B)| = N |S| : if S is a set or vector, |S| represents the size of S z : |F P (B) − F P (A)|, number of checksums in new “ngerprint vector which are not in the old “ngerprint vector Refer to Figure 11 for an illustration of X, Y , A and B. Let•s de“ne Pr (x) as the probability that x out of N checksums in F P (A) are obsolete, that is, not in B; de“ne Pr (y) as the probability that y out of N checksums in F P (B) are newly generated, that is, not in A. We have:     Pr (x) = Pr (|F P (A) ∩ X| = x) =

|X| x

×



|A| − |X| N −x

|A| N



(1)

Approximate Object Location and Spam Filtering on Peer-to-Peer Systems

19

A B

X

Y L−1 D

d

D’

Fig. 11. Relationship between X, Y , A and B.

Fig. 12. Update d chars.

d L−1

L−1

d

D

D

D’

D’

Fig. 13. Insert d chars.

Fig. 14. Delete d chars.

 Pr (y) = Pr (|F P (B) ∩ Y | = y) =

|Y | y



 ×



|B| − |Y | N −y

|B| N





(2)

If: 1. (N − x) + y < N (that is, x > y): F P (B) is composed of (N − x) checksums from F P (A), y checksums from newly generated set Y , and others from A ∩ B. That is, the y checksums from Y and others from A ∩ B are the new checksums in F P (B) since F P (A). Then, z = N − (N − x) = x. 2. (N − x) + y ≥ N (that is, y ≥ x): F P (B) is composed of y checksums from Y , other checksums from F P (A) − X. That is, the y checksums from Y are new checksums in F P (B) since F P (A). Then, z = y. So, when x > y, z = x; when y ≥ x, z = y. That is, z = max(x, y). Then, Pr (z) = Pr (y = z)

z  i=0

Pr (x = i) + Pr (x = z)

z 

Pr (y = i) − Pr (x = z)Pr (y = z) (3)

i=0

Let•s de“ne P (|F P (A) ∩ F P (B)| ≥ k) to be the probability that at least k checksums are in common between “ngerprint vector of new document and of old document. We have:



N −k

Pr (|F P (A) ∩ F P (B)| ≥ k) = Pr (|F P (B) − F P (A)| ≤ N − k) =

Pr (z = i)

(4)

i=0

Knowing of |X| and |Y |, we can apply results in equation (1)-(3) to equation (4), and then get the probability of the number of unchanged “ngerprints after modi“cation of the document.

20

Feng Zhou et al.

While |X| and |Y | are related to modi“cation pattern, we can further consider how  to get |X| and |Y |. X = Xi and Y = Yi , where Xi and Yi are changes made to checksums because of one modi“cation operation i. We have three types of operations: Update d characters : |Xi | = L − 1 + d, |Yi | = L − 1 + d. This is illustrated in Figure 12. Insert d characters : |Xi | = L − 1, |Yi | = L − 1 + d. This is illustrated in Figure 13. Delete d characters : |Xi | = L − 1 + d, |Yi | = L − 1. This is illustrated in Figure 14. X equals the union of each Xi and Y equals the union of each Yi . So, if there is only one modi“cation, we can exactly compute |X|  and |Y |. If there are more than onemodi“cation, |X| ranges from maxi |Xi | to i |Xi |, |Y | ranges from maxi |Yi | to i |Yi |. We can compute approximate average |X| and |Y | for a speci“c pattern of modi“cation operations according to equations above. Thus, we can use equation (4) to compute the probability distribution of number of unchanged “ngerprints in “ngerprint vector.

Efficient Peer-to-Peer Keyword Searching
Patrick Reynolds and Amin Vahdat Department of Computer Science, Duke University {reynolds,vahdat}@cs.duke.edu

Abstract. The recent file storage applications built on top of peer-to-peer distributed hash tables lack search capabilities. We believe that search is an important part of any document publication system. To that end, we have designed and analyzed a distributed search engine based on a distributed hash table. Our simulation results predict that our search engine can answer an average query in under one second, using under one kilobyte of bandwidth. Keywords: search, distributed hash table, peer-to-peer, Bloom filter, caching

1 Introduction Recent work on distributed hash tables (DHTs) such as Chord [19], CAN [16], and Pastry [17] has addressed some of the scalability and reliability problems that plagued earlier peer-to-peer overlay networks such as Napster [14] and Gnutella [8]. However, the useful keyword searching present in Napster and Gnutella is absent in the DHTs that endeavor to replace them. In this paper, we present a symmetrically distributed peerto-peer search engine based on a DHT and intended to serve DHT-based file storage systems. Applications built using the current generation of DHTs request documents using an opaque key. The means for choosing the key is left for the application built on top of the DHT to determine. For example, the Chord File System, CFS [6], uses hashes of content blocks as keys. Freenet [5, 9], which shares some characteristics of DHTs, uses hashes of filenames as keys. In each case, users must have a single, unique name to retrieve Fig. 1. Distributing an inverted index content. No functionality is provided for keyword across a peer-to-peer network. searches. The system described in this paper provides keyword search functionality for a DHT-based file system or archival storage system, to map keyword queries to the unique routing keys described above. It does so by mapping each keyword to a node in the DHT that will store a list of documents containing that keyword. Figure 1 shows how keywords in the index map into the hash range and, in turn, to nodes in the DHT.


This research is supported in part by the National Science Foundation (EIA-99772879, ITR0082912), Hewlett Packard, IBM, Intel, and Microsoft. Vahdat is also supported by an NSF CAREER award (CCR-9984328), and Reynolds is also supported by an NSF fellowship.

M. Endler and D. Schmidt (Eds.): Middleware 2003, LNCS 2672, pp. 21–40, 2003. c IFIP International Federation for Information Processing 2003


22

Patrick Reynolds and Amin Vahdat

% of searches

35 We believe that end-user latency is the 30 most important performance metric for a 25 search engine. Most end-user latency in a dis20 tributed search engine comes from network 15 transfer times. Thus, minimizing the number 10 of bytes sent and the number of times they are 5 sent is crucial. Both bytes and hops are easy 0 0 2 4 6 8 10 to minimize for queries that can be answered Words per search by a single host. Most queries, however, contain several keywords and must be answered Fig. 2. Number of keywords per search opby several cooperating hosts. Using a trace eration in the IRCache for a ten-day period of 99,405 queries sent through the IRCache in January 2002. proxy system to Web search engines during a ten-day period in January 2002, we determined that 71.5% of queries contain two or more keywords. The entire distribution of keywords per query is shown in Figure 2. Because multiple-keyword queries dominate the search workload, optimizing them is important for end-user performance. This paper focuses on minimizing network traffic for multiple-keyword queries.

1.1 Non-goals One extremely useful feature of distributed hash tables is that they provide a simple service model that hides request routing, churn costs, load balancing, and unavailability. Most DHTs route requests to nodes that can serve them in expected O(lg n) steps, for networks of n hosts. They keep churn costs [11] – the costs associated with managing node joins and departures – logarithmic with the size of the network. Using consistent hashing [10] they divide load roughly evenly among available hosts. Finally, they perform replication to ensure availability even when individual nodes fail. Our design uses a DHT as its base; thus, it does not directly address these issues. 1.2 Overview This paper describes our search model, design, and simulation experiments as follows. In Section 2 we describe several aspects of the peer-to-peer search problem space, along with the parts of the problem space we chose to explore. Section 3 describes our approach to performing peer-to-peer searches efficiently. Section 4 details our simulation environment, and Section 5 describes the simulation results. We present related work in Section 6 and conclude in Section 7.

2 System Model Fundamentally, search is the task of associating keywords with document identifiers and later retrieving document identifiers that match combinations of keywords. Most text searching systems use inverted indices, which map each word found in any document to a list of the documents in which the word appears. Beyond this simple description, many design trade-offs exist. How will the index be partitioned, if at all? Should

Efficient Peer-to-Peer Keyword Searching Node 1

Node 2

Node 3

K1

doc1

doc5

doc8

K2

doc2

doc4

doc9

K3

doc2

doc3

doc7

K4

doc1

doc6

doc9

doc3

doc7

doc8

K5

Horizontal partitioning

Node 1

Node 2

Node 3

K1

doc1

doc5

doc8

K2

doc2

doc4

doc9

K3

doc2

doc3

doc7

K4

doc1

doc6

doc9

doc3

doc7

doc8

K5

23

Vertical partitioning

Fig. 3. A horizontally partitioned index stores part of every keyword match-list on each node, often divided by document identifiers. Here we divide the index into document identifiers 1-3, 4-6, and 7-9. A vertically partitioned index assigns each keyword to a single node.

it be distributed, or would a centralized index suffice? In what order will matching documents be listed? How are document changes reflected in the index? We address these questions below. 2.1 Partitioning Although a sufficiently small index need not be partitioned at all, our target application is a data set large enough to overwhelm the storage and processing capacities of any single node. Thus, some partitioning scheme is required. There are two straightforward partitioning schemes: horizontal and vertical. For each keyword an index stores, it must store a match-list of identifiers for all of the documents containing the keyword. A horizontally partitioned index divides this list among several nodes, either sequentially or by partitioning the document identifier space. Google [3] operates in this manner. A vertically partitioned index assigns each keyword, undivided, to a single node. Figure 3 shows a small sample index partitioned horizontally and vertically, with K1 through K5 representing keywords and doc1 through doc9 representing documents that contain those keywords. A vertically partitioned index minimizes the cost of searches by ensuring that no more than k servers must participate in answering a query containing k keywords. A horizontally partitioned index requires that all nodes be contacted, regardless of the number of keywords in the query. However, horizontal indices partitioned by document identifier can insert or update a document at a single node, while vertically partitioned indices require that up to k servers participate to insert or update a document with k keywords. As long as more servers participate in the overlay than there are keywords associated with an average document, these costs favor vertical partitioning. Furthermore, in file systems, most files change rarely, and those that change often change in bursts and may be removed shortly after creation, allowing us to optimize updates by propagating changes lazily. In archival storage systems, files change rarely if at all. Thus, we believe that queries will outnumber updates for our proposed uses, further increasing the cost advantage for vertically partitioned systems. Vertically partitioned indices send queries to a constant number of hosts, while horizontally partitioned indices must broadcast queries to all nodes. Thus, the throughput of a vertically partitioned index theoretically grows linearly as more nodes are added.

24

Patrick Reynolds and Amin Vahdat

Query throughput in a horizontally partitioned index does not benefit at all from additional nodes. Thus, we chose vertical partitioning for our search engine. 2.2 Centralized or Distributed Organization Google has had great success providing centralized search services for the Web. However, we believe that for peer-to-peer file systems and archival storage networks, a distributed search service is better than a centralized one. First, centralized systems provide a single point of failure. Failures may be network outages; denial-of-service attacks, as plagued several Web sites in February of 2000; or censorship by domestic or foreign authorities. In all such cases, a replicated distributed system may be more robust. Second, many uses of peer-to-peer distributed systems depend on users voluntarily contributing computing resources. A centralized search engine would concentrate both load and trust on a small number of hosts, which is impractical if those hosts are voluntarily contributed by end users. Both centralized and distributed search systems benefit from replication. Replication improves availability and throughput in exchange for additional hardware and update costs. A distributed search engine benefits more from replication, however, because replicas are less susceptible to correlated failures such as attacks or network outages. Distributed replicas may also allow nodes closer to each other or to the client to respond to queries, reducing latency and network traffic. 2.3 Ranking of Results One important feature of search engines is the order in which results are presented to the user. Many documents may match a given set of keywords, but some may be more useful to the end user than others. Google’s PageRank algorithm [15] has successfully exploited the hyperlinked nature of the Web to give high scores to pages linked to by other pages with high scores. Several search engines have successfully used words’ proximity to each other or to the beginning of the page to rank results. Peer-to-peer systems lack the linking structure necessary for PageRank but may be able to take advantage of word position or proximity heuristics. We will discuss specific interactions between ranking techniques and our design in Section 3.5 after we have presented the design. 2.4 Update Discovery A search engine must discover new, removed, or modified documents. Web search engines have traditionally relied on enumerating the entire Web using crawlers, which results in either lag or inefficiency if the frequency of crawling differs from the frequency of updates for a given page. Popular file-sharing systems use a “push” model for updates instead: clients that have new or modified content notify servers directly. Even with pushed updates, the process of determining keywords and reporting them to server should occur automatically to ensure uniformity. The Web could support either crawled or pushed updates. Crawled updates are currently the norm. Peer-to-peer services may lack hyperlinks or any other mechanism

Efficient Peer-to-Peer Keyword Searching

25

for enumeration, leaving them dependent on pushed updates. We believe that pushed updates are superior because they promote both efficiency and currency of index information. 2.5 Placement All storage systems need techniques for placing and finding content. Distributed search systems additionally need techniques for placing index partitions. We use a DHT to map keywords to nodes for the index, and we claim that the placement of content is an orthogonal problem. There is little or no benefit to placing documents and their keywords in the same place. First, very few documents indicated as results for a search are later retrieved; thus, most locality would be wasted. Second, there is no overlap between an index entry and the document it indicates; both still must be retrieved and sent over the network. A search engine is a layer of indirection. It is expected that documents and their keywords may appear in unrelated locations.

3 Efficient Support for Peer-to-Peer Search In the previous section, we discussed the architecture and potential benefits of a fully distributed peer-to-peer search infrastructure. The primary contribution of this work is to demonstrate the feasibility of this approach with respect to individual end user requests. Conducting a search for a single keyword consists of looking up the keyword’s mapping in the index to reveal all of the documents containing that keyword. This involves contacting a single remote server, an operation with network costs comparable to accessing a traditional search service. A boolean “AND” search consists of looking up the sets for each keyword and returning the intersection. As with traditional search engines, we return a small subset of the matching documents. This operation requires contacting multiple peers across the wide area, and the requisite intersection operation across the sets returned by each peer can become prohibitively expensive, both in terms of consumed network bandwidth and the latency incurred from transmitting this data across the wide area. Consider the example in Figure 4(a), which shows a simple network with servers s A and sB . Server sA contains the set of documents A for a given keyword k A , and server sB contains the set of documents B for another keyword k B . |A| and |B| are the number of documents containing k A and kB , respectively. A ∩ B is the set of all documents containing both k A and kB . The primary challenge in performing efficient keyword searches in a distributed inverted index is limiting the amount of bandwidth used for multiple-keyword searches. The naive approach, shown in Figure 4(a), consists of the first server, s A , sending its entire set of matching document IDs, A, to the second server, s B , so that sB can calculate A ∩ B and send the results to the client. This is wasteful because the intersection, A ∩ B, is likely to be far smaller than A, resulting in most of the information in A getting discarded at sB . Furthermore, the size of A (i.e., the number of occurrences of the keyword kA ) scales roughly with the number of documents in the system. Thus, the cost of naive search operations grows linearly with the number of documents in the system. We

26

Patrick Reynolds and Amin Vahdat

B

A 12 34

(2) A

34 56

A

B

12 34

34 56

(2) F(A) 1234

1234 34

6

(3) B ∩ F(A) server sA server sA

server sB

server sB (4) A ∩ B

(1) request

34

(3) A ∩ B

(1) request

34

client

client

(a) A simple approach to “AND” queries. (b) Bloom filters help reduce the bandwidth Each server stores a list of document IDs cor- requirement of “AND” queries. The gray box responding to one keyword. represents the Bloom filter F(A) of the set A. Note the false positive in the set B ∩ F(A) that server sB sends back to server sA . Fig. 4. Network architecture and protocol overview.

propose three techniques to limit wasted bandwidth, to ensure scalability, and to reduce end-client latency: Bloom filters, caches, and incremental results. We discuss each of these approaches in turn and present analytical results showing the potential benefits of each technique under a variety of conditions before exploring these tradeoffs in more detail through simulation in Section 5. 3.1 Bloom Filters A Bloom filter [2, 7, 13] is a hash-based data structure that summarizes membership in a set. By sending a Bloom filter based on A instead of sending A itself, we reduce the amount of communication required for s B to determine A ∩ B. The membership test returns false positives with a tunable, predictable probability and never returns false negatives. Thus, the intersection calculated by s B will contain all of the true intersection, as well as a few hits that contain only k B and not kA . The number of false positives falls exponentially as the size of the Bloom filter increases. Given optimal choice of hash functions, the probability of a false positive is p f p = .6185m/n,

(1)

where m is the number of bits in the Bloom filter and n is the number of elements in the set [7]. Thus, to maintain a fixed probability of false positives, the size of the Bloom filter must be proportional to the number of elements represented. Our method for using Bloom filters to determine remote set intersections is shown in Figure 4(b) and proceeds as follows. A and B are the document sets to intersect, each containing a large number of document IDs for the keywords k A and kB , respectively.

Efficient Peer-to-Peer Keyword Searching

27

The client wishes to retrieve the intersection A ∩ B. Server s A sends a Bloom filter F(A) of set A to server sB . Server sB tests each member of set B for membership in F(A). Server sB sends the matching elements, B ∩ F(A), back to server s A , along with some textual context for each match. Server s A removes the false positives from s B ’s results by calculating A ∩ (B ∩ F(A)), which is equivalent to A ∩ B. False positives in B ∩ F(A) do not affect the correctness of the final intersection but do waste bandwidth. They are eliminated in the final step, when s A intersects B ∩ F(A) against A. It is also possible to send B ∩ F(A) directly from s B to the client rather than first sending it to sA and removing the false positives. Doing so eliminates the smaller transfer and its associated latency at the expense of correctness. Given reasonable values for |A|, |B|, the size of each document record, and the cache hit rate (see Section 3.2), the false-positive rate may be as high as 0.05 or as low as 0.00003. This means that B ∩ F(A) will have from 0.00003|B| to 0.05|B| extra elements that do not contain kA . For example, if 5% of the elements of B actually contain k A , then returning the 0.00003|B| = 0.06% rough intersection B ∩ F(A) to the client results in between (0.05+0.00003)|B| 0.05|B| and (0.05+0.05)|B| = 50% of the results being incorrect and not actually containing k A , where each expression represents the ratio of the number of false positives to the total number of elements in B ∩ F(A). The decision to use this optimization is made at run time, when the parameters are known and p f p can be predicted. Server s A may choose an m value slightly larger than optimal to reduce p f p and improve the likelihood that s B can return B ∩ F(A) directly to the client. The total number of bits sent during the exchange shown in Figure 4(b) is m + p f p |B| j + |A ∩ B| j, where j is the number of bits in each document identifier. For this paper, we assume that document identifiers are 128-bit hashes of document contents; thus, j is 128. The final term, |A ∩ B| j, is the size of the intersection itself. It can be ignored in our optimization, because it represents the resulting intersection, which must be sent regardless of our choice of algorithm. The resulting total number of excess bits sent (i.e., excluding the intersection itself) is

m + p f p|B| j. Substituting for p f p from Equation 1 yields the total number of excess bits as m + .6185m/|A||B| j.

(2)

Taking the first derivative with respect to m and solving for zero yields an optimal Bloom filter size of
 |A| m = |A| log.6185 2.081 . (3) |B| j Figure 5(a) shows the minimum number of excess bits sent for three sets of values for |A|, |B|, and j. The optimal m for any given |A|, |B|, and j is unique and directly determines the minimum number of excess bits sent. For example, when |A| and |B| are 10, 000 and j is 128, m is 85, 734, and the minimum number of excess bits sent is

Patrick Reynolds and Amin Vahdat 50 Excess traffic sent (KB)

50

|A|=10000, |B|=10000, j=128 |A|=2000, |B|=10000, j=128 |A|=10000, |B|=2000, j=128

45 40 35 30 25 20 15 10 5 0

hit rate=0% hit rate=50% hit rate=80% hit rate=90% hit rate=95%

45 Excess traffic sent (KB)

28

40 35 30 25 20 15 10 5

0

5

10 Size of filter, m (KB)

15

20

0

0

5

10

15

20

Size of filter, m (KB)

(a) Expected excess bits sent as a function (b) Improving cache hit rates reduces the of m amount of data sent and increases the size of the optimal Bloom filter. Fig. 5. Effects of Bloom filter size and cache hit rate.

106, 544, representing 12.01 : 1 compression when compared to the cost of sending all 1, 280, 000 bits (10, 000 documents, each with a 128-bit ID) of either A or B. As also shown in Figure 5(a), performance is not symmetric when A and B differ in size. With j constant at 128, the minimum number of excess bits for |A| = 2, 000 and |B| = 10, 000 is 28, 008, lower than the minimum number for |A| = 10, 000 and |B| = 2, 000, which is 73, 046. 28, 008 bits represents 9.14 : 1 compression when compared with the 256, 000 bits needed to send all of A. The server with the smaller set should always initiate the transfer. Our Bloom filter intersection technique can be expanded to arbitrary numbers of keywords. Server s A sends F(A) to server sB , which sends F(B ∩ F(A)) to sC , and so on. The final server, s Z , sends its intersection back to s A . Each server that encoded its transmission using a Bloom filter must process the intersection once more to remove any false positives introduced by its filter. Thus, the intersection is sent to each server except sZ a second time. As above, the expected number of excess bits is minimized when |A| ≤ |B| ≤ |C| ≤ . . . ≤ |Z|. 3.2 Caches Caching can eliminate the need for s A to send A or F(A) if server s B already has A or F(A) stored locally. We derive more benefit from caching Bloom filters than from caching entire document match lists because the smaller size of the Bloom representation means that a cache of fixed size can store data for more keywords. The benefit of caching depends on the presence of locality in the list of words searched for by a user population at any given time. To quantify this intuition, we use the same ten-day IRCache trace described in Section 1 to determine word search popularity. There were a total of 251,768 words searched for across the 99,405 searches, 45,344 of them unique. Keyword popularity roughly followed a Zipf distribution, with the most common keyword searched for 4,365 times. The dominance of popular keywords suggests that even a small cache of either the Bloom filter or the actual document list on A is likely to produce high hit rates.

Efficient Peer-to-Peer Keyword Searching

29

When server sB already has the Bloom filter F(A) in its cache, a search operation for the keywords k A and kB may skip the first step, in which server s A sends its Bloom filter to sB . On average, a Bloom filter will be in another server’s cache with probability r equal to the cache hit rate. The excess bits formula in Equation (2) can be adapted to consider cache hit rate, r, as follows: (1 − r)m + .6185m/|A||B| j Setting the derivative of this with respect to m to zero yields the optimal m as   |A| m = |A| log.6185 (1 − r)2.081 . |B| j

(4)

(5)

Figure 5(b) shows the effect of cache hit rates on the excess bits curves, assuming |A| and |B| are both 10, 000 and j is 128. Each curve still has a unique minimum. For example, when the hit rate, r, is 0.5, the minimum excess number of bits sent is 60, 486, representing 21.16 : 1 compression when compared with sending A or B. Improvements in the cache hit rate always reduce the minimum expected number of excess bits and increase the optimal m. The reduction in the expected number of excess bits sent is nearly linear with improvements in the hit rate. The optimal m increases because as we become less likely to send the Bloom filter, we can increase its size slightly to reduce the false-positive rate. Even with these increases in m, we can store hundreds of cache entries per megabyte of available local storage. We expect such caching to yield high hit rates given even moderate locality in the request stream. Cache consistency is handled with a simple time-to-live field. Updates only occur at a keyword’s primary location, and slightly stale match list information is acceptable, especially given the current state of Internet search services, where some degree of staleness is unavoidable. Thus, more complex consistency protocols should not be necessary. 3.3 Incremental Results Clients rarely need all of the results of a keyword search. By using streaming transfers and returning only the desired number of results, we can greatly reduce the amount of information that needs to be sent. This is, in fact, critical for scalability: the number of results for any given query is roughly proportional to the number of documents in the network. Thus, the bandwidth cost of returning all results to the client will grow linearly with the size of the network. Bloom filters and caches can yield a substantial constant-factor improvement, but neither technique eliminates the linear growth in cost. Truncating the results is the only way to achieve constant cost independent of the number of documents in the network. When a client searches for a fixed number of results, servers s A and sB communicate incrementally until that number is reached. Server s A sends its Bloom filter in chunks and server sB sends a block of results (true intersections and false positives) for each chunk until server s A has enough results to return to the client. Because a single Bloom filter cannot be divided and still retain any meaning, we divide the set A into chunks

30

Patrick Reynolds and Amin Vahdat

and send a full Bloom filter of each chunk. The chunk size can be set adaptively based on how many elements of A are likely to be needed to produce the desired number of results. This protocol is shown in Figure 6. Note that s A and sB overlap their communication: sA sends F(A2 ) as sB sends B∩F(A1 ). This protocol can be extended logically to more than two participants. Chunks are streamed in parallel from server s A to sB , from sB to sC , and so on. The protocol is an incremental version of the multi-server protocol described at the end of Section 3.1. When the system streams data in chunks, A1 caches can store several fractional Bloom filA2 F(A1 ) ters for each keyword rather than storing the A3 B F(A2 ) entire Bloom filter for each keyword. This alF(A3 ) lows servers to retain or discard partial entries in the cache. A server may get a partial cache B ∩ F(A1 ) hit for a given keyword if it needs several B ∩ F(A2 ) B ∩ F(A3 ) chunks but already has some of them stored server sA server sB locally. Storing only a fraction of each keyword’s Bloom filter also reduces the amount A∩B of space in the cache that each keyword conrequest sumes, which increases the expected hit rate. Sending Bloom filters incrementally substantially increases the CPU costs involved in processing a search. The cost for server s B to calculate each intersection B ∩ F(A i ) is the client same as the cost to calculate the entire inter- Fig. 6. Servers sA and sB send their data one section B ∩ F(A) at once because each ele- chunk at a time until the desired intersection ment of B must be tested against each chunk. size is reached. This added cost can be avoided by sending contiguous portions of the hash space in each chunk and indicating to s B which fraction of B (described as a portion of the hash space) it needs to test against F(A). 3.4 Virtual Hosts One key concern in a peer-to-peer system is the inherent heterogeneity of such systems. Randomly distributing functionality (e.g., keywords) across the system runs the risk of assigning a popular keyword to a relatively under-provisioned machine in terms of memory, CPU, or network capacity. Further, no hash function will uniformly distribute functionality across a hash range. Thus, individual machines may be assigned disproportionate numbers of keywords (recall that keywords are assigned to the host whose ID is closest to it in the hash range). Virtual hosts [6] are one technique to address this potential limitation. Using this approach, a node participates in a peer-to-peer system as several logical hosts, proportional to its request processing capacity. A node that participates as several virtual hosts is assigned proportionally more load, addressing heterogeneous node capabilities. Thus, a node with ten times the capacity of some baseline measure would be assigned ten virtual IDs (which means that it is mapped to ten different IDs in the hash range). An optional system-wide scaling factor for each node’s number of virtual hosts further reduces the probability that any single node is

Efficient Peer-to-Peer Keyword Searching

31

assigned a disproportionately large portion of the hash range. This effect is quantified in Section 5, but consider the following example: with 100 hosts of equal power, it is likely that one or more hosts will be assigned significantly more than 1% of the hash range. However, with a scaling factor of 100, it is much less likely that any host will be assigned much more than 1% of the range because an “unlucky” hash (large portion of the hash region) for one virtual host is likely to be canceled out by a “lucky” hash (small portion of the hash region) for another virtual host on the same physical node. 3.5 Discussion Two of the techniques described here, Bloom filters and caching, yield constant-factor improvements in terms of the number of bytes sent and the end-to-end query latency. Bloom filters compress document ID sets by about one order of magnitude, in exchange for either added latency or a configurable probability of false positives. Caching exploits re-referencing and sharing in the query workload to reduce the probability that document ID sets need to be sent. However, even together, these techniques leave both bytes sent and end-to-end query time roughly proportional to the number of documents in the system. The third technique, incremental results, reduces the number of bytes sent and the end-to-end query latency to a constant in most cases. As long as the user wants only a constant number of results, only a constant amount of work will be done, regardless of how many possible results exist in the system. Incremental results yield no improvement in some unusual cases, however. If the user searches for several keywords that are individually popular but mostly uncorrelated in the document space, there may be a small but nonzero number of valid results 1 . If the number of results is nonzero but smaller than the number that the client requests, the system must consider the entire search space, rendering incremental results useless. In cases such as this, the entire search space must be considered, and incremental results will increase, rather than decrease, the number of bytes sent and the end-to-end query latency. However, caching may alleviate the problem if the words used are popular in search queries, and Bloom filters still yield approximately a ten-to-one compression factor. We expect that searches containing popular but uncorrelated keywords will be rare. In our IRCache search trace, most of the queries with small numbers of results had uncommon (often misspelled) keywords. Uncommon keywords—i.e., those with few matching documents—are easy to handle, as discussed in Section 3.1. The system considers the least common keyword first, bounding the maximum size of any intersection set sent for the remainder of the query. 3.6 Ranking of Results Two of our optimization techniques, Bloom filters and incremental results, complicate problem of ranking results. Bloom filters roughly convey membership in a set, but they 1

One example of a difficult search is “OpenBSD birthday pony,” suggested by David Mazi`eres at New York University. In recent Google searches, these three keywords match two million, eight million, and two million documents, respectively. Only fifteen documents contain all three.

32

Patrick Reynolds and Amin Vahdat

do not provide the ability to order set members or to convey additional data with each member, such as a word’s position in a document. The uncompressed response message containing B ∩ F(A) can contain document-ranking or word-position information, which would give server s A enough information to generate rankings based on both keywords, k A and kB . However, in Section 3.1, we suggested eliminating this uncompressed response message. Doing so eliminates the ability to consider k A in any ranking techniques. Incremental results can alleviate the problems with Bloom filters. If each chunk sent contains document IDs with strictly lower rankings than in previous chunks, then the first results returned to the client will be the best, though order within a chunk will not be preserved. However, in Section 3.3 we suggested sending contiguous portions of the hash space in each chunk to save processing time on server s B . These two techniques are mutually exclusive. We believe that ranking documents is more important than eliminating one additional message or saving processing time. However, this trade-off can be determined at run time according to user preference. 3.7 Load Balancing A vertically partitioned index distributes keywords randomly, resulting in a binomial (roughly normal) distribution of the number of keywords on each node. However, keyword appearance popularity (i.e., the size of the keyword’s match-list) and search popularity are both roughly Zipf-distributed. Keyword appearance popularity determines the storage required, and keyword search popularity determines processing loads. Both contribute to network loads. The resulting storage, processing, and network loads are less evenly distributed than with a horizontally partitioned index. Virtual hosts alleviate the problem by assigning larger loads to more capable nodes, but they do not make load any more balanced. Increasing the size of the network and the number of documents results in somewhat more balanced load. As long as the network is over-provisioned, which many peer-to-peer networks are, we believe that load balancing will not be a problem.

4 Simulation Infrastructure The simple analysis described above in Section 3 provides some insight into the potential benefits of our three approaches toward efficiently supporting peer-to-peer search. However, the actual benefits and tradeoffs depend heavily upon target system characteristics and access patterns. To test the validity of our approach under a range of realistic circumstances, we developed a simulation infrastructure implementing our three techniques. In this section, we discuss the details of this simulation infrastructure before presenting the results of our evaluation in Section 5. 4.1 Goals Our goal in writing the simulator was to test the system with a realistic workload and to test the effects of parameters and features that did not lend themselves to tractable analysis. In particular, we tested the effects of the number of hosts in the network, the

Efficient Peer-to-Peer Keyword Searching

33

use of virtual hosts, the Bloom filter threshold, Bloom filter sizes, caching techniques, and the use of incremental results. We also tested the system’s sensitivity to varying network characteristics. The Bloom filter threshold refers to the document set size below which a host transmits a full list rather than a Bloom-compressed set. For small documents, the total bandwidth consumed for transmission to a remote host (for set intersection) may be so small that it may not be worth the CPU time required to compress the set. Eliminating the Bloom step further eliminates the need to return to the transmitting host to eliminate false positives from the intersection. Typically, we find that the extra CPU overhead and network overhead of returning the result is worth the substantial saving in network bandwidth realized by using Bloom filters. In Section 5, we quantify this effect for a variety of Bloom thresholds. Bloom filter sizes affect the number of false positives transmitted during the search process. If the client is willing to accept some probability of false positives (a returned document containing only a subset of the requested keywords), sufficiently large Bloom filters can meet the client’s accepted false-positive rate and eliminate the need to revisit nodes to remove false positives, as described in Section 3.1. That is, small Bloom filters result in significant compression of a keyword-set size at the cost of either generating more false positives in the result returned to the client or requiring the transmission of the intersection back to the originating host for false positive elimination. 4.2 Design The simulator runs as a single-threaded Java application. We implement the inverted index, word-to-host mapping, and host measurement (in this case, random generation) in separate classes so that much of the simulator could be reused in a full implementation of our protocol. Our simulations use a real document set and search trace. The document set totals 1.85 GB of HTML data, comprising 1.17 million unique words in 105,593 documents, retrieved by crawling to a recursion depth of five from 100 seed URLs [4]. The searches performed are read from a list of 95,409 searches containing 45,344 unique keywords. The search trace is the IRCache log file described in Section 1. Note that the results presented in this paper are restricted to these particular traces. However, we do not expect the benefits of our techniques to differ significantly for other workloads. Hosts in the network are generated at random based on configurable distributions for upload speed, download speed, CPU speed, and local storage capacity. We use three distributions for network speeds: one with all modems, one with all backbone links, and one based on the measurements of the Gnutella network performed by Saroiu et al [18]. This last heterogeneous set contains a mixture of modems, broadband connections (cable/DSL) and high-speed LAN connections. Our CPU speed distribution is roughly a bell curve, with a mean of 750 MIPS, and our local storage distribution is a heavytailed piece-wise function ranging from 1 MB to 100 MB. We experimented with a broad range of host characteristics and present the results for this representative subset in this paper. To generate random latencies, we place hosts at random in a 2,500-mile square grid and assume that network packets travel an average of 100,000 miles per second.

34

Patrick Reynolds and Amin Vahdat

The time required to send a network message is the propagation time, as determined by the distance between the hosts involved, plus the transmission time, as determined by the minimum of the sender’s upload speed and the recipient’s download speed, and the size of the packet. The total network time for a search is the sum of the latency and transmission time for all packets sent among server nodes processing the query. We ignore the time spent by the client sending the initial query and receiving the results because these times are constant and independent of any search architecture, whether centralized or distributed. Document IDs are assumed to be 128 bits. The time required to look up words in a local index or perform intersections or Bloom filter operations is based on the CPU speed and the following assumptions for operation costs: 1,500 simple operations per hit to look up words in an index, 500 simple operations per element to intersect two result sets, and 10,000 simple operations per document ID inserted into a Bloom filter or checked against a Bloom filter received from another host. We believe that in general, these assumptions place an upper bound on the CPU cost of these operations. Even with these assumptions, we find that network time typically dominates CPU time for our target scenarios. We determine the number of virtual hosts to assign each simulated node based on its network and CPU speeds when compared to a baseline host. The baseline host has a 57.5 MIPS CPU and 30 Kbit/s network links. These speeds were chosen as those required to compute and transmit 5,000 Bloom operations per second. Each node is compared to the baseline host in three categories: upload speed, download speed, and CPU speed. The nodes’s minimum margin over the baseline host in these three categories is rounded down and taken to be its number of virtual hosts. To perform each query, the simulator looks up each keyword in the inverted index, obtaining up to M results for each, where M is the incremental result size. Each host intersects its set with the data from the previous host and forwards it to the subsequent host, as described in Section 3.1. Each node forwards its current intersected set as either a Bloom filter or a full set, depending on whether or not the set is larger than the Bloom threshold. After each peer performs its part of the intersection, any node that sent a Bloom filter in the first pass is potentially revisited to remove false positives. If the number of resulting documents is at least as large as the the desired number, the search is over. Otherwise, M is increased adaptively to twice what appears to be needed to produce the desired number of results, and the search is rerun. At each step, a host checks its cache to see if it has data for the subsequent host’s document list in its local cache. If so, it performs the subsequent host’s portion of the intersection locally and skips that host in the sending sequence. 4.3 Validation We validated our simulator in two ways. First, we calculated the behavior and performance of short, artificial traces by hand and confirmed that the simulator returns the same results. Second, we varied the Bloom filter size, m, in the simulator and compared the results to the analytical results presented in Section 3.1. The analytical results shown in Figure 5(b) closely resemble the simulated results shown in Figure 9(a).

Efficient Peer-to-Peer Keyword Searching 14

Total network time per query (ms)

Traffic per query (KB)

600

No virtual hosts Virtual hosts, scaling=1 Virtual hosts, scaling=10 Virtual hosts, scaling=100

12 10 8 6 4 2 0

1

10

100

1000

10000

No virtual hosts Virtual hosts, scaling=1 Virtual hosts, scaling=10 Virtual hosts, scaling=100

500 400 300 200 100 0

1

10

Number of hosts

(a) The number of bytes sent increases very little beyond networks of 100 hosts. Enabling virtual hosts reduces the number of bytes sent by about 18%. Scaling the number of virtual hosts reduces the number of bytes sent by an additional 18%.

35

100 Number of hosts

1000

10000

(b) Virtual hosts cut the amount of time spent transmitting by up to 60%. Scaling the number of virtual hosts yields a small additional improvement.

Fig. 7. Network scaling and virtual hosts.

5 Experimental Results The goal of this section is to understand the performance effects of our proposed techniques on a peer-to-peer search infrastructure. Ideally, we wish to demonstrate that our proposed peer-to-peer search system scales with system size (total resource consumption per search grows sub-linearly with the number of participating hosts) and that techniques such as Bloom filters and caching improve the performance of individual requests. Primarily, we focus on the metric of bytes sent per request. Techniques such as caching and the use of Bloom filters largely serve to reduce this metric. Reducing bytes per request has the added benefit of reducing total time spent in the network and hence end-to-end client perceived latency. We also study the effects of the distribution of network and CPU characteristics on overall system performance. One challenge with peer-to-peer systems is addressing the subset of hosts that have significantly less computation power and network bandwidth than is required to support a high-performance search infrastructure. Finally, although we implemented incremental results, we do not present results for this technique here because our target document set is not large enough to return large numbers of hits for most queries. For our workload, this optimization reduces network utilization by at most 30% in the best case. However, we believe this technique will be increasingly valuable as the document space increases in size. 5.1 Scalability and Virtual Hosts A key goal of our work is to demonstrate that a peer-to-peer search infrastructure scales with the number of participating hosts. Unless otherwise specified, the results presented in this section all assume the heterogeneous distribution [18] of per-peer network connectivity and the default distribution of CPU power described in Section 4. Caching and Bloom filters are both initially turned off. As shown in Figure 7(a), increasing the number of hosts in the simulation has little effect on the total number of bytes sent. With

36

Patrick Reynolds and Amin Vahdat

very small networks, several keywords from a query may be located on a single host, resulting in entirely local handling of parts of the query. However, beyond 100 hosts, this probability becomes insignificant, and each n-keyword query must contact n hosts, independent of the size of the system. In addition to demonstrating the scalability of the system, Figures 7(a) and 7(b) also quantify the benefits of the use of virtual hosts in the system. Recall that when virtual hosts are turned on, each node is assigned a number of hosts based on its capacity relative to the predefined baseline described in Section 4. The virtual host scaling factor further multiplies this number of hosts by some constant value to ensure that each physical host is assigned a uniform portion of the overall hash range as discussed in Section 4. Overall, virtual hosts have a small effect on the number of total bytes sent per query. This is because enabling virtual hosts concentrates data mostly on powerful hosts, increasing the probability that parts of a query can be handled entirely locally. Virtual host scaling results in better expected load balancing, which very slightly decreases the amount of data that must be sent on average. Although virtual hosts have little effect on how much data must be sent, they can significantly decrease the amount of time spent sending the data, as shown in Figure 7(b). By assigning more load to more capable hosts, the virtual hosts technique can cut network times by nearly 60%. Using virtual host scaling further decreases expected network times by reducing the probability that a bottleneck host will be assigned a disproportionate amount of load by mistake. Thus, while total bytes sent decreases only slightly as a result of better load balancing, total network time decreases significantly because more capable hosts (with faster network connections) become responsible for a larger fraction of requests. 5.2 Bloom Filters and Caching

Traffic per query (KB)

14 Having established the scalability of our 12 general approach, we now turn our atten10 tion to the additional benefits available from the use of Bloom filters to reduce 8 network utilization. In particular, we fo6 cus on how large the Bloom filter should 4 be and for what minimum data set size 2 it should be invoked. Using Bloom filters 0 1 10 100 1000 10000 100000 for every transfer results in substantial unBloom filter threshhold necessary data transmissions. Any time a Bloom filter is used, the host using it must Fig. 8. Using Bloom filters less often signifilater revisit the same query to eliminate cantly reduces the amount of data sent by elimany false positives. Thus, Bloom filters inating the need to revisit nodes to eliminate should only be used when the time saved false positives. will outweigh the time spent sending the clean-up message. Figure 8 shows the total bytes transmitted per query as a function of the Bloom filter threshold, assuming the default value of 6 bits per Bloom entry. We find that the optimal Bloom filter threshold for our trace was approximately 300. Any set below this size should be sent in its entirety as the savings from using Bloom filters

Efficient Peer-to-Peer Keyword Searching 8

6 5 4 3 2 1 0

No caching, 10% false positives No caching, 1% false positives No caching, 0% false positives Caching, 10% false positives Caching, 1% false positives Caching, 0% false positives

16000 Total network time (s)

Traffic per query (KB)

18000

No caching, 10% false positives No caching, 1% false positives No caching, 0% false positives Caching, 10% false positives Caching, 1% false positives Caching, 0% false positives

7

37

14000 12000 10000 8000 6000 4000 2000

0

5

10

15 20 25 30 Bloom filter size (bits/entry)

(a) Bytes per query

35

40

0

0

5

10 15 20 25 30 Bloom filter size (bits/entry)

35

40

(b) Latency plus transmission time

Fig. 9. Network costs as a function of Bloom filter size.

do not outweigh the network (not to mention latency) overhead of revisiting the host to eliminate false positives. Next, we consider the effects of varying the number of bits per entry in the Bloom filter and of caching on total network traffic. Figure 9(a) plots the total number of bytes transmitted as a function of the Bloom filter size. The two sets of curves represent the case when we enable and disable caching. Within each set, we set a maximum rate of allowable false positives in the set of documents returned to the user for a particular query, at 0%, 1%, and 10%. When the client allows 1% or 10% false positives, false-positive removal steps may sometimes be eliminated; increasing the Bloom filter size enhances this effect. Figure 9(b) shows that allowing false positives has significantly more effect on varying total network time than it does on bytes transferred as it eliminates a number of required message transmissions. The effects of caching shown in Figure 9(a) are similar to those derived analytically in Figure 5(b). Caching decreases the total amount of data sent and increases the optimal Bloom filter size: in this case, from 18 bits per entry to 24 bits per entry. For optimal Bloom filter sizes of 18 and 24 bits per entry in the no-caching and caching cases respectively, our caching technique introduces more than a 50% reduction in the total number of bytes transmitted per query. 5.3 Putting It All Together We now present the end-to-end average query times considering all of our optimizations under a variety of assumed network conditions. We break down this end-to-end time into the three principal components that contribute to end-to-end latency: CPU processing time, network transmission time (bytes transferred divided by the speed of the slower network connection speed of the two communicating peers), and latency (determined by the distance between communicating peers). Recall from Section 4 that we do not measure the time associated with either the client request or the final response as the size of these messages is independent of our optimization techniques. Figure 10 shows three bar charts that break down total end-to-end search time under the three network conditions described in Section 4: WAN, Heterogeneous, and Modem. For each network setting there are four individual bars, representing the effects of virtual hosts on or off and of caching on or off. Each bar is further broken down

38

Patrick Reynolds and Amin Vahdat

Bloom filters, caches, and virtual hosts off caches off, virtual hosts off caches on, virtual hosts off caches off, virtual hosts on caches on, virtual hosts on

Bloom filters, caches, and virtual hosts off caches off, virtual hosts off caches on, virtual hosts off caches off, virtual hosts on caches on, virtual hosts on

Bloom filters, caches, and virtual hosts off caches off, virtual hosts off caches on, virtual hosts off caches off, virtual hosts on caches on, virtual hosts on

Total time per query (ms)

into network transmission time, CPU processing time, and network latency. In the case of an all-modem network, end-to-end query time is dominated by network transmission time. The use of virtual hosts has no effect on query times because the network set is homogeneous. Caching does reduce the network transmission portion by roughly 30%. All queries still manage to complete in 1 second or less because, as shown in Figure 9(a) the use of all our optimizations reduces the total bytes transferred per query to less than 1,000 bytes for our target workload; a 56K modem can transfer 6 KB/sec in the best case. However, our results are limited by the fact that our simulator does not model network contention. In general, we expect the per-query average to be worse than our reported results if any individual node’s network connection becomes saturated. This limitation is significantly mitigated under different network conditions as individual nodes are more likely to have additional bandwidth available and the use of virtual hosts will spread the load to avoid underprovisioned hosts. In the homogeneous WAN case, network time is negligible in all 2100 CPU time cases given the very high transmission 2000 Latency 1900 speeds. The use of caching reduces laTransmission time 1800 tency and CPU time by 48% and 30%, 1700 1600 respectively, by avoiding the need to 1500 calculate and transmit Bloom filters in 1400 1300 the case of a cache hit. Enabling vir1200 tual hosts reduces the CPU time by 1100 1000 concentrating requests on the subset 900 800 of WAN nodes with more CPU pro700 cessing power. Recall that although 600 500 the network is homogeneous in this 400 case we still have heterogeneity in 300 200 CPU processing power as described in 100 Section 4. WAN Heterogeneous Modems Finally, the use of virtual hosts and caching together has the most proFig. 10. Isolating the effects of caching, virtual nounced effect on the heterogeneous hosts, and different network characteristics for opnetwork, together reducing average timal Bloom threshold (300) and Bloom filter sizes per-query response times by 59%. In (18/24 for caching on or off). particular, the use of virtual hosts reduces the network transmission portion of average query response times by 48% by concentrating keywords on the subset of nodes with more network bandwidth. Caching uniformly reduces all aspects of the average query time, in particular reducing the latency components by 47% in each case by eliminating the need for a significant portion of network communication.

6 Related Work Work related to ours can be divided into four categories: the first generation of peerto-peer systems; the second-generation, based on distributed hash tables; Web search

Efficient Peer-to-Peer Keyword Searching

39

engines; and database semijoin reductions. We dealt with DHT-based systems in Section 1. The others, we describe here. The first generation of peer-to-peer systems consists of Napster [14], Gnutella [8], and Freenet [5, 9]. Napster and Gnutella both use searches as their core location determination technique. Napster performs searches centrally on well-known servers that store the metadata, location, and keywords for each document. Gnutella broadcasts search queries to all nodes and allows each node to perform the search in an implementation-specific manner. Yang and Garcia-Molina suggest techniques to reduce the number of nodes contacted in a Gnutella search while preserving the implementation-specific search semantics and a satisfactory number of responses [20]. Freenet provides no search mechanism and depends instead on well-known names and well-known directories of names. Web search engines such as Google [3] operate in a centralized manner. A farm of servers retrieves all reachable content on the Web and builds an inverted index. Another farm of servers performs lookups in this inverted index. When the inverted index is all in one location, multiple-keyword searches can be performed with entirely localarea communication, and the optimizations presented here are not needed. Distributing the index over a wide area provides greater availability than the centralized approach. Because our system can take advantage of the explicit insert operations in peer-to-peer systems, we also provide more up-to-date results than any crawler-based approach can. The general problem of remotely intersecting two sets of document IDs is equivalent to the database problem of performing a remote natural join. We are using two ideas from the database literature. Sending only the data necessary for the intersection (i.e., join) comes from work on semijoin reductions [1]. Using a Bloom filter to summarize the set of document IDs comes from work on Bloom joins [12, 13].

7 Conclusions This paper presents the design and evaluation of a peer-to-peer search infrastructure. In this context we make the following contributions. First, we show that our architecture is scalable; global network state and message traffic grows sub-linearly with increasing network size. Next, relative to a centralized search infrastructure, our approach can maintain high performance and availability in the face of individual failures and performance fluctuations through replication. Finally, through explicit document publishing, our distributed keyword index delivers improved completeness and accuracy relative to traditional spidering techniques. One important consideration in our architecture is reducing the overhead of multikeyword conjunctive searches. We describe and evaluate a number of cooperating techniques—Bloom filters, virtual hosts, caching, and incremental results—that, taken together, reduce both consumed network resources and end-to-end perceived client search latency by an order of magnitude for our target workload.

Acknowledgments We are grateful to Duane Wessels of the IRCache project (supported by NSF grants NCR-9616602 and NCR-9521745) for access to their trace data files. We would also

40

Patrick Reynolds and Amin Vahdat

like to thank Lipyeow Lim for access to the 1.85 GB HTML data set we used for our document trace. Finally, Rebecca Braynard, Jun Yang, and Terence Kelly provided helpful comments on drafts of this paper.

References 1. Philip Bernstein and Dah-Ming Chiu. Using semi-joins to solve relational queries. Journal of the Association for Computing Machinery, 28(1):25–40, January 1981. 2. Burton H. Bloom. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM, 13(7):422–426, 1970. 3. Sergey Brin and Lawrence Page. The anatomy of a large-scale hypertextual web search engine. In 7th International World Wide Web Conference, 1998. 4. Junghoo Cho and Hector Garcia-Molina. The evolution of the web and implications for an incremental crawler. In The VLDB Journal, September 2000. 5. I. Clarke. A distributed decentralised information storage and retrieval system, 1999. 6. Frank Dabek, M. Frans Kaashoek, David Karger, Robert Morris, and Ion Stoica. Wide-area cooperative storage with CFS. In Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP’01), October 2001. 7. Li Fan, Pei Cao, Jussara Almeida, and Andrei Broder. Summary cache: A scalable wide-area web cache sharing protocol. In Proceedings of ACM SIGCOMM’98, pages 254–265, 1998. 8. Gnutella. http://gnutella.wego.com/. 9. T. Hong. Freenet: A distributed anonymous information storage and retrieval system. In ICSI Workshop on Design Issues in Anonymity and Unobservability, 2000. 10. David R. Karger, Eric Lehman, Frank Thomson Leighton, Rina Panigrahy, Matthew S. Levine, and Daniel Lewin. Consistent hashing and random trees: Distributed caching protocols for relieving hot spots on the World Wide Web. In ACM Symposium on Theory of Computing, pages 654–663, 1997. 11. David Liben-Nowell, Hari Balakrishnan, and David Karger. Analysis of the evolution of peer-to-peer systems. In Proceedings of ACM Conference on Principles of Distributed Computing (PODC), 2002. 12. Lothar Mackert and Guy Lohman. R∗ optimizer validation and performance evaluation for local queries. In ACM-SIGMOD Conference on Management of Data, 1986. 13. James Mullin. Optimal semijoins for distributed database systems. IEEE Transactions on Software Engineering, 16(5):558–560, May 1990. 14. Napster. http://www.napster.com/. 15. Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The PageRank citation ranking: Bringing order to the web. Technical report, Stanford University, 1998. 16. Sylvia Ratnasamy, Paul Francis, Mark Handley, Richard Karp, and Scott Shenker. A scalable content-addressable network. In Proceedings of ACM SIGCOMM’01, 2001. 17. Antony Rowstron and Peter Druschel. Storage management and caching in PAST, a largescale, persistent peer-to-peer storage utility. In Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP’01), 2001. 18. Stefan Saroiu, P. Krishna Gummadi, and Steven D. Gribble. A measurement study of peerto-peer file sharing systems. In Proceedings of Multimedia Computing and Networking 2002 (MMCN’02), January 2002. 19. Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek, and Hari Balakrishnan. Chord: A scalable peer-to-peer lookup service for Internet applications. In Proceedings of ACM SIGCOMM’01, 2001. 20. Beverly Yang and Hector Garcia-Molina. Efficient search in peer-to-peer networks. Technical Report 2001-47, Stanford University, October 2001.

NaradaBrokering: A Distributed Middleware Framework and Architecture for Enabling Durable Peer-to-Peer Grids Shrideep Pallickara and Geoffrey Fox Community Grid Labs, Indiana University, 501 N. Morton St, Suite 224 Bloomington, IN-47404. USA {spallick,gcf}@indiana.edu

Abstract. A Peer-to-Peer (P2P) Grid would comprise services that include those of Grids and P2P networks and naturally support environments that have features of both limiting cases. Such a P2P grid integrates the evolving ideas of computational grids, distributed objects, web services, P2P networks and message oriented middleware. In this paper we investigate the architecture, comprising a distributed brokering system that will support such a hybrid environment. Access to services can then be mediated either by the middleware or alternatively by direct P2P interactions between machines.

1

Introduction

The Grid [1-4] has made dramatic progress recently with impressive technology and several large important applications initiated in high-energy physics [5,6], earth science [7,8] and other areas [9,10]. At the same time, there have been equally impressive advances in broadly deployed Internet technology. We can cite the dramatic growth in the use of XML, the “disruptive” impact of peer-to-peer (P2P) approaches [11] that have resulted in a slew of powerful applications, and the more orderly, but still widespread adoption, of a universal Web Service approach to Web based applications [12,13]. There are no crisp definitions of Grids and P2P Networks that allow us to unambiguously discuss their differences and similarities and what it means to integrate them. However these two concepts conjure up stereotype images that can be compared. Taking “extreme” cases, Grids are exemplified by the infrastructure used to allow seamless access to supercomputers and their datasets. P2P technology facilitates sophisticated resource sharing environments between “consenting” peers over the “edges” of the Internet, enabling ad hoc communities of low-end clients to advertise and access resources on communal computers. Each of these examples offers services but they differ in their functionality and style of implementation. The P2P example could involve services to set-up and join peer groups, browse and access files on a peer, or possibly to advertise one’s interest in a particular file. The “classic” grid could support job submittal and status services and access to sophisticated data management systems. Grids typically have structured robust security services while P2P networks can exhibit more intuitive trust mechanisms reminiscent of the “real world”. Grids typically offer robust services that scale well in pre-existing hierarchically arranged organizations. P2P networks are often used when a best effort service is needed in a M. Endler and D. Schmidt (Eds.): Middleware 2003, LNCS 2672, pp. 41–61, 2003. © IFIP International Federation for Information Processing 2003

42

Shrideep Pallickara and Geoffrey Fox

dynamic poorly structured community. If one needs a particular “hot digital recording”, it is not necessary to locate all sources of this, a P2P network needs to search enough plausible resources to ensure that success is statistically guaranteed. On the other hand, a 3D simulation of the universe might need to be carefully scheduled and submitted in a guaranteed fashion to one of the handful of available supercomputers that can support it. There are several attractive features in the P2P model, which motivate the development of hybrid systems. Deployment of P2P systems is entirely user driven, obviating the need for any dedicated management of these systems. Resource discovery and management is an integral part of P2P computing with peers exposing the resources that they are willing to share and the system (sometimes) replicating these resources based on demand. Grids might host different persistent services and they must be able to discover these services and the interfaces they support. Peers can form groups with the fluid group memberships and are thus very relevant for collaboration [14, 15]. This is an area that has been addressed for the Grid in Ref [16] and also in a seminal paper by Foster and collaborators [17] addressing broad support for communities. A P2P Grid would comprise services that include those of Grids and P2P networks while naturally supporting environments that have features of both limiting cases. We can discuss two examples where such a model is naturally applied. In the High Energy Physics data analysis (e-Science [18]) problem discussed in [19], the initial steps are dominated by the systematic analysis of the accelerator data to produce summary events roughly at the level of sets of particles. This Grid-like step is followed by “physics analysis”, which can involve many different studies and much debate between involved physicists regarding the appropriate methods to study the data. Here we see some Grid and some P2P features. As a second example, consider the way one uses the Internet to access information – either news items or multimedia entertainment. Perhaps the large sites like Yahoo, CNN and future digital movie distribution centers have Grid like organization. There are well-defined central repositories and high performance delivery mechanisms involving caching to support access. Security is likely to be strict for premium channels. This structured information is augmented by the P2P mechanisms popularized by Napster with communities sharing MP3 and other treasures in a less organized and controlled fashion. These simple examples suggest that whether for science or commodity communities, information systems should support both Grid and P2P capabilities [20,21]. The proposed P2P grid, which integrates the evolving ideas of computational grids, distributed objects, web services, P2P networks and message oriented middleware, comprises resources such as relatively static clients, high-end resources and a dynamic collection of multiple P2P subsystems. We investigate the architecture, comprising a distributed brokering system that will support such a hybrid environment. Services can be hosted on such a P2P grid with peer groups managed locally and arranged into a global system supported by core servers. Access to services can then be mediated either by the “broker middleware” or alternatively by direct P2P interactions between machines “on the edge”. The relative performance of each approach (which could reflect computer/network cycles as well as the existence of firewalls) would be used in deciding on the implementation to use. Such P2P Grids should seamlessly integrate users to themselves and to resources, which are also linked to each other. We can abstract such environments as a distributed system of “clients” which consist either of “users” or “resources” or proxies thereto. These clients must be linked together in a flexible fault tolerant efficient high performance fashion. The

NaradaBrokering: A Distributed Middleware Framework and Architecture

43

messaging infrastructure linking clients (both users and resources of course) would provide the backbone for the P2P grid. The smallest unit of this messaging infrastructure should be able to intelligently process and route messages while working with multiple underlying communication protocols. We refer to this unit as a broker, where we avoid the use of the term servers to distinguish it clearly from the application servers that would be among the sources/sinks to messages generated within the integrated system. For our purposes (registering, transporting and discovering information), we use the term events/messages interchangeably where events are just messages − typically with time stamps. We may enumerate the following requirements for the messaging infrastructure − 1. Scaling: This is of paramount importance considering the number of devices, clients and services that would be aggregated in the P2P grid. The distributed broker network should scale to support the increase in these aggregated entities. However the addition of brokers to aid the scaling should not degrade performance by increasing communication pathlengths or ineffective bandwidth utilizations between broker nodes within the system. This calls for efficient organization of the broker network to ensure that the aforementioned degradations along with concomitant problems such as increased communication latencies do not take place. 2. Efficient disseminations: The disseminations pertain to routing content, queries, invocations etc. to the relevant destinations in an efficient manner. The routing engine at each broker needs to ensure that the paths traversed within the broker network to reach destinations are along efficient paths that eschew failed broker nodes. 3. Guaranteed delivery mechanisms: This is to ensure persistent delivery and reliable transactions within P2P grid realms. 4. Location independence: To eliminate bandwidth degradations and bottlenecks stemming from entities accessing a certain known broker over and over again to gain access to services, it must be ensured that any broker within the broker network is just as good as the other. Services and functionality would then be accessible from any point within the broker network. 5. Support for P2P interactions: P2P systems tend to be autonomic, obviating the need for dedicated management. P2P systems incorporate sophisticated search and subsequent discovery mechanisms. Support for P2P interactions facilitates access to information resources and services hosted by peers at the “edge” of the network. 6. Interoperate with other messaging clients: Enterprises have several systems that are built around messaging. These clients could be based on enterprise vendors such as IBM’s MQSeries or Microsoft’s MSMQ. Sometimes these would be clients conforming to mature messaging specifications such as the Java Message Service (JMS) [22]. JMS clients, existing in disparate enterprise realms, can utilize the distributed broker network as a JMS provider to communicate with each other. 7. Communication through proxies and firewalls: It is inevitable that the realms we try to federate would be protected by firewalls stopping our elegant application channels dead in their tracks. The messaging infrastructure should thus be able to communicate across firewall, DHCP and NAT boundaries. Sometimes communications would also be through authenticating proxies. 8. Extensible transport framework: Here we consider the communication subsystem, which provides the messaging between the resources and services. Examining the

44

Shrideep Pallickara and Geoffrey Fox

growing power of optical networks we see the increasing universal bandwidth that in fact motivates the thin client and server based application model. However the real world also shows slow networks and links(such as dial-ups), leading to a high fraction of dropped packets. We also see some chaos today in the telecom industry which is stunting, somewhat, the rapid deployment of modern “wired’ (optical) and wireless networks. We suggest that key to future federating infrastructures will be messaging subsystems that manage the communication between external resources, services and clients to achieve the highest possible system performance and reliability. We suggest this problem is sufficiently hard that we only need solve this problem “once” i.e. that all communication – whether TCP/IP, UDP, RTP (A Transport Protocol for Real-Time Applications) [23], RMI, XML/SOAP [24] or you-name-it be handled by a single messaging or event subsystem. 9. Ability to monitor the performance of P2P grid realms: State of the broker network fabric provides a very good indicator of the state of the P2P grid realm. Monitoring the network performance of the connections originating from individual brokers enables us to identify bottlenecks and performance problems, if any, which exist within a P2P grid realm. 10.Security Infrastructure: Since it is entirely conceivable that messages (including queries, invocations and responses) would have to traverse over hops where the underlying communication mechanisms are not necessarily secure, a security infrastructure that relies on message level security needs to be in place. Furthermore, the infrastructure should incorporate an authentication and authorization scheme to ensure restricted access to certain services. The infrastructure must also ensure a secure and efficient distribution of keys to ensure access by authorized clients to content encapsulated in encrypted messages. In this paper we base our investigations on our messaging infrastructure, NaradaBrokering [25-31], which addresses or provides the foundations for the issues discussed above. The remainder of this paper is organized as follows. In section 2.0 we discuss broker network organization, routing of events and support for durable interactions in the NaradaBrokering system. Section 3.0 presents the rationale, and our strategy, to support P2P interactions. Section 4.0 presents an extensible transport framework that addresses the transport issues alluded to earlier. A performance aggregation framework for monitoring and responding to changing network conditions is discussed in Section 5.0. Section 6.0 presents an overview of the message based security framework in the system. Finally, in section 7.0 we present our conclusions and outline future work.

2

NaradaBrokering

To address the issues [31] of scaling, load balancing and failure resiliency, NaradaBrokering is implemented on a network of cooperating brokers. Brokers can run either on separate machines or on clients, whether these clients are associated with users or resources. This network of brokers will need to be dynamic for we need to service the needs of dynamic clients. Communication within NaradaBrokering is asynchronous and the system can be used to support different interactions by encapsulating them in specialized events. Clients reconnecting after prolonged disconnects, connect to the local broker instead of the remote broker that it was last attached to. This eliminates

NaradaBrokering: A Distributed Middleware Framework and Architecture

45

bandwidth degradations caused by heavy concentration of clients from disparate geographic locations accessing a certain known remote broker over and over again. NaradaBrokering goes beyond other operational publish/subscribe systems [32-37] in many (support for JMS, P2P interactions, audio-video conferencing, integrated performance monitoring, communication through firewalls among others) ways. The messaging system must scale over a wide variety of devices − from hand held computers at one end to high performance computers and sensors at the other extreme. We have analyzed the requirements of several Grid services that could be built with this model, including computing and education. Grid Services (including NaradaBrokering) being deployed in the context of Earthquake Science can be found in [29]. NaradaBrokering supports both JMS and JXTA [44] (from juxtaposition), which are publish/subscribe environments with very different interaction models. NaradaBrokering also provides support for legacy RTP clients. 2.1

Broker Organization

Uncontrolled broker and connection additions result in a broker network susceptible to network-partitions and devoid of any logical structure thus making the creation of efficient broker network maps (BNM) an arduous if not impossible task. The lack of this knowledge hampers the development of efficient routing strategies, which exploit the broker topology. Such systems then resort to “flooding” the entire broker network, forcing clients to discard events they are not interested in. To circumvent this, NaradaBrokering incorporates a broker organization protocol, which manages the addition of new brokers and also oversees the initiation of connections between these brokers. In NaradaBrokering we impose a hierarchical structure on the broker network, where a broker is part of a cluster that is part of a super-cluster, which in turn is part of a super-super-cluster and so on. Clusters comprise strongly connected brokers with multiple links to brokers in other clusters, ensuring alternate communication routes during failures. This organization scheme results in “small world networks” [38,39] where the average communication “pathlengths” between brokers increase logarithmically with geometric increases in network size, as opposed to exponential increases in uncontrolled settings. This cluster architecture allows NaradaBrokering to support large heterogeneous client configurations that scale to arbitrary size. Creation of BNMs and the detection of network partitions are easily achieved in this topology. We augment the BNM hosted at individual brokers to reflect the cost associated with traversal over connections, for e.g. intra-cluster communications are faster than inter-cluster communications. The BNM can now be used not only to compute valid paths but also for computing shortest paths. Changes to the network fabric are propagated only to those brokers that have their broker network view altered. Not all changes alter the BNM at a broker and those that do result in updates to the routing caches, containing shortest paths, maintained at individual brokers. 2.2

Dissemination of Events

Every event has an implicit or explicit destination list, comprising clients, associated with it. The brokering system as a whole is responsible for computing broker destina-

46

Shrideep Pallickara and Geoffrey Fox

tions (targets) and ensuring efficient delivery to these targeted brokers en route to the intended client(s). Events as they pass through the broker network are updated to snapshot its dissemination within the network. The event dissemination traces eliminate continuous echoing and in tandem with the BNM –computes shortest paths – at each broker, is used to deploy a near optimal routing solution. The routing is near optimal since for every event the associated targeted brokers are usually the only ones involved in disseminations. Furthermore, every broker, either targeted or en route to one, computes the shortest path to reach target destinations while eschewing links and brokers that have failed or have been failure-suspected. In NaradaBrokering topics could be based on tag-value pairs, Integer and String values. Clients can also specify SQL queries on properties contained in a JMS message. Finally, NaradaBrokering currently incorporates a distributed XML matching engine, which allows clients to specify subscriptions in XPath queries and store advertisements in XML encapsulated events. Real-time XML events are evaluated against the stored XPath subscriptions, while stored XML advertisements are evaluated against a real-time XPath query for discovery purposes. Figures 2 and 3 illustrate some results [14] from our iniPublisher tial research where we studied 22 the message delivery time as a function of load. The results are from a system comprising 22 10 11 k broker processes and 102 clients 12 in the topology outlined in Figure 1. Each broker node process 4 i 5 13 14 1 2 6 is hosted on 1 physical Sun l h 15 3 SPARC Ultra-5 machine (128 MB RAM, 333 MHz), with no 7 j 8 SPARC Ultra-5 machine host9 ing more than one broker node 16 17 m process. The publisher and the 18 measuring subscriber reside on 19 20 the same SPARC Ultra-5 man Measuring 21 chine. In addition to this there Subscriber are 100 subscribing client processes, with 5 client processes attached to every other broker Fig. 1. The NaradaBrokering Test Topology node (broker nodes 22 and 21 do not have any other clients besides the publisher and measuring subscriber respectively) within the system. The 100 client node processes all reside on a SPARC Ultra-60 (512 MB RAM, 360 MHz) machine. The run-time environment for all the broker node and client processes is Solaris JVM (JDK 1.2.1, native threads, JIT). The machines involved in the experiment reside on a 100 Mbps network. We measure the latencies at the client under varying conditions of publish rates, event sizes and matching rates. In most systems where events are continually generated a “typical” client is generally interested in only a small subset of these events. This behavior is captured in the matching rate for a given client. Varying the match-

NaradaBrokering: A Distributed Middleware Framework and Architecture

47

ing rates allows us to perform measurements under conditions of varying selectivity. The 100% case corresponds to systems that would flood the broker network. In systems that resort to flooding (routing a message to every router node) the system performance does not vary with changes in the match rate. Furthermore, in most cases a given message would only be routed to a small set of targeted client nodes. Transit Delays under different matching rates: 22 Brokers 102 Clients Match Rate=100% Match Rate=50% Match Rate=15%

Transit Delay (MilliSeconds) 450 400 350 300 250 200 150 100 50 0

0 100 200 300 450500 400 350400 250300 Publish Rate 500 600 150200 700 0 50100 (Events/sec) Event Size (Bytes)

Transit Delays under different matching rates: 22 Brokers 102 Clients Match Rate=50% Match Rate=33% Match Rate=4% Transit Delay (MilliSeconds) 160 140 120 100 80 60 40 20 0 500 450 400 350 300 0 100 250 200 200 300 150 100 Event Size (Bytes) 400 500 600 7000 50 Publish Rate (Events/sec)

Fig. 2. NaradaBrokering Performance at match Fig. 3. NaradaBrokering Performance at match rates of 100%, 50% and 15% rates of 50%, 33% and 4%

As the results demonstrate, the system performance improves significantly with increasing selectivity from subscribers. The distributed broker network scaled well, with adequate latency, unless the system became saturated at very high publish rates. 2.3

Failures and Recovery

In NaradaBrokering, stable storages existing in parts of the system are responsible for introducing state into the events. The arrival of events at clients advances the state associated with the corresponding clients. Brokers do not keep track of this state and are responsible for ensuring the most efficient routing. Since the brokers are stateless, they can fail and remain failed forever. The guaranteed delivery scheme within NaradaBrokering does not require every broker to have access to a stable store or DBMS. The replication scheme is flexible and easily extensible. Stable storages can be added/removed and the replication scheme can be updated. Stable stores can fail but they do need to recover within a finite amount of time. During these failures the clients that are affected are those that were being serviced by the failed storage. 2.4

JMS Compliance

NaradaBrokering is JMS compliant and provides support not only for JMS clients, but also for replacing single/limited server JMS systems transparently [28] with a distributed NaradaBrokering broker network. Since JMS clients are vendor agnostic, this JMS integration has provided NaradaBrokering with access to a plethora of applications built around JMS, while the integrated JMS solution provides these applications with scaling, availability and dynamic real time load balancing. Among the applica-

48

Shrideep Pallickara and Geoffrey Fox

tions ported to this solution are the Anabas distance education conferencing system [40] and the Online Knowledge Center (OKC) portal [41]. 2.4.1 JMS Performance Data To gather performance data, we run an instance of the SonicMQ (version 3.0) [42] broker and NaradaBrokering broker on the same dual CPU (Pentium-3, 1 GHz, 256MB) machine. We then setup 100 subscribers over 10 different JMS TopicConnections on another dual CPU (Pentium-3, 866MHz, 256MB) machine. There is also a measuring subscriber and a publisher that are set up on a third dual CPU (Pentium 3, 866MHz, 256MB RAM) machine. The three machines (residing on a 100 Mbps network) have Linux (version 2.2.16) as their operating system. The runtime environment for all the processes is Java 2 JRE ( Blackdown-FCS). Transit Delays for Message Samples in NaradaBr NaradaBrokering & SonicMQ SonicMQ

Mean Transit Delay (MilliSeconds) 30 25 20 15 10 5 0

550 500 450 400 350 0 50 300 250 Payload Size 100150 200 200250 150 (Bytes) 100 Publish Rate 300350400 45050 (Messages/sec)

Fig. 4. Transit Delays for messages

Standard Deviation in the Message Samples NaradaBrokering and SonicMQ NaradaBr SonicMQ

Standard Deviation (MilliSeconds) 14 12 10 8 6 4 2 0

550 500 450 400 350 0 50 300 250 Payload Size 100150 200 200250 150 (Bytes) 100 Publish Rate 300350400 45050 (Messages/sec)

Fig. 5. Standard Deviation for messages

The topic, which the subscribers subscribe to and the publisher publishes to, is the same. We vary the rates at which the publisher publishes messages while varying the payload sizes associated with these messages. We compute the transit delays associated with individual messages and also the standard deviation in the delays (used to compute the mean transit delay) associated with messages in a given test case. Figure 4 depicts the mean transit delays for the measuring subscriber under NaradaBrokering and SonicMQ for high publish rates and smaller payload sizes. Figure 5 depicts the standard deviation associated with message samples under the same conditions. As can be seen NaradaBrokering compares very well with SonicMQ. Also, the standard deviation associated with message samples in NaradaBrokering were for the most part lower than in SonicMQ. Additional results can be found in [28].

3

Support for P2P Interactions in NaradaBrokering

Issues in P2P systems pertaining to the discovery of services and intelligent routing can be addressed very well in the NaradaBrokering system. The broker network would be used primarily as a delivery engine, and a pretty efficient one at that, while locating peers and propagating interactions to relevant peers. The most important aspect in P2P systems is the satisfaction of peer requests and discovery of peers and associated resources that could handle these requests. The broker network forwards

NaradaBrokering: A Distributed Middleware Framework and Architecture

49

these requests only to those peers that it believes can handle the requests. Peer interactions in most P2P systems are achieved through XML-based data interchange. XML’s data description and encapsulation properties provide easy access to specific elements of data. Individual brokers routing interactions could access relevant elements, cache this information and use it subsequently to achieve the best possible routing characteristics. The brokering system, since it is aware of advertisements, can also act as a hub for search and discovery operations. These advertisements when organized into “queryspaces” allow the integrated system to respond to search operations more efficiently. Resources in NaradaBrokering are generally within the purview of the broker network. P2P systems replicate resources in an ad hoc fashion, the availability of which is dependent on the peer’s active digital presence. Some resources, however, are best managed by the brokering system rather than being left to the discretion of peers who may or may not be present at any given time. An understanding of the network topology and an ability to pin point the existence of peers interested in that resource are paramount for managing the efficient replications of a resource. The distributed broker network, possessing this knowledge, best handles this management of resources while ensuring that these replicated resources are “closer” and “available” at locations with a high interest in that resource. Furthermore, the broker network is also better suited, than a collection of peers, to eliminate race conditions and deadlocks that could exist due to a resource being accessed simultaneously by multiple peers. The broker network can also be responsive to changes in peer concentrations, volumes of peer requests, and resource availability. There are also some issues that need to be addressed while incorporating support for P2P interactions. P2P interactions are self-attenuating with interactions dying out after a certain number of hops. These attenuations in tandem with traces of the peers, which the interactions have passed through, eliminate the continuous echoing problem that result from loops in peer connectivity. However, attenuation of interactions sometimes prevents peers from discovering certain services that are being offered. This results in P2P interactions being very “localized”. These attenuations thus mean that the P2P world is inevitably fragmented into many small subnets that are not connected. Furthermore, sophisticated routing schemes are seldom in place and interactions are primarily through simple forwarding of requests with the propagation range determined by the attenuation indicated in the message. NaradaBrokering could also be used to connect islands of peers together. Peers that are not directly connected through the peer network could be indirectly connected through the broker network. Peer interactions and resources in the P2P model are traditionally unreliable, with interactions being lost or discarded due to peer failures or absences, overloading of peers and queuing thresholds being reached. Guaranteed delivery properties existing in NaradaBrokering can augment peer behavior to provide a notion of reliable peers, interactions and resources. Such an integrated brokering solution would also allow for hybrid interaction schemes to exist alongside each other. Applications could be built around hybrid-clients that would exhibit part peer behavior and part traditional client behavior (e.g. JMS). P2P communications could be then used for traffic where loss of information can be sustained. Similarly, hybrid-clients needing to communicate with each other in a “reliable” fashion could utilize the brokering system’s capabilities to achieve that. Sometimes, hybrid-clients satisfy each other’s requests, obviating the need for funneling interactions through the broker network. Systems tuned towards large-scale P2P systems include

50

Shrideep Pallickara and Geoffrey Fox

Pastry [43] from Microsoft, which provides an efficient location and routing substrate for wide-area P2P applications. Pastry provides a self-stabilizing infrastructure that adapts to the arrival, departure and failure of nodes. The JXTA [44] project at Sun Microsystems is another effort to provide such large-scale P2P infrastructures. 3.1

JXTA

JXTA is a set of open, generalized protocols [45] to support P2P interactions and core P2P capabilities such as indexing, file sharing, searching, peer grouping and security. The JXTA peers, and rendezvous peers (specialized routers), rely on a simple forwarding of interactions for dissemination. Time-to-live (TTL) indicators and peer traces attenuate interaction propagations. JXTA interactions are unreliable and tend to be localized. It is expected that existing P2P systems would either support JXTA or have bridges initiated to it from JXTA. Support for JXTA would thus enable us to leverage other P2P systems along with applications built around those systems. 3.2

JXTA & NaradaBrokering

In our strategy for providing support for P2P interactions within NaradaBrokering, we impose two constraints. First, we make no changes to the JXTA core and the associated protocols. We make additions to the rendezvous layer for integration purposes. Second, this integration should entail neither any changes to the peers nor a straitjacketing of the interactions that these peers could have had prior to the integration. The integration is based on the proxy model, which essentially acts as the bridge between the NaradaBrokering system and JXTA. The Narada-JXTA proxy, operating inside the JXTA rendezvous layer, serves in a dual role as both a rendezvous peer and as a NaradaBrokering client providing a bridge between NaradaBrokering and JXTA. NaradaBrokering could be viewed as a service by JXTA. The discovery of this service is automatic and instantaneous due to the Narada-JXTA proxy’s integration inside the rendezvous layer. Any peer can utilize NaradaBrokering as a service so long as it is connected to a Narada-JXTA proxy. Nevertheless, peers do not know that the broker network is routing some of their interactions. Furthermore, these NaradaJXTA proxies, since they are configured as clients within the NaradaBrokering system, inherit all the guarantees that are provided to NaradaBrokering clients. 3.2.1 The Interaction Model Different JXTA interactions are queued at the queues associated with the relevant layers comprising the JXTA protocol suite. Each layer performs some operations including the addition of additional information. The rendezvous layer processes information arriving at its input queues from the peer-resolving layer and the pipebinding layer. Since the payload structure associated with different interactions is different we can easily identify the interaction types associated with the payloads. Interactions pertaining to discovery/search or communications within a peer group would be serviced both by JXTA rendezvous peers and also by Narada-JXTA proxies. Interactions that peers have with the Narada-JXTA proxies are what are routed through the NaradaBrokering system. JXTA peers can continue to interact with each

NaradaBrokering: A Distributed Middleware Framework and Architecture

51

other and of course some of these peers can be connected to pure JXTA rendezvous peers. Peers have multiple routes to reach each other and some of these could include the NaradaBrokering system and some of them need not. Such peers can interact directly with each other during the request/response interactions. 3.2.2 Interaction Disseminations Peers can create a peer group; request to be part of a peer group; perform search/request/discovery all with respect to a specific targeted peer group. Peers always issue requests/responses to a specific peer group and sometimes to a specific peer. Peers and peer groups are identified by UUID [46] (IETF specification guarantees uniqueness until 3040 A.D.) based identifiers. Every peer generates its own peer id while the peer that created the peer group generates the associated peer group id. Each rendezvous peer keeps track of multiple peer groups through peer group advertisements that it receives and is responsible for forwarding interactions. Narada-JXTA proxies are initialized both as rendezvous peers and also as NaradaBrokering clients. During its initialization as a NaradaBrokering client every proxy is assigned a unique connection ID by the NaradaBrokering system, after which the proxy subscribes to a topic identifying itself as a Narada-JXTA proxy. This enables NaradaBrokering to be aware of all the Narada-JXTA proxies that are present in the system. The Narada-JXTA proxy in its role as a rendezvous peer to peers receives – 1) Peer group advertisements 2) Requests from peers to be part of a certain peer group and responses to these requests 3) Messages sent to a certain peer group or a targeted peer 4) Queries and responses to these queries To ensure the efficient dissemination of interactions, it is important to ensure that JXTA interactions that are routed by NaradaBrokering are delivered only to those Narada-JXTA proxies that should receive them. This entails that the Narada-JXTA proxy perform a sequence of operations, based on the interactions that it receives, to ensure selective delivery. The set of operations that the Narada-JXTA proxy performs comprise gleaning relevant information from JXTA’s XML encapsulated interactions, constructing an event based on the information gleaned and finally in its role as a NaradaBrokering client subscribing (if it chooses to do so) to a topic to facilitate selective delivery. By subscribing to relevant topics, and creating events targeted to specific topics each proxy ensures that the broker network is not flooded with interactions routed by them. The events constructed by the Narada-JXTA proxies include the entire interaction as the event’s payload. Upon receipt at a proxy, this payload is deserialized and the interaction is propagated as outlined in the proxy’s dual role as a rendezvous peer. Additional details pertaining to this integration can be found in [27]. 3.3

Performance Measurements

For comparing JXTA performance in NaradaBrokering we setup the topologies depicted in Figure 6. We then compare the performance of the pure JXTA environment, the integrated Narada-JXTA system and the native NaradaBrokering system. The rendezvous peers connected to brokers in topology 6.(b) are Narada-JXTA proxies.

52

Shrideep Pallickara and Geoffrey Fox

To compute communication delays R R while obviating the need for clock synchronizations and the need to R R account for clock drifts, the reN NaradaBrokering broker R R ceiver/sender pair is setup on the (a) R JXTA Rendezvous same machine (Pentium-3, 1 GHz, 256 MB RAM). In all the test cases, JXTA Peer R R a message published by the sender is NaradaBrokering client received at the receiver and the delay R N N R is computed. For a given message payload this is done for a sample of R R messages and we compute the mean (b) delay and the standard deviation N N associated with the samples. This is N N repeated for different payload sizes. For every topology every node (broN N ker or rendezvous peer) involved in (c) the experimental setup is hosted on a different machine (Pentium-3, 1 GHz, 256MB RAM). The run-time Fig. 6. The JXTA Test Topologies environment for all the processes is (JDK-1.3 build Blackdown-1.3.1, Red Hat Linux 7.3). The machines involved in the experimental setup reside on a 100 Mbps LAN. Figures 7 and 8 depict the mean transit delay and standard deviation for the message samples under the different test topologies. These results indicate the superior performance of the integrated NaradaJXTA system compared to that of the pure JXTA system. The results [27] follow the same general pattern for measurements under other test topologies.

Fig. 7. Mean Transit Delay for samples

4

Fig. 8. Standard Deviation for samples

NaradaBrokering’s Transport Framework

In the distributed NaradaBrokering setting it is expected that when an event traverses an end-to-end channel across multiple broker hops or links the underlying transport protocols deployed for communications would vary. The NaradaBrokering Transport

NaradaBrokering: A Distributed Middleware Framework and Architecture

53

framework aims to abstract the operations that need to be supported for enabling efficient communications between nodes. These include support for − 1) 2) 3) 4) 5)

Easy addition of transport protocols within the framework. Deployments of specialized links to deal with specific data types. Negotiation of the best available communication protocol between two nodes Adaptability in communications by responding to changing network conditions. Accumulating performance data measured by different underlying protocol implementations.

TCP, UDP, Multicast, SSL, HTTP and RTP based implementations of the transport framework are currently available in NaradaBrokering. It is also entirely conceivable that there could be a JXTA link, which will defer communications to the underlying JXTA pipe mechanism. NaradaBrokering can also tunnel through firewalls such as Microsoft’s ISA [47] and Checkpoint [48] and proxies such as iPlanet [49]. The user authentication modes supported include Basic, Digest and NTLM. Operations that need to be supported between two communication endpoints are encapsulated within the “link” primitive in the transport framework. The adaptability in communications is achieved by specifying network constraints and conditions under which to migrate to another underlying protocol. For e.g. a UDP link may specify that when the loss rates increase substantially communication should revert to TCP. Though there is support for this adaptability in the transport framework, this feature is not yet implemented in the current release. Figure 9 provides an overview of the NaradaBrokering transport framework. Negotiated with info exchanged over Administrative Link Optimal Transport

Broker node

Administrative Link (HTTP)

Broker node

Alternate Link

(Application and Content Dependent)

Transport Interfaces

Transport Interfaces

Transport Handler Sp

ec ific to

Link Factory

a

tra

ns po rt

Links

Link Factory

Link ors onit Performance M k Lin Data

Data accumulated by Monitoring Service

Fig. 9. Transport Framework Overview

Monitoring Service

54

Shrideep Pallickara and Geoffrey Fox

A Link is an abstraction that hides details pertaining to communications. A Link has features, which allow it to specify a change in the underlying communications and the conditions under which to do so. An implementation of the Link interface can incorporate its own handshaking protocols for setting up communications. The Link also contains methods, which allow for checking the status of the underlying communication mechanism at specified intervals while reporting communication losses to the relevant error handlers within the transport framework. Each implementation of the Link interface can expose and measure a set of performance factors. Measurement of performance factors over a link requires cooperation from the other end-point of the communication link; this particular detail should be handled within the Link implementation itself. How the Link implementation computes round trip delays, jitter factors, bandwidth, loss rates etc. should be within the domain of the implementer. The Link also has methods which enable/disable the measurement of these performance factors. Links expose the performance related information in the LinkPerformanceData construct using which it is possible to retrieve information (type, value, description) pertaining to the performance factors being measured. In the distributed NaradaBrokering setting it is expected that when an event traverses across multiple broker hops it could be sent over multiple communication links. In places where links optimized to deal with the specialized communication needs of the event exist (or can exist) they will be used for communications. While routing events between two NaradaBrokering brokers (that already have a link established between them) it should be possible for the event routing protocol to specify the creation of alternate communication links for disseminations. Support for this feature arises when routing handlers request the deployment of specific transport protocols for routing content, for e.g. a NaradaRTP event router could request that RTP links be used for communication. Sometimes such links will be needed for short durations of time. In such cases one should be able to specify the time for which the link should be kept alive. Expiry of this timer should cause the garbage collection of all resources associated with the link. The keepalive time corresponds to the period of inactivity after which the associated link resources must be garbage collected. All broker locations need not have support for all types of communication links. Information regarding the availability of a specific link type could be encapsulated in an URI. This information could be exchanged along with the information regarding supported link types (at a given node) exchanged over the AdministrativeLink, which is different from that of a link in the methods that can be invoked on it. This URI could then possibly be used to dynamically load services. The AdministrativeLink exchanges information regarding the various communication protocols (along with information pertaining to them such as server, port, multicast group etc) that are available at a broker/client node. This is then used to determine the best link to use to communicate with the broker. Communication over the AdministrativeLink will be HTTP based to ensure the best possibility for communications between two nodes. All link implementations need to have an implementation of the LinkNegotiator interface. Based on the information returned on the AdministrativeLink, the LinkNegotiators are initialized for the common subset of communications and then deployed to negotiate the transport protocol for communications. The LinkNegotiator determines whether communication is possible over a specified link and also returns metrics that would enable the AdministrativeLink in arriving at a decision regarding the deployment of the best possible link.

NaradaBrokering: A Distributed Middleware Framework and Architecture

55

All links of a specific communications type are managed by a LinkFactory instance. The LinkFactory for a particular communications protocol enables communications to and from other nodes over a specific link type. The LinkFactory also controls the intervals at which all its managed links check their communication status. Links also allow the specification of constraints (usually on the set of performance factors that it measures) and the link type that the communication must migrate to when those conditions are satisfied. This feature allows a link to revert to an alternate underlying transport protocol when communication degrades or is impossible to achieve. For example, it is conceivable that while communicating using TCP, bandwidth and latency constraints force a switch to UDP communications. The LinkFactory is also used to manage the migration of communication protocols from links of different types. Based on the set of supported communication protocol migrations, which a LinkFactory exposes, adaptive communications between nodes is enabled. Protocol layers use the TransportHandler interface to invoke methods for communications with other NaradaBrokering nodes. LinkFactories are loaded at run-time by the TransportHandler implementation and it is then that TransportHandler interface is passed to the LinkFactory implementation. The reference to the transport handler is passed to every link created by the link factory. This is the reference that is used by individual links to report the availability of data on a link. Individual links use this interface to report data streams that are received over the link, loss of communications and requests to migrate transport protocols if the migration constraint is satisfied. Based on the LinkFactories that are loaded at run-time the transport handler can expose the set of link types (generally corresponding to transport types) that it supports. Transport Handler manages all Link factories and Links. LinkFactories are responsible for the creation of links. Links have methods for sending data (while also indicating the data type). Data received on a communication link is reported to the TransportHandler by invoking the appropriate methods within the interface.

Fig. 10. Transit Delay for message samples

4.1

Fig. 11. Standard deviation for samples

Some Performance Measurements

Figures 10 and 11 depict results for the TCP implementation of the framework. The graphs depict the mean transit delays, and the accompanying standard deviations, for native NaradaBrokering messages traversing through multiple (2, 3, 5 and 7) hops with multiple brokers (1, 2, 4 and 6 respectively) in the path from the sender of the message to the receiver. For each test case the message payload was varied. The tran-

56

Shrideep Pallickara and Geoffrey Fox

sit delay plotted is the average of the 50 messages that were published for each payload. The sender/receiver pair along with every broker involved in the test cases were hosted on different physical machines (Pentium-3, 1 GHz, 256 MB RAM). The machines reside on a 100 Mbps LAN. The run-time environment for all the processes is JRE-1.3 build Blackdown-1.3.1, Red Hat Linux 7.3 The average delay per inter-node (broker-broker, broker-client) hop was around 500-700 microseconds. The standard deviation varies from 0 microseconds for 50 byte messages traversing a hop to 800 microseconds over 7 hops.

5

Performance Monitoring and Aggregation

The performance monitoring scheme within the distributed broker network needs to have two important characteristics. First, it should be able to work with different transport protocols with no straitjacketing Broker Broker of the performance Monitoring Node Node Service factors being measured. The Link and LinkPerprimiformanceData Link Link tives that abstract Data Data transport details and performance data reAggregates info Control Message spectively, as outlined from nodes in a Exchange in the preceding seccertain domain tion, ensure the ability Performance Aggregation to work with unlimited Service performance factors over different transport Fig. 12. Performance Aggregation Overview protocols. Different nodes, with different types of links originating from them, can end up measuring a different set of performance factors. Second, the scheme should be to federate with other network measurement services such as the network weather service (NWS) [50]. An added feature would be to allow administrators to monitor specific realms or domains. Every broker in NaradaBrokering incorporates a monitoring service (as shown in Figure 12) that monitors the state of the links originating from the broker node. Metrics computed and reported over individual links, originating from a broker node, include bandwidth, jitter, transit delays, loss rates and system throughputs. Factors are measured in a non-intrusive way so as to ensure that the measurements do not further degrade the metrics being measured in the first place. Factors such as bandwidth measurements, which can pollute other metrics being measured, are measured at lesser frequencies. Furthermore, once a link is deemed to be at the extreme ends of the performance spectrum (either very good or very bad) the measurement of certain factors are turned off while others are measured at a far lower frequency. Each link can measure different set of parameters. So the set of parameters being measured would be extensible and flexible. The monitoring service that runs at every node encapsulates performance data gathered from each link in an XML structure. The moni-

NaradaBrokering: A Distributed Middleware Framework and Architecture

57

toring service then reports this data to a performance aggregator node, which aggregates information from monitoring services running at other nodes. Performance aggregators monitor the state of the network fabric at certain realms; the aggregators themselves may exchange information with each other to provide a state of the integrated network realm. The performance aggregators exchange information with the monitoring services pertaining to the measurement and reporting of performance factors. For example, the aggregator can instruct the monitoring service running at a broker node to stop (or modify the intervals between) the measurement of certain factors. Similarly, an aggregator may instruct the monitoring service to report only certain performance factors and that too, only if the factors have varied by the amount (absolute value or a percentage) specified in it’s request. Information accumulated within the aggregators is accessible to administrators via a portlet residing in a portal such as Apache Jetspeed [51]. Note that, since the information returned to the aggregators in encapsulated in an XML structure, it is very easy to incorporate results gathered from another network monitoring service such as NWS. All that needs to be done is to have a proxy, residing at a NWS node that encapsulates the monitored data into an XML structure. The aggregated XML performance data (from the monitoring service at each node and other third-party services) would be mined to generate information, which would then be used to achieve to certain objectives. (a) The ability to identify, circumvent, project and prevent system bottlenecks: Different transports would reveal this in different ways. As system performance degrades UDP loss rates may increase, TCP latencies increase. Similarly as available bandwidths decrease the overheads associated with TCP error correction and in order delivery may become unacceptable for certain applications. (b) To aid routing algorithms: Costs associated with link traversals in BNM's would be updated to reflect the state of the fabric and the traversal times associated with links in certain realms. Routes computed based on this information would then reveal "true" faster routes. (c) To be used for Dynamic topologies to address both (a) and (b): The aggregated performance information would be used to identify locations to upgrade the network fabric of the messaging infrastructure. This upgrade would involve brokers/connections be instantiated/purged dynamically to assuage system bottlenecks and to facilitate better routing characteristics. Although multicasting and bandwidth reservation protocols such as RSVP [52] and ST-II [53] can help in better utilizing the network they require support at the router level, more conceited effort is need at higher levels, and dynamic topologies coupled with efficient routing protocols can help in the efficient utilization of network resources. (d) To determine the best available broker to connect to: Based on the aggregated information it should be possible to determine the best broker that a client can connect to within a certain realm. Scaling algorithms, such as the one derived from item (c), would benefit greatly from this strategy by incorporating newly added broker nodes (which would be the best available ones) into the routing solution. (e) Threshold notifications: Administrators can specify thresholds, which when reached by specific monitored factors, results in notifications being sent to them.

58

Shrideep Pallickara and Geoffrey Fox

6

Security Framework

Since it is entirely conceivable that messages (including queries, invocations and responses) would have to traverse over hops where the underlying communication mechanisms are not necessarily secure, a security infrastructure that relies on message level security needs to be in place. The security framework in NaradaBrokering tries to address the following issues 1. Authentication: Confirm whether a user is really who he says he is. 2. Authorization: Identify if the user is authorized to receive certain events 3. Key distribution: Based on the authentication and authorization, distribute keys, which ensure that only the valid clients are able to decrypt encrypted data. 4. Digital Signing: Have the ability to verify the source of the event and whether the source is authorized to publish events conforming to the specified template. 5. Communication Protocol Independence: Have the ability to work over normal communication channels. Communications need not to be over unencrypted links. 6. End-to-End integrity: Ensure that the only place where the unencrypted event is seen at the authorized publisher of the event and the authenticated (and authorized) subscribers to the event. 7. Detection of security compromise: Check whether the publisher’s signature is a valid one. This approach would be similar to the Certificate Revocation Lists (CRL) scheme. 8. Qualities of Service detecting compromise: Clients may be asked to answer questions to verify its authenticity at regular intervals to facilitate detection of compromise. 9. Response to security compromise: This would involve invalidating certain signatures and discarding the use of certain keys for encrypted communications. In our approach we secure messages independently of any transport level security. This provides a fine-grained security structure suitable for distributed systems and multiple security roles. For example, parts of the message may be encrypted differently, allowing users with different access privileges to access different parts of the message. Basic security operations such as authentication should be performed in a mechanism-independent way, with specific mechanisms (Kerberos [54], PKI) plugged into specific applications. The message level security framework allows us to deploy communication links where data is not encrypted. Furthermore, this scheme also ensures that no node/unauthorized-entity ever sees the unencrypted message. In our strategy we incorporate schemes to detect and respond to security compromises while also dealing with various attack scenarios. Security specifications for Web Services [55, 56] are just starting to emerge, but generally follow the same approach: the message creator adds a signed XML message containing security statements to the SOAP envelope. The message consumer must be able to check these statements and the associated signature before deciding if it can execute the request. Legion (http://www.cs.virginia.edu/ ~legion/) is a long-standing research project for building a “virtual computer” out of distributed objects running on various computing resources. Legion objects communicate within a secure messaging framework [57] with an abstract authentication/identity system that may use either PKI or Kerberos. Legion also defines an access control policy on objects. Additional details pertaining to the NaradaBrokering security infrastructure can be found in [58].

NaradaBrokering: A Distributed Middleware Framework and Architecture

7

59

Conclusions and Future Work

This paper outlined an extensible messaging framework that, we propose, would be appropriate to host P2P grids. Our results demonstrate that the framework can indeed be deployed for both synchronous and asynchronous applications while incorporating performance-functionality trade-offs for different scenarios (centralized, distributed and peer-to-peer mode). We believe we are now well positioned to incorporate support, within the messaging infrastructure, for Web/Grid Services. We have recently incorporated an XML matching engine within the distributed brokering framework. This allows us to facilitate richer discovery mechanisms. Trade-offs in performance versus functionality inherent in such matching engines is a critical area that needs to be researched further. Another area that we intend to investigate is the model of dynamic resource management. A good example of a dynamic peer group is the set of Grid/Web Services [59, 60] generated dynamically when a complex task runs – here existing registration/discovery mechanisms are unsuitable. A P2P like discovery strategy within such a dynamic group combined with NaradaBrokering’s JMS mode between groups seems attractive. We have also begun investigations into the management of distributed lightweight XML databases using P2P search and discovery mechanisms. Another area amenable to immediate investigation and research is the federation of services in multiple grid realms.

Bibliography 1. The Grid Forum http://www.gridforum.org 2. GridForum Grid Computing Environment working group(http://www.computingportals.org) and survey of existing grid portal projects. http://www.computingportals.org/ 3. “The Grid: Blueprint for a New Computing Infrastructure”, Ian Foster and Carl Kesselman (Eds.), Morgan-Kaufman, 1998. See especially D. Gannon, and A. Grimshaw, “ObjectBased Approaches”, pp. 205-236, of this book. 4. Globus Grid Project http://www.globus.org 5. GriPhyN Particle Physics Grid Project Site, http://www.griphyn.org/ 6. International Virtual Data Grid Laboratory at http://www.ivdgl.org/ 7. NEES Earthquake Engineering Grid, http://www.neesgrid.org/ 8. SCEC Earthquake Science Grid, http://www.scec.org 9. W. Johnston, D. Gannon, B. Nitzberg, A. Woo, B. Thigpen, L. Tanner, “Computing and Data Grids for Science and Engineering,” Proceedings of Super Computing 2000. 10. DoE Fusion Grid at http://www.fusiongrid.org 11. Oram, A. (eds) 2001. Peer-To-Peer: Harnessing the Power of Disruptive Technologies. O’Reilly, CA 95472. 12. Web Services Description Language (WSDL) 1.1 http://www.w3c.org/TR/wsdl 13. Definition of Web Services and Components http://www.stencilgroup.com/ideas_scope_200106wsdefined.html#whatare 14. Geoffrey Fox and Shrideep Pallickara, An Event Service to Support Grid Computational Environments. Concurrency and Computation: Practice and Experience. Volume 14(13-15) pp 1097-1129. 15. Fox, G. Report on Architecture and Implementation of a Collaborative Computing and Education Portal. http://aspen.csit.fsu.edu/collabtools/updatejuly01/erdcgarnet.pdf. 2001. 16. V. Mann and M. Parashar, Middleware Support for Global Access to Integrated Computational Collaboratories, Proc. of the 10th IEEE symposium on High Performance Distributed Computing (HPDC-10), CA, August 2001.

60

Shrideep Pallickara and Geoffrey Fox

17. Ian Foster, Carl Kesselman, Steven Tuecke, The Anatomy of the Grid: Enabling Scalable Virtual Organizations http://www.globus.org/research/papers/anatomy.pdf 18. Kingdom e-Science Activity http://www.escience-grid.org.uk/ 19. Julian Bunn and Harvey Newman. Chapter on Data Intensive Grids for High Energy Physics in Grid Computing: Making the Global Infrastructure a Reality. Editors Berman, Fox and Hey. John Wiley. April 2003. 20. Hasan Bulut et al. An Architecture for e-Science and its Implications. Proceedings of the International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS 2002) July 17 2002. 21. Geoffrey Fox, Ozgur Balsoy, Shrideep Pallickara, Ahmet Uyar, Dennis Gannon, and Aleksander Slominski, "Community Grids" invited talk at International Conference on Computational Science, April, 2002, Netherlands. 22. Java Message Service Specification”. Mark Happner, Rich Burridge and Rahul Sharma. Sun Microsystems. 2000. http://java.sun.com/products/jms. 23. RTP: A Transport Protocol for Real-Time Applications (IETF RFC 1889) http://www.ietf.org/rfc/rfc1889.txt. 24. XML based messaging and protocol specifications SOAP. http://www.w3.org/2000/xp/. 25. The NaradaBrokering System http://www.naradabrokering.org 26. Geoffrey Fox and Shrideep Pallickara. “The Narada Event Brokering System: Overview and Extensions”. Proceedings of the International Conference on Parallel and Distributed Processing Techniques and Applications, June 2002. pp 353-359. 27. Geoffrey Fox, Shrideep Pallickara and Xi Rao. “A Scaleable Event Infrastructure for Peer to Peer Grids”. Proceedings of ACM Java Grande ISCOPE Conference 2002. Seattle, Washington. November 2002. 28. Geoffrey Fox and Shrideep Pallickara. “JMS Compliance in the Narada Event Brokering System”. Proceedings of the International Conference on Internet Computing. June 2002. pp 391-402. 29. “Grid Services For Earthquake Science”. Geoffrey Fox et al. Concurrency & Computation: Practice and Experience. 14(6-7): 371-393 (2002). 30. Hasan Bulut, Geoffrey Fox, Shrideep Pallickara, Ahmet Uyar and Wenjun Wu. “Integration of NaradaBrokering and Audio/Video Conferencing as a Web Service”. Proceedings of the IASTED International Conference on Communications, Internet, and Information Technology, November, 2002, in St.Thomas, US Virgin Islands. 31. Geoffrey Fox and Shrideep Pallickara “An Approach to High Performance Distributed Web Brokering”, ACM Ubiquity Volume2 Issue 38. November 2001. 32. Gurudutt Banavar, et al. An Efficient Multicast Protocol for Content-Based PublishSubscribe Systems.In Proceedings of the IEEE International Conference on Distributed Computing Systems, Austin, Texas, May 1999. 33. Bill Segall and David Arnold. Elvin has left the building: A publish/subscribe notification service with quenching. In Proceedings AUUG97, pages 243–255, Australia, 1997. 34. Fiorano Corporation. A Guide to Understanding the Pluggable, Scalable Connection Management (SCM) Architecture - White Paper. Technical report, http://www.fiorano.com/products/fmq5 scm wp.htm, 2000. 35. Talarian Corporation. Smartsockets: Everything you need to know about middleware: Mission critical interprocess communication. Technical report, URL: http://www.talarian.com/products/smartsockets, 2000. 36. TIBCO Corporation. TIB/Rendezvous White Paper. Technical report, URL: http://www.rv.tibco.com/whitepaper.html, 1999. 37. The Object Management Group (OMG). OMG’s CORBA Event Service. URL: http://www.omg.org/. 38. D.J. Watts and S.H. Strogatz. “Collective Dynamics of Small-World Networks”. Nature. 393:440. 1998.

NaradaBrokering: A Distributed Middleware Framework and Architecture

61

39. R. Albert, H. Jeong and A. Barabasi. “Diameter of the World Wide Web”. Nature 401:130. 1999. 40. The Anabas Conferencing System. http://www.anabas.com 41. The Online Knowledge Center (OKC) Web Portal http://ptlportal.ucs.indiana.edu 42. SonicMQ JMS Server http://www.sonicsoftware.com/ 43. Antony Rowstron and Peter Druschel. Pastry: Scalable, decentralized object location and routing for large-scale peer-to-peer systems. Proceedings of Middleware 2001. 44. Sun Microsystems. The JXTA Project and Peer-to-Peer Technology http://www.jxta.org 45. The JXTA Protocol Specifications. http://spec.jxta.org/v1.0/docbook/JXTAProtocols.html 46. Paul J. Leach and Rich Salz. Network Working Group. UUIDs and GUIDs. February, 1998. 47. Microsoft Internet Security and Acceleration (ISA) Server. http://www.microsoft.com/isaserver/ 48. Checkpoint Technologies. http://www.checkpoint.com/ 49. iPlanet. http://www.iplanet.com/ 50. The Network Weather Service: A Distributed Resource Performance Forecasting Service for Metacomputing Rich Wolski, Neil Spring, and Jim Hayes, Journal of Future Generation Computing Systems,Volume 15, Numbers 5-6, pp. 757-768, October, 1999 51. Apache Jetspeed. http://jakarta.apache.org/jetspeed/site/index.html 52. Zhang, L. et al. “ReSource ReserVation Protocol (RSVP) – Functional Specification”, Internet Draft, March 1994. 53. Topolcic, C., “Experimental Internet Stream Protocol: Version 2 (ST-II)”, Internet RFC 1190, October 1990. 54. J. Steiner, C. Neuman, and J. Schiller. “Kerberos: An Authentication Service For Open Networked Systems”. In Proceedings of the Winter 1988 USENIX Conference. 55. B.Atkinson, et al. “Web Services Security (WS-Security) Version 1.0 05 April 2002,” Available from http://www-106.ibm.com/developerworks/webservices/library/ws-secure/. 56. “Assertions and Protocol for the OASIS Security Assertion Markup Language,” P. HallamBaker and E. Maler, eds. Available from http://www.oasis-open.org/committees/security/docs/ cs-sstc-core-01.pdf. 57. Adam Ferrari et al. "A Flexible Security System for Metacomputing Environments". (HPCN Europe 99), pp 370-380. April 1999 58. Pallickara et. al. A Security Framework for Distributed Brokering Systems available at http://www.naradabrokering.org 59. Semantic Web from W3C to describe self organizing Intelligence from enhanced web resources. http://www. w3c.org/2001/sw/ 60. Berners-Lee, T., Hendler, J., and Lassila, O., "The Semantic Web," Scientific American, May2001.

A Framework for Event Composition in Distributed Systems Peter R. Pietzuch
, Brian Shand

, and Jean Bacon University of Cambridge Computer Laboratory Cambridge CB3 0FD, UK {Peter.Pietzuch,Brian.Shand,Jean.Bacon}@cl.cam.ac.uk

Abstract. For large-scale distributed applications such as internet-wide or ubiquitous systems, event-based communication is an effective messaging mechanism between components. In order to handle the large volume of events in such systems, composite event detection enables application components to express interest in the occurrence of complex patterns of events. In this paper, we introduce a general composite event detection framework that can be added on top of existing middleware architectures – as demonstrated in our implementation over JMS. We argue that the framework is flexible, expressive, and easy to implement. Based on finite state automata extended with a rich time model and support for parameterisation, it provides a decomposable core language for composite event specification, so that composite event detection can be distributed throughout the system. We discuss the issues associated with automatic distribution of composite event expressions. Finally, tests of our composite event system over JMS show reduced bandwidth consumption and a low notification delay for composite events.

1

Introduction

Event-based communication has become a new paradigm for building large-scale distributed systems. It has the advantages of loosely coupling communication partners, being extremely scalable, and providing a simple application programming model. In event-based systems, events are the basic communication mechanism. An event can be seen as a notification that something of interest has occurred within the system. Components either act as event sources and publish new events, or event sinks and subscribe to events by providing a specification of events that are of interest to them. A publish/subscribe (pub/sub) communication layer [1] is then responsible for disseminating events; for efficiency, it can often also filter events by topic or content, according to client specifications. Many existing pub/sub systems [2–4] restrict subscriptions to single events only and thus lack the ability to express interest in the occurrence of patterns of events. However, especially in large-scale applications, event sinks may be




Research supported by UK EPSRC and QinetiQ, Malvern. Research supported by ICL, now part of Fujitsu, and the SECURE EU consortium.

M. Endler and D. Schmidt (Eds.): Middleware 2003, LNCS 2672, pp. 62–82, 2003. IFIP International Federation for Information Processing 2003

A Framework for Event Composition in Distributed Systems Management

Sales Department

e e

Plant B

Client Network

63

e

e e

e

e e

e

Plant A

e

Corporate Backbone

e

Event Message

e e

Plant C

e

Supplier Network

Fig. 1. A publish/subscribe system in a corporate network

overwhelmed by the vast number of primitive, low-level events, and would benefit from a higher-level view. Such a higher-level view is given by composite events (CE) that are published when an event pattern occurs. To date, it is usually left to the event sink to implement a detector for composite events making it unnecessarily complex and error-prone. In this paper, we address the problem by proposing a general framework for composite event detection that works on top of a range of pub/sub systems. This framework includes a generic language for specifying composite events and CE detectors that can detect composite events in a distributed way. The paper is organised as follows: Section 2 motivates the necessity of composite event detection in large-scale distributed systems. After related work (Sect. 3), we discuss prerequisites of the detection framework (Sect. 4) such as the pub/sub infrastructure requirements, the time model and the event model. The CE detectors and the associated core language are presented in Sect. 5, and Sect. 6 discusses distributed detection. In Sect. 7, we present our implementation over JMS, and evaluate its performance. The paper finishes with an introduction to higher-level specification languages (Sect. 8) and conclusions (Sect. 9).

2

Motivation

Large-scale event systems need to support CE detection, in order to quickly and efficiently notify their clients of new, relevant information in the network. This is particularly important for widely distributed systems where bandwidth is limited and components are loosely coupled. In such systems, distributed CE detection can improve efficiency and robustness. For example, consider a large corporate network which connects disparate information systems, illustrated in Fig. 1. The computer system at one site might use the network to notify a supplier that more raw materials were required. At the same time, the sales department might notify all plants of projected regional demand for each product, in order to guide production. Finally, management might want to be informed of all orders over 10 000 from new clients, or of plants increasing production when demand was falling.

64

Peter R. Pietzuch, Brian Shand, and Jean Bacon W

L

Meeting Room 1 A

A

D

D

L

T

A

A

Meeting Room 2

D D

D D

W

Office A FE03

L T

D

Office FE04

L T

A

Kitchen

Seminar Room

L

Office FE02 D

W T

L T

Print Room

T

Office FE01

A Location Sensor (Active Bat) L T

T

Temperature Sensor

W

Whiteboard Sensor

L

Lighting Sensor

D

Door Sensor

L T

Fig. 2. An Active Office environment

In a small company, simple point-to-point messaging between departments would be sufficient. However, this would require considerable administration in a larger organisation, as each information producer would need a list of all intended recipients. A pub/sub system would reduce this overhead, allowing more flexible communication and easier bootstrapping of the system. Nevertheless, without CE detection, many messages would still be sent unnecessarily, because specific event combinations or patterns could not be expressed by recipients. Instead, in the example above, management would have to be notified independently of all large orders and of all new clients. Furthermore, reuse of common subexpressions would be impossible, if for example both management and accounting were interested in orders over 10 000. For reliability and efficiency, each CE detector should be distributed near to its event sources. Otherwise, if one site’s connection to the rest of the network failed, local notification of composite events might fail unnecessarily. Besides, sending these events off-site for detection would have been a waste of bandwidth, if all relevant events were known to be locally produced. Just as a general purpose pub/sub system supports flexible messaging, so too can a generic CE framework extend this support. Therefore, this paper proposes a general purpose middleware system for CE detection, independent of the specific underlying pub/sub infrastructure. By making CE detection closely interoperate with the underlying communication infrastructure, we obtain a system that is more efficient than an ad hoc implementation of CE detectors at the application level. 2.1

Application Scenario: The Active Office

The Active Office is a computerised building which is aware of its inhabitants’ behaviour (cf. Fig. 2). Workers wear Active Bats [5] to inform the building of their movements at least once a minute. Other sensors monitor doors, office temperatures, electronic whiteboard usage, and lighting. A content-based pub/sub system is used so that applications can be notified of specific events, such as

A Framework for Event Composition in Distributed Systems

65

‘location events where Peter is seen in room FE04’. We used the following two application scenarios to test our CE detection framework: Scenario 1. The building services manager wants to know about temperature events under 15
in an occupied room. Scenario 2. Jean wants the list of participants and the electronic whiteboard contents of any meeting she attended to be sent to her wireless PDA, but only if she does not login to the workstation in her office within 5 min of the meeting. There are many advantages of using a CE middleware for services in an Active Office, instead of (or perhaps as well as) offering predefined composite subscriptions on dedicated servers. The most important are the flexibility with which recipients can compose personal subscriptions, and the ease with which composite patterns can be reused and distributed close to event sources. The cost of establishing this network of CE detection broker nodes is then offset by the simplicity of configuring it for new CE subscriptions.

3

Related Work

Historically, composite event detection first arose in the context of triggers in active databases. Early languages for specifying composite events follow the Event-Condition-Action (ECA) model and resemble database query algebras with an expressive, yet complex syntax. In general, the detection process is not distributed. In the Ode object database [6], composite events are specified with a regularexpression-like language and detected using finite state automata (FSA). Equivalence between the CE language and regular expressions is shown. Since a composite event has a single timestamp of the last event that led to its detection, a total event order is created that makes it difficult to deal with clock synchronisation issues. The pure FSAs do not support parameterised events. CE detectors based on Petri Nets are used in the SAMOS database [7]. Coloured Petri Nets can represent concurrent behaviour and manage complex data such as event parameters during detection. However, even for simple expressions, they quickly become complicated. SAMOS does not support distribution and has a simple time model that is not suitable for distributed systems. The motivation for Snoop [8] was to design an expressive CE specification language with powerful temporal support. A CE detector is a tree that reflects the structure of the event expression. Its nodes implement language operators and conform to a particular consumption policy. A consumption policy influences the semantics of an operator by resolving which events are consumed from the event history in case of ambiguity. For example, under a recent policy only the most recently occurring event is considered; others are ignored. Detection propagates up the tree with the leaves of the tree being primitive event detectors. A disadvantage is that the nodes are essentially Turing-complete making it difficult to formalise their semantics and to reason about their behaviour. The use of consumption policies can be non-intuitive and operator-dependent.

66

Peter R. Pietzuch, Brian Shand, and Jean Bacon

In [9], Schwiderski presents a distributed CE architecture based on the 2gprecedence model for monitoring distributed systems. This model makes strong assumptions about the clock granularity in the system and thus does not scale to large, loosely-coupled distributed systems. The language and the detection algorithm used are similar to Snoop and suffer from the same shortcomings. It addresses the issue of events being delayed during transport by evaluation policies: asynchronous evaluation enables a detector to consume an event as soon as it arrives sometimes leading to incorrect detection, whereas synchronous evaluation forces a detector to delay evaluation until all earlier events have arrived, and assumes a heartbeat infrastructure. Although detection is distributed, no decision on the efficient placement of detectors in the network is made. The GEM system [10] has a rule-based event monitoring language. It follows a tree-based detection approach and assumes a total time order. Communication latency is handled by annotating rules with tolerable delays. Such an approach is not feasible in an environment with unpredictable delays. Research efforts in ubiquitous computing have led to CE languages that are intuitive to use in environments such as the Active Office. The work by Hayton [11] on composite events in the Cambridge Event Architecture (CEA) [12] is similar to ours in the sense that it defines a language that non-programmers can use to specify occurrences of interest. Hayton uses push-down FSAs to handle parameterised events. However, the language itself can become non-intuitive as the semantics of some operators is not obvious. Even though detectors can use composite events as their input, distributed detection is not dealt with explicitly. As in previous work, scalar timestamps are used. Distributed pub/sub architectures such as Hermes [4], Gryphon [3, 13], and Siena [2] only provide parameterised primitive events and leave the task of CE detection to the application programmer. Siena supports restricted event patterns, but it does not define a complete pattern language. In our CE detection framework, we adopt the interval timestamp model introduced in [14]. The partial order of timestamps in a distributed system is made explicit by having timestamps associated with an uncertainty interval. A CORBA-based detection architecture is presented in [14] that implements this time model. The notion of event stability is defined in order to handle communication delays. We extend this to cope with delays in wide-area systems.

4

Design and Architecture

The CE detectors in our framework recognise concurrent patterns of simpler events, generating a composite event whenever a match is found. The component layers of our detection architecture are illustrated in Fig. 3: Distributed CE detectors are compiled from expressions in our core CE language. Patterns can be specified using higher-level languages, which are first translated into the core CE language before compilation and execution. The CE framework relies on and interacts with the underlying event system, in order to detect complex patterns of events. This section outlines the prereq-

A Framework for Event Composition in Distributed Systems

67

Application Human Specification

Higherlevel Languages

Factorisation and Distribution

Core Composite Event Language

Execution and Matching

CE Detection Automata

sub(CE)

CE Detection Framework sub

notify

pub

sub

notify

pub

Publish/Subscribe System Network Transport

Expressiveness

Fig. 3. Components of the composite event detection framework

Fig. 4. Interface between the CE detection framework and the pub/sub system

uisites for this interaction: an interface to a pub/sub infrastructure, and formal models of events and time. Given these prerequisites, the full expressive power of our CE languages can be used. 4.1

Publish/Subscribe Infrastructure Support

One of our design goals was to keep the CE detection framework strictly separated from the pub/sub infrastructure used. The interface to the event system (Fig. 4) makes only minimal assumptions about the functionality supported allowing our framework to be deployed on a large variety of pub/sub systems. Our current test-bed uses the Java Message Service (JMS) [15], but other pub/sub systems could equally be used: earlier work was based on Hermes [4], a distributed event-based middleware architecture, and CORBA Events would also be suitable. In addition to the time and event model described below, the underlying pub/sub system needs to support (1) publication of primitive events by event sources, (2) subscription to these events by event sinks, and (3) relaying of events from sources to sinks. Many systems also filter events en route for efficiency; our CE framework uses this if available, but no particular publication or subscription model is assumed. Our event model uses the abstraction of a describable event set as an atom for CE detection. If the pub/sub system supports content-based filtering, a describable event set will be defined by a parameterised filtering expression. In a topic-based system, it will conform to a certain event type only. In particular, the pub/sub system does not need to be aware of CE types. As illustrated in Fig. 4, application event sources submit CE subscriptions to the CE detection layer. Any composite events that are then detected by a CE detector are published to the pub/sub system disguised as primitive events. It is then the responsibility of the pub/sub system to disseminate these encapsulated CE occurrences to all interested event sources. The same mechanism is used for the communication between distributed event detectors (cf. Sect. 6).

68

Peter R. Pietzuch, Brian Shand, and Jean Bacon t3l t

l 2

t

l 1

t1

t

h 1

t2

t

h 2

t3

t3h

t

Fig. 5. Illustration of interval timestamps for events

4.2

Composite Event Detection Framework

The Java interface to the CE detection service, presented to applications, is shown below in part. Applications may use this for all event services, or contact the underlying pub/sub infrastructure directly for primitive event subscriptions. public interface DistCEDServiceInf { public void registerCEType(CEType type, CEPublisherInf publisher); public void unregisterCEType(CEType type, CEPublisherInf publisher); public CEInf createCE(CEType ceType); public CEType createCEType(String typeName); public void publish(CEInf ce, CEPublisherInf publisher); public void subscribe(CEType type, CESubscriberInf subscriber, CEQoSInf qos, CESubscriberCallbackInf callback); public void unsubscribe(CEType type, CESubscriberInf subscriber); }

Before an event type can be published it must be registered with the CE detection service so that e.g. an appropriate type/topic is created in the underlying pub/sub system. After that, a new event instance can be created using the createCE method. The publish method will pass the publication down to the pub/sub system. A call to subscribe subscribes to primitive or composite events. A CE subscription may trigger the instantiation of new CE detectors. Time Model. Each event in our framework has an associated timestamp, denoting when it occurred. In a large-scale system, it may often be impossible to decide which of two events occurred first. Therefore we assume that there is a partial order relation on timestamps ‘