Parallel Processing for Artificial Intelligence [1st Edition] 9781483295749

Table of contents :
Content:
Machine Intelligence and Pattern RecognitionPage ii
Front MatterPage iii
Copyright pagePage iv
PrefacePages v-viiiLaveen N. Kanal, Vipin Kumar, Hiroaki Kitano, Christian Suttner
EditorsPage xi
AuthorsPages xiii-xv
Chapter 1 - A Perspective on Parallel Processing in Computer Vision and Image UnderstandingPages 3-20Alok Choudhary, Sanjay Ranka
Chapter 2 - On Supporting Rule-Based Image Interpretation Using a Distributed Memory MulticomputerPages 21-44Chen-Chau Chu, Joydeep Ghosh, J.K. Aggarwal
Chapter 3 - Parallel Affine Image Warping*Pages 45-66George Gusciora, Jon A. Webb
Chapter 4 - Image Processing On Reconfigurable Meshes With Buses*Pages 67-91Jing-Fu Jenq, Sartaj Sahni
Chapter 5 - Inheritance Operations in Massively Parallel Knowledge RepresentationPages 95-113James Geller
Chapter 6 - Providing Computationally Effective Knowledge Representation via Massive ParallelismPages 115-135Matthew P. Evett, William A. Andersen, James A. Hendler
Chapter 7 - Speeding Up Production Systems: From Concurrent Matching to Parallel Rule Firing*Pages 139-160José Nelson Amaral, Joydeep Ghosh
Chapter 8 - Guaranteeing Serializability in Parallel Production SystemsPages 161-205James G. Schmolze
Chapter 9 - Parallel Automated Theorem Proving*Pages 209-257Christian B. Suttner, Johann Schumann
Chapter 10 - Massive Parallelism in Inference SystemsPages 259-277Franz Kurfeß
Chapter 11 - Representing Propositional Logic and Searching for Satisfiability in Connectionist NetworksPages 279-301Gadi Pinkas
Chapter 12 - Parallel and Distributed Finite Constraint Satisfaction: Complexity, Algorithms and ExperimentsPages 305-334Ying Zhang, Alan K. Mackworth
Chapter 13 - Parallel Algorithms and Architectures for Consistent LabelingPages 335-362Wei-Ming Lin, Viktor K. Prasanna
Chapter 14 - Massively Parallel Parsing Algorithms for Natural Language*Pages 365-407Michael A. Palis, David S.L. Wei
Chapter 15 - Process Trellis and FGP: Software Architectures for Data Filtering and MiningPages 409-428Michael Factor, Scott J. Fertig, David H. Gelernter

Citation preview

Machine Intelligence and Pattern Recognition Volume 14

Series

Editors

L.N. KANAL and A. ROSENFELD University of Maryland College Park, Maryland, U.S.A.

NORTH-HOLLAND AMSTERDAM · LONDON · NEW YORK · TOKYO

Parallel Processing for Artificial Intelligence 1 Edited

by

Laveen N. K A N A L University of Maryland College Park, Maryland,

U.S.A.

Vipin K U M A R University of Minnesota Minneapolis, Minnesota,

U.S.A.

Hiroaki KITANO Sony Computer Science Laboratory, and Carnegie Mellon University Pittsburgh, Pennsylvania, U.S.A.

Japan

Christian B. SUTTNER Technical Munich,

University Germany

of

Munich

1994 NORTH-HOLLAND AMSTERDAM · LONDON · NEW YORK · TOKYO

ELSEVIER SCIENCE B.V. Sara Burgerhartstraat 25 P.O. Box 211, 1000 AE Amsterdam, The Netherlands

Library of Congress Cataloglng-ln-PublIcatIon Data

P a r a l l e l p r o c e s s i n g f o r a r t i f i c i a l i n t e l l i g e n c e / e d i t e d by L . N . K a n a l , ... [et a l . ] . p. cm. — (Machine Intelligence and pattern recognition ; v. 14-15) C o l l e c t i o n o f p a p e r s f r o m t h e I J C A I - 9 3 , h e l d In C h a m b e r y , F r a n c e . Includes bibliographical references. I S B N 0 - 4 4 4 - 8 1 7 0 4 - 2 (v. 1 ) . — I S B N 0 - 4 4 4 - 8 1 8 3 7 - 5 ( v . 2 ) 1. P a r a l l e l p r o c e s s i n g ( E l e c t r o n i c c o m p u t e r s ) 2. Artificial Intelligence. I. K a n a l , L a v e e n Ν . I I . I n t e r n a t i o n a l J o i n t C o n f e r e n c e on Artificial Intelligence ( 1 9 9 3 : C h a m b e r y , F r a n c e ) III. Series: Machine Intelligence and pattern recognition ; v. 14-. QA76.58.P37775 1994 006.3—dc20 94-15133 CIP

ISBN: 0 444 81704 2 © 1994 Elsevier Science B.V. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior written permission of the publisher, Elsevier Science B.V., Copyright & Permissions Department, P.O. Box 521, 1000 AM Amsterdam, The Netherlands. Special regulations for readers in the U.S.A. - This publication has been registered with the Copyright Clearance Center Inc. (CCC), Salem, Massachusetts. Information can be obtained from the CCC about conditions under which photocopies of parts of this publication may be made in the U.S.A. All other copyright questions, including photocopying outside of the U.S.A., should be referred to the copyright owner, Elsevier Science B.V., unless otherwise specified. No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions or ideas contained in the material herein. This book is printed on acid-free paper. Printed in The Netherlands

PREFACE Parallel processing for AI problems is of great current interest because of its potential for alleviating the computational demands of AI procedures. The articles in this book consider parallel processing for problems in several areas of artificial intelligence: image processing, knowledge representation in semantic networks, production rules, mechanization of logic, constraint satisfaction, parsing of natural language, data filtering and data mining. The articles are grouped into six sections. The first four papers address parallel computing for processing and understanding images. Choudhary and Ranka set the stage for this section, titled Image Processing, by examining, as an example, the computationally intensive nature of feature extraction in image processing. They note that vision systems involve low-level, medium-level and high-level operations and require integrating algorithms for image processing, numerical analyis, graph theory, AI and databases. They discuss issues of mesh and pyramid architectures, spatial and temporal parallelism, local and global communication, data decomposition and load balancing and the status of architectures, programming models and software tools in parallel computing for computer vision and image understanding. The second article, by Chu, Gosh, and Aggarwal, focusses on high-level operations for image understanding and reports on a parallel implementation of a rule-based image interpretation system on a distributed memory architecture machine with message passing. The authors discuss the results of their experimental investigation of this implementation from the perspectives of data access locality and task granularity. The next article by Gusciora and Webb presents results of their investigation of several methods for parallel affine image warping on a linear processor array. Evaluating the efficiency, capability, and memory use of the various methods, the authors describe and recommend a method which they call "sweep-based" warping. In the final article of Section I, Jenq and Sahni examine a class of parallel computers called reconfigurable mesh with bus (RMB). They present fast algorithms for template matching, clustering, erosion and dilation, and area and perimeter computation of image components for some members of the RMB family of architectures. The articles in Section II discuss parallel processing for semantic networks. Semantic networks are a widely used means for representing knowledge; methods which enable efficient and flexible processing of semantic networks are expected to have high utility

vi

for building large-scale knowledge-based systems. Geller presents a method to speed up search in an inheritance tree. The method is based on a parallel class tree representation that uses a list in combination with a preorder number scheme. In addition to standard inheritance, a newly defined operation, upward inductive inheritance, is introduced. The proposed methods exhibit high performance which is largely independent of tree depth and branching factor. The second article in this section, by Evett, Andersen, and Hendler, describes PARKA, a frame-based knowledge representation system implemented on the Connection Machine. PARKA performs recognition queries, finding all objects with a set of specified properties in 0(d+m) time — proportional primarily to the depth d of the knowledge base. They have tested PARKA's performance using knowledge from MCC's Cyc commonsense knowledge base, and have obtained very high performance. Section III deals with the automatic parallel execution of production systems. Production systems are used extensively in building rule-based expert systems. Systems containing large numbers of rules are slow to execute, and can significantly benefit from automatic parallel execution. The article by Amaral and Ghosh provides a survey of the research on parallel processing of production systems. It covers the early systems in which all productions are matched in parallel but only one rule is fired at a time, as well as more recent research in which different rules are fired concurrently. The article concludes with observations on the overall effectiveness of these different approaches and suggestions for further work. The next article, by Schmölze, deals extensively with the problems involved in concurrent firing of rules. The original production systems were designed for executing one rule at a time. If multiple rules are executed simultaneously, then in some cases, the parallel execution may lead to undesirable results. Schmölze presents a framework for guaranteeing the serializable behavior in the parallel execution of productions. Section IV deals with the exploitation of parallelism for the mechanization of logic. While sequential control aspects pose problems for the parallelization of production systems (see Section III), logic has a purely declarative interpretation which does not demand a particular evaluation strategy. Therefore, in this area, very large search spaces provide a significant potential for parallelism. In particular, this is true for automated theorem proving. The three articles in Section IV deal with parallel deduction at various levels, ranging from coarse granular parallel deduction for predicate logic to connectionist processing of propositional logic.

vii

Suttner and Schumann's article gives a comprehensive survey of the approaches for parallel automated theorem proving for first order logic. It includes both implemented and proposed systems, and briefly describes for each approach the underlying logic calculus, the model of computation, and available experimental results, together with an assessment. In addition the authors present a classification scheme for parallel search based systems which leads to an adequate grouping of the approaches investigated. The orthogonal distinctions proposed are partitioning versus competition parallelization (with further subclasses), and cooperative versus uncooperative parallelization. Together with an extensive list of references, the article provides a classification and overview of the state-of-the-art and the history of parallel theorem proving. The article by Kurfess primarily provides an overview of the potential application of parallelism in logic. The author first describes various levels at which parallelism can be utilized, ranging from the formula, clause, literal, term, and symbol level down to the subsymbolic level. He then focuses on the prospects of massive parallelism for these levels, and discusses their potential regarding computation, communication, synchronization, and memory. In this regard, the author proposes the "term level" as the most promising. Finally, a fine-grain deductive system is described in a connectionist setting, in which parallel unification is based on a matrix representation of terms. Pinkas describes a connectionist approach towards the mechanization of logic. His method is based on a linear time transformation of propositional formulas into symmetric connectionist networks representing quadratic energy functions. The global minima of such a function are equivalent to the satisfying models of the propositional formula. The author presents simulation results for randomly generated satisfiability problems using three algorithms based on Hopfield networks, Boltzmann machines, and Mean-Field networks, respectively. He also describes the application of the approach for propositional inference and the incremental adaptation of energy functions to represent changes in the knowledge base. In the first article in Section V, Zhang and Mackworth consider the problem of constraint satisfaction, which is a useful abstraction of a number of important problems in AI and other fields of computer science. They present parallel formulations of some well-known constraint satisfaction algorithms and analyze their performance theoretically and experimentally. In particular they show that certain classes of constraint satisfaction problems can be solved in polylog time using a polynomial number of processors. In the second article in this section, Lin and Prasanna discuss the technique of consistent labeling as a preprocessing step in the constraint satisfaction problem and present several parallel implementations for the technique.

viii Section VI consists of two articles, each on a different, important topic. Palis and Wei discuss parallel formulations for the Tree Adjoining Grammar (TAG), which is a powerful formalism for describing natural languages. The serial complexity of TAG parsing algorithms is much higher than the complexity of simpler grammars such as context-free grammars. Hence it is important to be able to reduce the overall run time using parallel computing. Palis and Wei present two parallel algorithms for TAG parsing, and present theoretical and experimental results. In the final article, Factor, Fertig, and Gelernter discuss the suitability of a parallel programming paradigm called Linda for solving problems in artificial intelligence. They introduce two software architectures, the FGP machine and the Process Trellis. The FGP machine is a software architecture for a data-base driven expert system that learns. Process Trellis is a software architecture for real time heuristic monitors. Both of these systems have been implemented in the language Linda. As is noted in the first article in the book, while some of the individual components of parallelism in vision systems may be understood, the overall process of vision remains very much an open problem. A similar remark applies to knowledge representation, reasoning, problem solving, natural language processing and other capabilities desired for machine intelligence. We think parallel processing will be a key ingredient of (partial) solutions in all the above areas. We thank the authors for the articles included here and hope their work inspires some readers to take on these challenging areas of inquiry.

Laveen N. Kanal College Park, MD Vipin Kumar Minnaepolis, MN Hiroaki Kitano Tokyo, Japan Christian Suttner Munich, Germany

xi

EDITORS Laveen Ν . Kanal is a Professor of Computer Science at the University of Maryland, College Park, and Managing Director of L Ν Κ Corporation, Inc. In 1972 he was elected a Fellow of the IEEE and a Fellow of the American Association for the Advancement of Science. In 1992 he was elected a Fellow of the American Association for Artificial Intelligence and received the King-Sun Fu award of the International Association for Pattern Recognition. Vipin K u m a r is currently an Associate Professor in the Department of Computer Science at the University of Minnesota. His co-authored text book, Introduction to Parallel Computing, was published in 1993 by Benjamin Cummings. He is on the editorial board of IEEE Transactions on Data and Knowledge Engineering. Hiroaki Kita no, Ph.D, is with Sony Computer Science Laboratory and Carnegie Mellon University. In 1993 he received the Computers and Thought Award of the International Joint Conference on Artificial Intelligence. Christian Suttner is currently writing his doctoral dissertation and working on tools and methods for the utilization of parallel computers at the TU München. With Laveen Kanal he co-edited the Proceedings of a Workshop on Parallel Processing held in Sydney, Australia in August 1991.

xiii

AUTHORS J . K . Aggarwal is Cullen Professor of Electrical and Computer Engineering at The University of Texas at Austin and Director of the Computer and Vision Research Center. An IEEE Fellow, he is an Editor of IEEE Transactions on Parallel and Distributed Systems. In 1992, Dr. Aggarwal received the Senior Research Award of the American Society of Engineering Education. José Nelson Amaral is a professor at Pontificia Universidade Catolica do Rio Grande do Sul (PUCRS) - Brazil. He is currently working towards his Ph.D. at the Univ. of Texas at Austin. Alok Choudhary is an associate professor at Syracuse University. His research interests include high-performance parallel and distributed computing, software environments and applications. He received an NSF Young Investigator Award in 1993. Chen-Chau Chu received the Ph.D. degree in electrical and computer engineering from The University of Texas at Austin in 1991. He is currently employed by the Schlumberger Austin Systems Center. His research interests include machine intelligence, computer vision, and data modeling. M a t t Evett received his Ph.D. in computer science from the University of Maryland in 1994, and joined the faculty at Florida Atlantic University in Boca Raton. James Geller is an associate professor at the Computer and Information Sciences Department of the New Jersey Institute of Technology. He has worked in artificial intelligence and object-oriented databases and is currently involved in a large research project on distance learning. Joydeep Ghosh is currently an associate professor of electrical and computer engineering at the University of Texas at Austin where he conducts research on parallel, intelligent computer architectures and artificial neural systems. He received the 1992 Darlington Award for best journal paper from the IEEE Circuits and Systems Society.

xiv George Gusciora expects to receive the Ph.D.in computer engineering from Carnegie Mellon University in February 1994. He plans to work at the Maui High Performance Computing Center in Hawaii. His research interests include parallel programming tools, parallel computer architectures and parallel algorithms. Jim Hendler is an associate professor and head of the Autonomous Mobile Robots Laboratory at the University of Maryland. He serves as the Artificial Intelligence area editor for the journal Connection Science is an associate editor of the Journal of Experimental and Theoretical AI, and is on the editorial board of Autonomous Robots. Jing-Fu Jeng, Ph.D., University of Minnesota, 1991, is currently an assistant professor of physics, mathematics, and computer science at Tennessee State University in Nashville, Tennessee. His research interests include parallel and distributed processing algorithms, computer architectures and systems, and computer vision. Franz J . Kurfess joined the Neural Information Processing Department at the University of Ulm in 1992 after a stay as postdoctoral fellow with the International Computer Science Institute in Berkeley. His main interest lies in combining artificial intelligence, neural networks and parallel processing techniques, in particular for logic and reasoning. Wei-Ming Lin, Ph.D., University of Southern California, 1991, is currently an assistant professor at the University of Texas at San Antonio. His research interests include parallel and distributed computing, parallel processing for artificial intelligence, image processing and computer vision. Alan Mackworth is a professor of computer science at the University of British Columbia and the Shell Canada Fellow of the AI and Robotics Program of the Canadian Institute for Advanced Research. He currently serves as the Director of the UBC Laboratory for Computational Intelligence. He is known for his work on constraint satisfaction and its applications in perception, reasoning and situated robots Michael A. Palis is currently an associate professor of electrical and computer engineering at the New Jersey Institute of Technology. He is an editorial board member of the IEEE Transactions on Computers, and subject area editor of the Journal of Parallel and Distributed Computing. Gadi Pinkas is currently director of research and development at Amdocs Inc., a senior fellow of the Center for Optimization and Semantic Control and a senior research associate at the Department of Computer Science at Washington University, St. Louis, Missouri.

XV

Viktor Κ. P r a s a n n a (V. Κ. Prasanna Kumar) is an associate professor in the Department of Electrical Engineering-Systems, University of Southern California, Los Angeles. He is the editor of the book: Parallel Architectures and Algorithms for Image Understanding, published by Academic Pres. He is a subject area editor of the Journal of Parallel and Distributed Computing. Sanjay R a n k a is an associate professor of computer science at Syracuse University. He has co-authored a book on Hypercube Algorithms with Applications on Image Processing and Pattern Recognition. Sartaj Sahni is a professor of computer and information science, University of Florida and a Fellow of the IEEE. He is well known for his research on algorithms in several areas of computer science, and for his text books. James G. Schmölze is an associate professor of electrical engineering and computer science at Tufts University in Boston, Mass. In addition to parallel production systems, he works in knowledge representation, real time planning and control, and computational vision. Christian Suttner is currently employed as a full time researcher at the TU München on a large project on parallel computation. His main interests are the parallelization and evaluation of search-based systems, as well as issues on heuristic search control and automated theorem proving. Jon A. W e b b , Ph.D., is a senior systems scientist in the School of Computer Science at Carnegie Mellon University. He contributed to the design of the Carnegie Mellon Warp machine and the Carnegie Mellon-Intel Corporation iWarp Computer and developed applications in computer vision, including the use of the Warp on the Navlab autonomous land vehicle. He has recently implemented a fast stereo vision system on an 64-cell iWarp computer. David S.L. Wei, Ph.D., University of Pennsylvania, is currently an associate professor in the School of Computer Science and Engineering, at the University of Aizu, Japan. His current research is on developing efficient algorithms, tools and programming environments for various models of parallel machines. Ying Zhang is a Ph.D. student of computer science at the University of British Columbia. Her dissertation work is on modeling and reasoning about robotic systems and behaviors. She is interested in constraint satisfaction, dynamic systems and their applications to real-time embedded control.

Parallel Processing for Artificial Intelligence 1 L.N. Kanal, V. Kumar, H. Kitano and C.B. Suttner (Editors) © 1994 Elsevier Science B.V. All rights reserved.

3

Chapter 1 A Perspective on Parallel Processing in Computer Vision and Image Understanding Alok C h o u d h a r y and Sanjay R a n k a 121 Link Hall, ECE Department, Syracuse University, Syracuse, NY 13244 4-116 CST, School of CJS, Syracuse University, Syracuse, Ny 13244-4100

In this paper we review the basic issues and current status of parallel processing for Computer Vision and Image Understanding. 1. 1.1.

Introduction Vision

There are as m a n y definitions of vision as there are opinions. From one point of view, "vision is t h e process of recognizing objects of interest from images". T h e word "process" refers t o some form of processing performed on t h e input data, which may be one or more images. T h e phrase "objects of interest" implies t h e existence of a context under which this processing takes place and t h e existence of a representation used in t h e processing. For example, if we were asked, "Is there a book-shelf in this r o o m ? " , we would have some representation of a book-shelf in our mind and look for something similar t o it in the room. In t h a t process, we would ignore objects t h a t did not look like a book-shelf. T h a t is, we started with a model and searched for some instance of t h e model in t h e room. On the other h a n d , if we were asked, "Describe all t h e objects in this room", we would scan t h e room, form some representation of t h e objects, and m a t c h t h e m t o some objects t h a t are p a r t of our "knowledge base". However, we m a y or may not know w h a t t o expect in t h e r o o m . B o t h problems can be considered "vision problems". A general vision problem, therefore, is considered t o be an ill-posed problem from a c o m p u t a t i o n a l perspective. Vision has fascinated researchers from various disciplines such as psychology, neural science, computer science and engineering, etc. for a long time. A good discussion on various aspects of vision can be found in [2].

4

1.2.

The Role of Parallel

Processing

Problems in computer vision are computationally intensive. Consider a sequence of images at m e d i u m resolution (512 χ 512 pixels) and s t a n d a r d frame rate (30 frames per second) in color (3 bytes per pixel) [3]. This represents a rate of almost 24 million bytes of d a t a per second. A simple feature extraction algorithm m a y require thousands of basic operations per pixel, a n d a typical vision system requires significantly more complex c o m p u t a t i o n s . Parallel computing is essential in solving such a problem. Historically, t h e need to speed up image processing computations brought parallel processing into the computer vision domain. Most image processing algorithms are inherently parallel because they involve similar computations for all pixels in an image. This has inspired the development of array processors. For example, NASA G o d d a r d Center's M P P [1] has a 128 x 128 processor mesh connected array specifically built for image processing. A mesh architecture is one in which each processor is connected t o four neighboring processors; North, South, East and West. It is suitable for image processing, because its structure closely mimics t h a t of a two-dimensional image, a n d it provides an efficient local communication structure. T h e lack of efficient global communication capabilities in a mesh architecture, and the requirement for fast top-down and b o t t o m - u p image processing, led researchers to propose another architecture known as a "Pyramid" . A P y r a m i d architecture normally consists of several levels of meshes in which the top level has one processor and each succeeding level has four times as m a n y processors as its parent array. In addition, to the mesh interconnections within each level, each processor is also connected to its four children (except in t h e b o t t o m layer), and to its parent (except the r o o t ) . Therefore, a pyramid architecture maintains several levels of image representations simultaneously. Both mesh and pyramid architectures have contributed significantly to the understanding and development of new algorithms for image analysis and vision, and they have considerably influenced the subsequent designs of parallel architectures for vision. Parallel processing has taken tremendous strides in t h e last decade. It has enabled scientists t o perform very large scientific computations, which were impractical a few years ago. B u t the immense computational challenge presented by vision is yet t o be satisfied. In fact, compared to the impact of parallel processing in other areas, the impact of parallel processing in t h e vision domain has been minimal. A typical vision system requires integrating algorithms from diverse areas such as image processing, numerical analysis, graph theory, artificial intelligence and databases. There is no clear understanding and consensus on how to achieve this. Specific prob-

5

lems in integration can also be a t t r i b u t e d t o a lack of understanding of the vision process itself, even if t h e c o m p u t a t i o n s and parallelism of some individual components are well understood. 2.

Parallelism in Vision Systems

Available parallelism in integrated vision systems can be placed in two broad categories: Spatial and Temporal Parallelism. 2.1.

Spatial

Parallelism

Spatial parallelism is one in which similar operations are applied in all p a r t s of the image d a t a . T h a t is, the d a t a can be divided into m a n y granules and distributed to subtasks which m a y execute on different processors in parallel. Most vision algorithms exhibit this type of parallelism. In an CVS, each task operates on the o u t p u t d a t a of the previous task in the system. Therefore, the type of d a t a , and d a t a structures m a y be different for each task in the system b u t each form of d a t a can be partitioned into several granules to be processed in parallel. For example, consider an C V S t h a t performs object recognition. T h e i n p u t image is smoothed using some filtering operation, then on the smoothed image an operator is applied for feature extraction, features with similar characteristics are grouped, t h e n matching with t h e models is performed. Each of these tasks takes the o u t p u t of the previous tasks as its i n p u t a n d produces an o u t p u t which becomes the input for the next task. Note t h a t within spatial parallelism, depending on the computation involved, an algorithm implementation m a y be suitable for d a t a parallelism or task parallelism or both. 2.2.

Temporal

Parallelism

Temporal parallelism is available when these tasks are repeated on a t i m e sequence of images or on different resolutions of images. For example, the system in which motion of a moving object is estimated takes a sequence of images of the moving object and performs the same set of c o m p u t a t i o n on all image frame(s). T h e processing of each frame or a set of frames can be done in parallel with the processing of frames of other time instances. Figure 1 shows the computational model for CVS which illustrates the above mentioned characteristics of an C V S . Each pipeline shows a number of tasks applied to a set of inputs. T h e input to the first task in a pipeline is the image, and the input to the rest of the tasks is the o u t p u t of the previous task. T h e set of pipelines illustrates t h a t the entire pipeline of tasks is repeated on different images in time a n d / o r resolution. Each block

6

in the pipeline represents one task. Each task is decomposed into subtasks to be performed in parallel. For example, T\ is one task, and Ti(d\) is a subtask of T\ operating on d a t a granule d\. T h e figure shows m tasks in the pipeline. T h e number of subtasks depends on the a m o u n t of d a t a in a granule and number of available processors. Α\ί+ι represents d a t a transfer from task T{ to task 7*+i in the pipeline. T h e model does not m a k e any assumptions about a particular implementation of a task. S.S.

Data

Dependencies

Existence of spatial and temporal parallelism m a y also result in two types of d a t a dependencies, namely, spatial data dependency and temporal data dependency. Spatial d a t a dependency can be classified into intratask d a t a dependency and intertask d a t a dependency. Each task itself is a collection of subtasks which may be represented as a graph with nodes representing the subtasks and edges representing communication between subtasks. Intratask d a t a dependencies arise when a set of subtasks needs t o exchange d a t a in order to execute a task in parallel. T h e exchange of d a t a may be needed during the execution of the algorithm, or t o combine the partial results, or both. Intertask d a t a dependency denotes t h e transfer and reorganization of d a t a to be passed onto the next task in the pipeline. T h e mode of communication may be subtasks of the current tasks to the subtasks of the next task, or collection and reorganization of the o u t p u t d a t a of the current task and then redistribution of the d a t a for the next task. T h e choice of methods depend on the underlying parallel architecture, m a p p i n g of algorithms and input-output relationships between tasks. T h e set of algorithms which perform the reorganization of d a t a are crucial to exploit the available parallelism. We will call these algorithms as D a t a Conversion Algorithms. Temporal d a t a dependency is similar to spatial d a t a dependency except t h a t some form of o u t p u t generated by tasks executed on the previous image frames may be needed by one or more tasks executing on the current image frames. 3.

Representation B a s e d Classification of Vision C o m p u t a t i o n s

Currently, the dominant approach to characterizing vision computations is to classify the processing requirements into three levels; viz., low-level, intermediate-level and high-level. T h e most recent image understanding benchmark [3] embodies this characterization. (i) L o w - L e v e l P r o c e s s i n g : This level of processing is normally termed bottom-up processing [3]. Most image processing operations fall

7

Data Dependencies (Spatial) Ti(.

>.

ïi(d 2)

Data Dependencies Γι(*)

Output

• • • Tm{dnm)

T 2( 1, as illustrated in Figure 3, a naive implementation of the forward algorithm will miss output pixels, since each input pixel is mapped to several output pixels. A naive implementation will only assign one of them the correct value and skip the others. This is illustrated in Figure 4, where a white pattern on expansion gets uninitialized black pixels inserted randomly. For the correct implementation of affine image warps, therefore, we must

48

Fig. 2. Aliasing can result from resampling incorrectly.

Β Β Fig. 3. Expanding transformation.

average (or combine in some other way) all pixels that map onto the output pixel in contracting transforms, and we must be sure to assign each pixel a value in expanding transformations. This is easy to do when the transform type and the algorithm type are matched. If the transformation is contracting and we are using a forward algorithm, output pixels will be assigned multiple values, which can be averaged. If the transformation is expanding and the algorithm is inverse, each output pixel will be considered individually, and its correct interpolated value can be calculated. However, if the transform type and algorithm type are not matched, one image must be accessed in an irregular pattern. For example, if the algorithm type is forward and the transform type is contracting, all the pixels in the parallelogram that maps onto an output pixel must be accessed and averaged to calculate the correct value. It is necessary to treat this case when implementing general affine image warping with a fixed algorithm type. In the discussion below, we will refer to correctly sampling the image in all cases as dealing with the sampling problem. Another complication in comparing forward with inverse algorithms results from interpolation. In the inverse algorithm, the input address is computed from the output pixel position; as a result, the input address can lie between pixels. An interpolation method, such as Lagrangean interpolation, can be used to accurately

II

Fig. 4. A naive implementation of an expanding transformation can skip pixels.

49

estimate the value of the corresponding output pixel. In the forward algorithm, however, the output pixel address is calculated from the input; if this address lies between pixels, there is nothing to be done but to assign the value to the nearest output pixel. Therefore, Lagrangean interpolation cannot be used in forward algorithms. 3.

Other important characteristics of affine image warping

There are two other characteristics that greatly affect the nature of parallel implementations of affine image warping. They are (1) The mapping between input and output pixels is known at the start of execution, and (2) nearby input pixels usually map to nearby locations in the output image (locality). Because the mapping is known before execution, the warping parameters can be broadcast to all processors so that each can calculate a part of the warp in parallel. As a result either the input or the output image, or both, can be partitioned. This is in contrast to, for example, histogram, in which the input to output mapping is not known until each input pixel is accessed, so that it is not possible to easily divide the histogram among processors (though the input image can still be divided easily.) Locality exists because useful warping transformations do not stretch the image by a large factor. Large stretches involve interpolating several pixels between two input pixels, and there is not enough information in the input image to do this correctly, so such warps introduce interpolation artifacts. This characteristic allows us to make useful assumptions about efficient ways to distribute the input and output images among processors. 4.

Machines

Early experiments in this study were done on Warp [1], a systolic array computer developed at Carnegie Mellon. A simple diagram of Warp's architecture sufficient for the purposes of this paper is given in Figure 5. The Warp machine consists of ten Warp cells arranged in a linear pipeline and connected to an external host. Each Warp cell can perform a floating point add and multiply in one 200 ns cycle (with a seven stage pipeline delay), as well as read and write its memory and send and receive data from adjacent cells. The machine thus develops 100 MFLOPS. A special consideration is that memory is extremely limited; each Warp cell has only 32 Κ words of data memory, which allows storage of only one image on the entire array since all addressing is word-based and only one pixel can be stored in each word. However, the external host memory is much larger (up to 60 MB), and can store pixels four bytes to a word. (Special hardware is provided for converting images to and from packed storage as data is transmitted between the external host and the Warp cell array). This makes it possible to do large image warps on Warp

50

External host memory

External host

Warp * cell

Warp cell

Warp cell

Warp cell memory

Warp cell memory

t t

Warp cell memory

t

External host

Warp * cell 1 τ

Warp cell memory

Fig. 5. Simplified Warp Architecture.

provided only a portion of either of the images is stored in the Warp array at any one time. Times for the Warp machine are reported in (200 ns) cycles per pixel; for example, ten cycles per pixel means about 524 ms for processing a 5\2times5\2 image. The execution time of some methods were estimated, while others were actually implemented on Warp: the times below noted as 'executes' are the result of actual runs using the Warp machine. Warp times include the time to transfer the images to the processor array from Warp's external host. Later experiments, and all scaling experiments, were done on the Carnegie Mellon/Intel Corporation iWarp computer [2]. The experiments used an iWarp configuration with an torus of iWarp cells, each of which developed 20 MFLOPS (with no pipeline delay) and a corresponding number of MIPS. The torus thus produced 1.28 GLOPS. The current C compiler for iWarp typically allowed us to achieve between 100 and 200 MFLOPS in image processing tasks. Scaling experiments were done using three different image warps: rotate, which scaled the input image by shrinking the χ and y axes by Λ / 2 / 2 , and then rotated it 45°; shrink, which scaled the χ and y axes by 1/2; and expand, which scaled the χ and y axes by 2. Numbers reported for Warp are largely independent of the particular affine transformation used, except as noted. 5.

Classification of implementations

The implementations of affine image warping described in this paper differ primarily in how they partition the images in space and time. All parallel implementations of image warping must partition one of the images in space, in order to give dif-

51

ferent processors different work to do. Some parallel implementations may also partition the other image in space, or partition it in time, or duplicate it on all processors. Spatial partitioning is to be preferred since it allows the manipulation of large images without the restrictions and architectural requirements imposed by temporal partitioning, as discussed below. Processing large images is important because in many commercial applications of image warping the images are very large, from thousands to tens of thousands of pixels on a side. These images are too large to store in a single processor's memory. This is especially so since high-speed parallel computers (including Warp and iWarp) commonly use fast SRAM chips for memory, which are not dense, and rarely have attached disks to support virtual memory. 5.7.

Temporal

partitioning

In temporal partitioning an image is partitioned so that different sections of the image are processed at different times. Any image warping algorithm can be modified in this way to run on a machine with limited memory. In a forward algorithm, - The input image can be partitioned in time by warping each section of the image independently. If pixels in different sections map to the same output location, their values can be averaged. - The output image can be partitioned in time by discarding pixels that map to locations outside the current section, and completely recalculating the warp for each section. While in an inverse algorithm, - The input image can be partitioned in time by not calculating the output pixel value when the corresponding input pixel falls outside the current section, and completely recalculating the warp for each input section. - The output image can be partitioned in time by calculating only the appropriate input locations for each output pixel. Note that the cost of temporally partitioning the output in a forward algorithm, or the input in an inverse algorithm, is substantial and linearly proportional to the number of sections into which the image is divided, since the entire affine transformation must be performed for each section. The cost of partitioning the input in a forward algorithm, or the output in an inverse algorithm, is not nearly as significant, at most some extra bookkeeping and extra I/O overhead since more communication operations may take place. Efficiently dealing with the sampling issues discussed in Section 2 is difficult with temporal partitioning, because the image that must be accessed irregularly (i.e., a parallelogram of pixels must be accessed to map each input, or produced

52

Method

How is secondary image treated?

Lagrangean interpolation?

Sampling?

Systolic Data partitioned Scanline Sweep-based

Streamed Duplicated

Yes Yes

No Yes

Spatially partitioned Spatially partitioned

No Yes

Yes Yes

Table 1. Capabilities of implementations of parallel affine image warping.

each output) is the same image that is partitioned across processors in the efficient case; the output image in a forward algorithm, and the input image in an inverse algorithm. This complicates bookkeeping and leads to poor load balancing. Temporal partitioning imposes certain architectural requirements. The architecture must be such that the temporally partitioned image can be stored elsewhere while it is not being processed: either in an external memory, or within the processor array if the processors support parallel I/O while processing. 5.2.

Spatial

partitioning

As noted above, in all methods at least one image is spatially partitioned. The differences between the algorithms discussed below lie in how the other image (called the secondary image in the table below) is treated: it can be duplicated on all processors, treated as a stream, or spatially partitioned. (We do not treat temporal partitioning as a separate case because it applies to all algorithms.) This affects what pixels are stored locally at a processor at any given time, so that some methods can support Lagrangean interpolation in the inverse algorithm, and others cannot. Similarly, some methods can handle the sampling issue raised in Section 2 and some cannot. Table 1 summarizes these capabilities of the implementations of parallel affine image warping discussed here. 6.

Systolic methods

In systolic methods either the input or output image is spatially partitioned and the other image is fed across the array in a stream, as shown in Figure 6. As is usual with systolic methods, good use is made of memory. Very large images can be processed since no cell must store the entire image. As the stream arrives at processors, either the local state or the stream is modified, depending on whether the stream is the input or the output image. Systolic methods effectively partition one image in space and the other temporally. The trade-offs in forward versus inverse algorithms are exactly as discussed

53

Input image (if output is partitioned).

—•

Processor ιr

Input or output image swath

t

Processor

Processor

Inputor output image swath

Input or output image swath

—1

Output image (if input is partitioned).

ιr

Fig. 6. Systolic image warping.

in Section 5.1; if the input image is streamed (i.e., partitioned in time) then the forward algorithm should be used; if the output image is streamed then the inverse algorithm should be used. Also, as discussed in Section 5.1, dealing with the sampling issues discussed in Section 2 is difficult in these algorithms. 6.7.

Methods that partition the input image

We considered two different systolic implementations that partition the input image. They differ in the granularity of the output image portion processed at a time. In the first the granularity is one pixel, while in the second it is a band of rows. In the first method the output image is fed through the processor array in raster order, and as it arrives at each processor that processor checks to see if the input pixels needed to calculate this output pixel are in its memory. If so, the output pixel is updated, and passed from processor to processor until it flows out of the array. After the last processor, the output pixel either has its correct value, or zero if the corresponding input pixel lay outside the input image. We note that the input pixel address calculation is redundant, since it is fixed for any output pixel and does not depend on the processor. We therefore factor out this address computation and perform it in the first Warp cell in the Warp implementation of this algorithm. The address and output pixel value are then passed together through the processor array. This method takes advantage of the independence of the output to input mapping from input image values because only a single cell does the address calculation. It does not exploit locality; the input pixel addresses could be sent in a random order without affecting execution time. It executes at a speed of about twelve cycles per output pixel. This method would not work well with Lagrangean interpolation. Each output

54

pixel will be calculated by only one processor. Since Lagrangean interpolation is slow, that calculation will form a bottleneck in the Warp pipeline and only one or two processors will be active at any given time. We estimate this method would take about 250 cycles per output pixel on Warp. This method is not applicable to iWarp for two reasons. First, iWarp does not have an external memory in which large images can be stored and pumped through the processor array systolically. Second, iWarp has many more processing cells; without Lagrangean interpolation the available computation is too small to justify the use of so many processors, while with Lagrangean interpolation the bottleneck problems mentioned in the last paragraph are more severe. In the second method, a swath of the output image is replicated in the memory of every cell as it is pumped systolically through the array. For every pixel in the swath of the output image, each cell calculates the corresponding input pixel coordinate and checks to see if this input pixel is stored in its own local memory. If so, then that input pixel is stored into the output image swath. If not, then computation proceeds with the next output pixel. When a cell's work with an output image swath is done, it is sent to the next cell and the next output image swath is worked on, in a pipelined fashion. There are two problems with this method: First, much time is wasted because each cell repeats calculating the same input coordinates for every output pixel. (In the previous method, this computation was done only once, on the first Warp cell.) Also, this method suffers from load-balancing problems, due to locality. It is possible that only one cell would score 'hits' for each pixel in a output image swath. This will happen, for example, when the input and output images are distributed in the same way (e.g., both by row swaths) and the affine transformation is the identity. This is shown in Figure 7. The output image is distributed by row swaths, in the same way as the input image is pumped in. The only times useful work is done (shown by the vertical arrows from the data being pumped through to the data stored locally) is when each input swath arrives at the corresponding processor, in other words at times Τ = 0, Τ = 2, etc. In each of these working periods, only one processor is busy, Thus, no parallelism is achieved. We did not implement this method for image warping without Lagrangean interpolation, because of the load balancing problem. However, by modifying this method to improve load balancing, we obtain a good approach for image warping with Lagrangean interpolation. Instead of processing a contiguous swath of the output image at a time, we take ten rows (one for each Warp processor) distributed across the image. Because of locality, the corresponding input pixels for the output pixels in that swath will be evenly distributed across the array, so that good load balancing is achieved. This method still duplicates the input coordinate calculation across every cell in the array. However, this calculation consists of only a few floating-point calculations per output pixel - very few when compared to Lagrangean interpolation (about fifty floating-point operations). As

55

/

)

f

1

1

\

Image stored locally

/;

\

I I I I I fl Ik

I 111 I l/l I

I III I in I

I Processor!!"] | ProcessorT^ | Processor?] | ProcessorT| Data being pumped through

τ=ο I ΙΛ-f-kl Iτ=ι

I II I I I η

τ=2

ι

in ι

I

I

i/i ι

ι il I I i n

l/f+kl

•

ι ni ι i/i ι

ι il ι ι i n

ι ij-kli

Fig. 7. Image warping with swath distribution. a result, this method runs at a speed of about fifty cycles per output pixel, which is the the best time achieved on Warp. Performance of this method on iWarp is illustrated in Figure 8. As the graph clearly shows, this method does not scale well; in fact, scaling beyond 16 processors increases execution time. The method is inefficient here because of the problems mentioned above and because the output of the processors (the partial output images) must be combined with other processors for final output. This introduces synchronization between the processors, creating load-balancing problems, and also requires more time than on Warp because there are more processors which must combine their results. 6.2.

Methods that partition the output image

These methods are forward algorithms, and therefore cannot perform Lagrangean interpolation, as discussed in Section 2. In the first method, the input image is pumped through systolically and the output image is partitioned. As in the first input partitioning method, the first cell

56

Fig. 8. Scaling of input partitioned method on iWarp. Image size is 512 χ 512.

calculates the corresponding output pixel locations for each input pixel and pumps the coordinates along with that pixel through the rest of the array. When the input pixel is passing through the cell which holds the output pixel which corresponds to this input pixel, then that cell stores this pixel at the correct location in its swath of the output pixel. After the entire image has been generated, the output image is concatenated and passed out of the array to the external host memory in raster order. This method executes on Warp at a speed of about fifteen pixels per input pixel. In the second method, the entire output image is partitioned across the array. A swath of the input image is stored on every cell. A swath of the input image is processed at a time. Each cell receives the current swath, which is broadcast systolically. For every pixel in the swath of the input image, each cell calculates the corresponding output pixel coordinate and checks if this output pixel is stored in its own local memory. If so, then the input pixel is stored into that output image location. If not, then computation proceeds onto the next input pixel. When all of the input swaths have been processed, the output image is concatenated and sent out to the external memory in raster order. There are two main problems with this method: First, much time is wasted because each cell is calculating the same output coordinates for every input pixel. Second, this method suffers from

57

load-balancing problems due to locality, as illustrated in Figure 7. 7.

Data partitioned methods

In data-partitioned methods the input and output images are stored on the processor array and the warping is done at a different time than the input or output of data. In most cases, this means that one of the input or output images must be replicated on all processors, limiting the maximum image size to be the size that will fit in one processor's memory (though, of course, temporal partitioning can be used). 7J.

A forward

algorithm

In the forward algorithm we replicate the output image on all processors. We also replicate an additional "count" matrix, of the same size as the image, that keeps track of the number of input pixels that map to each output pixel. Each processor takes a portion of the input image and calculates, for its portion, the output coordinates corresponding to these pixels. The input pixels are then added to the corresponding output location, and the corresponding count element is incremented. After the entire input image has been processed, the output images are passed from cell to cell and added, as are the count images. The last cell divides the output image by the count to get the final image. As noted in Section 2, this method works only for affine warps in which the determinant of the homogeneous matrix is less than or equal to one. (By maintaining the count matrix and averaging, we correctly subsample the input image for contracting transformations.) With other warps, output pixels will be skipped. As this is a forward algorithm, it cannot be used with Lagrangean interpolation. On Warp this method executes in about ten cycles per pixel, which is near optimal. Each cell is calculating the warp over a different portion of the image, so there is no duplicated effort. The cells work independently except when the output images are added together, so it is not possible that one cell will delay the others. The only drawback to this algorithm, and it is a significant one, is that the entire output image must be stored at each cell. With Warp's limited memory per cell, the output image must be quite small. An Adapt implementation of this same mapping on iWarp gives very poor results, as illustrated in Figure 9. Here the output image is combined in a binary tree. (Combining in a binary tree is inferior to pipelined addition in the Warp implementation because processing and I/O are not overlapped between processors. However, Adapt does not support pipelined combining because it does not restrict the order of accesses to arrays.) This limits the size of output image that can

58

400.0 ο—Θ Rotate • — • Expand ο—OShrink

0.45

350.0 0.40

0.35

300.0

0.30 250.0 Τ 0.25 200.0 16

32

Processors

48

0.20 64

Fig. 9. mput partitioned image warping with Adapt. Image size is 128 χ 128. Solid curves give time, dashed curves give speedup. be produced to only 128 χ 128, and also completely dominates other processing, leading to slowdown with increasing processor array size. This method can be modified to address the sampling issues raised in Section 2. All that is necessary is to update all pixels that an input pixel maps onto, instead of only one. Before processing any pixels, each cell calculates the coordinates of the parallelogram that each input pixel maps onto. The shape of this parallelogram does not vary from pixel to pixel; only its position changes. Thus, in processing an input pixel each cell needs only to add its value to all of the corresponding output pixels, according to the previously calculated parallelogram. An implementation of this modified algorithm on iWarp gives similar scaling to that illustrated in Figure 9. 7.2.

An inverse

algorithm

In the inverse algorithm, we replicate the input image on all processors. Each processor takes a portion of the output image, and calculates the input coordinates corresponding to each of its output pixels. It then accesses its input image and stores the correct output pixel, using Lagrangean interpolation if desired. Without interpolation, this method also executes in about ten cycles per pixel

59 700.0

-; 15.0

600.0

(Λ

Ε

500.0

ω

Ε

400.0

300.0

200.0

8

16

32

ν ,0.0 64

Processors

Fig. 10. Scaling of input replicated, data partitioned image warping. Image size is 512 χ 512. Solid lines give execution time, dashed lines give speedup.

on Warp, once again near optimal. No cell duplicates the work of others, and each works independently, so there are no bottlenecks. Lagrangean interpolation is easy to introduce into this method since the entire input image is accessible at every processor. The biggest drawback of this method is that the entire input image must fit in a processor's memory. Overcoming this drawback through temporal partitioning leads to an algorithm that shows the best speedup so far, though poor scaling beyond 32 processors. Figure 10 shows the results of an implementation of this algorithm (including Lagrangean interpolation) on iWarp. In this implementation the input image is partitioned temporally; a swath of input rows, interleaved across the input image, is processed at a time. The speedup is limited by the number of different output rows that the input rows being processed in the current pass map onto. Thus, there are two contrary effects: as the number of processors increase, the processing power increases; but the number of rows per processor, and hence the maximum speedup, decreases. The speedup for contract is worse than for expand or rotate because the number of output rows for our contract image is half the number of input rows, while it is equal to or greater than the number of input rows for expand and rotate.

60

8.

A scanline method

In Catmull-Smith scanline methods, we make two passes over the image, skewing the image in each pass. The first pass skews the image horizontally; the effect is to align each of the image columns with their ultimate destination. The second pass skews the image vertically, and completes the warp by aligning each of the rows with their destination. (The order of the two passes must be exchanged if the image is being rotated by more than 45°.) It has been shown that any image warp, including higher-order warps, can be decomposed in this way [5]. In an implementation of this method, we distribute both the input and output images across processors. We process the image in two identical passes (except that the skew parameters change.) In each pass, the input image is read in transposed order, so that each cell gets a swath of columns of the input. Each cell calculates the skew for its columns, and stores the input into the appropriate place in the output image. The output image is stored into the external memory, and becomes the input for the next pass. The Warp implementation of this algorithm suffers in comparison to the others because two passes are required through Warp's external host, in which access to the relatively slow DRAM memory in transposed order is a bottleneck. We estimate that the algorithm will take approximately seventeen cycles per pixel without Lagrangean interpolation. The iWarp implementation of this algorithm gives only moderate speedup, as shown in Figure 11, and poor performance overall given that this algorithm does not perform Lagrangean interpolation. There are two reasons for this: (1) Since the algorithm performs two passes, the mapping from input to output must be done twice, doubling total computation; (2) The transpose step is fairly expensive., and dominates computation, especially for small images and large processor arrays. The effect of transpose on execution in the iWarp scanline-transpose algorithm is shown in Figure 12. On iWarp, the image is stored distributed on the processor array by row swaths. Transposing the image thus requires that each processor send some of its data to every other processor. Each processor forms messages to be sent to all other processors; the messages are rotated horizontally through the processor array until they reach the right processor column, and then vertically until they reach the right processor. This implementation does not take advantage of iWarp's systolic capabilities or its logical channels, but it is easier to program than a more efficient method. The result is that for small images (e.g., 128 χ 128) the message passing overhead on large processor arrays increases transpose time, and dominates overall execution time. With large images performance is improved as processor array size increases, because the formation of the messages, which is done in parallel, dominates total computation time.

61

Fig. 11. Execution time, scanline-transpose algorithm.

9.

A Sweep-Based Method

We are developing a new parallel prograrnming model based on encouraging the user to access data structures in a way which is amenable to efficient implementation on large processor arrays. We call this the sweep model because control information is intended to sweep across the data structures. A sweep is a procedure which accesses the elements of a data structure in monotonically non-decreasing order. In the sweep model, large data structures are distributed on a processor array and sweeps are sent through the processor array in a pipelined fashion. As the sweeps arrive at a processor computation is done. Alternatively, the sweeps may execute to completion at the processor which invoked them while the data structure is pumped through the processor array. As the elements of the data structure flow through each processor computation is done. There are two advantages of this model as we have implemented. First, it is natural. It uses the notion that in many do-loops over an array, accesses to the array occur in monotonically increasing order. Other than this, it does not impose any programming constraints on the user. Second, it is efficient. Because array accesses are always monotonically increasing, the sweeps can be set up as a

62

Fig. 12. Breakdown of execution time in scanline-transpose algorithm.

63

pipeline over the array of processors, which can be a very efficient model if there is enough work to fill the pipeline and the state describing the computation to be done at each processor is not too large. In the sweep-based algorithm for performing affine image warping with Lagrange interpolation, both input and output images are partitioned across the processor array. Each processor maps their own output image swath into the input image space and calculates the set of input pixels which will be needed to perform the image warping operation with interpolation (in order to minimize the computation time, this step exploits the fact that in affine image warping, parallelograms in the output space map to parallelograms in the input image space). Each processor then invokes a sweep which accesses each row of the input image and stores the necessary input pixels in a separate data structure. In our implementation model, the input image is pumped through the processor array and each processor locally stores the necessary input pixels. Processors can then perform the affine image warping and Lagrange interpolation on their own output image swath in parallel. Using this method it is straightforward to implement a load balancing technique not used in any of the other implementations of affine image warping discussed here. We note that the work per output image row is not uniform, but depends on the number of pixels that must be processed on that row. This work load can be calculated from the parameters of the image warp before any other work is done. We can then assign rows of the output image to processors so that the workload on each processor is uniform. For example, in an inverse algorithm for the rotation parameters, processors near the top and bottom of the image will get several rows, while processors near the middle of the image will get only a few. Then, while distributing the input image, each processor can take only the portion of the image it needs, given its portion of the output image. This leads to significantly improved performance on affine image warping with Lagrangean interpolation, and the best performance and speedup overall, as illustrated in Figure 13. In the best case (expand) we achieve a speedup of over 50 on a 64-processor, which is far better than any other algorithm reported here; similarly, execution times are the best reported here. This method shows the best scaling of any parallel warping technique discussed in this paper. The limits of its scaling capabilities can be calculated from a simple model. The sweep method treats the processor array as a pipeline, and loads it with pixels from the image. If there are Ν pixels in the image, this will take NI steps, where / is the I/O time for one pixel. Let the time to do Lagrangean interpolation for one output pixel be L and the number of processors be P. The total compute time on Ν processors is then NL/P + NI. The speedup over one processor is therefore NL

_

NL/P + N1 ~

L L/P+I

64

Fig. 13. Performance of sweep-based algorithm. Image size is 512 χ 512. Solid lines show execution time, dotted lines give speedup.

65

For large numbers of processors, the limit is L/I: in the case of the iWarp implementation, this ratio is approximately 200, so this is the maximum speedup. This implementation does not address the sampling issues raised in Section 2, but it is not hard to modify it to do so. The cost function that assigns work to a processor does not change, since the amount of work per pixel is constant in affine image warping. The code that takes the appropriate input image pixels for the assigned output image would be altered to take a window dependent on the affine image parameters. 10.

Conclusions

This study considered a variety of methods for implementing affine image warping in parallel: -

Systolic Data partitioned Scanline-transpose Sweeps

Key issues in the evaluation of these methods are efficiency, capabilities, and memory use. Efficiency relates to the execution time of the algorithm and its scaling with processors. Capabilities relate to where interpolation can be done using the algorithm, and whether it can deal with the sampling issues raised in Section 2. Memory use relates to how the images are partitioned on the array, and whether they can deal with large images without requiring excessively large processor memories. The systolic methods studied here use memory efficiently (except that they require a large memory for the source or sink of images), but they do not scale well and are limited in capability. Except in specialized situations (as, for example, the Warp machine, which had very limited per-processor memory but a large external memory) they are not useful. Data partitioned methods show promising efficiency for processor arrays of up to 32 processors but poor scaling beyond that. Memory use can be good with an appropriate combination of temporal and spatial partitioning of the images. Interpolation is easy to incorporate but temporal partitioning limits the ability to deal with sampling issues. Note that data partitioned methods are easy to implement, and for small input or output images they may be the most efficient and flexible. Scanline-transpose methods show promise but need further development to be competitive with the other methods. They make especially good use of memory, partitioning both images at all times, but the transpose step is a bottleneck in efficiency. The scanline-transpose method considered here does not do Lagrangean interpolation, so that the extra computation required because of its two passes and

66

the transpose step makes it inefficient. Sampling seems to be easily addressed in scanline-transpose methods. If the transpose bottleneck can be overcome, and interpolation addressed, scanline-transpose methods may become some of the best performers. The sweep-based method shows much better performance than any other technique, as well as excellent scaling up to a maximum speedup of approximately 200 (based on the compute time from the iWarp). The virtue of the sweep method is that it takes an irregular problem, and turns it into an efficient pipelined algorithm. Moreover, this technique makes it easy to perform good load balancing, which is especially important to get good efficiency with large processor arrays. The sweep-based method is more complex than the other techniques, but since we believe that sweep techniques will be applicable to a wide variety of problems implementation complexity can be amortized and the difficulty of any particular implementation reduced. References [1] Annaratone, M, Amould, E., Gross, T., Kung, H. T., Lam, M, Menzilcioglu, O. and Webb, J. A. (December 1987). The Warp Computer: Architecture, Implementation and Performance. IEEE Transactions on Computers, C-36 (12), 1523-1538. [2] Borkar, S., Cohn, R., Cox, G., Gleason, S., Gross, T., Kung, H. T., Lam, M., Moore, B., Peterson, C , Pieper, J., Rankin, L., Tseng, P. S., Sutton, J., Urbanski, J. and Webb, J. (November 1988). Warp: An Integrated Solution to High-Speed Parallel Computing. Proceedings of Supercomputing '88. Orlando, Florida: IEEE Computer Society and ACM SIGARCH. [3] Faux, I. D., and Pratt, M. J. (1979). Computational Geometry for Design and Manufacture. Ellis Horwood. [4] Fisher, A. J. and Higihnam, P. T. (December 1987). Computing the Hough transform on a scan-line array processor. Computer Architecures for Pattern Analysis and Machine Intelligence. Seattle, Washington: IEEE Computer Society. [5] Wolberg, G. (1990). Digital Image Warping. 10662 Los Vaqueras Circle, P.O. Box 3014, Los Alamitos, CA, 90720-1264: IEEE Computer Society Press.

Parallel Processing for Artificial Intelligence 1 L.N. Kanal, V. Kumar, H. Kitano and C.B. Sultner (Editors) © 1994 Elsevier Science B.V. All rights reserved.

67

Chapter 4

Image Processing On Reconfigurable Meshes With Buses* Jing-Fu Jenq Tennessee State University and Sartaj Sahni University of Florida

Abstract In this chapter, we describe different reconfigurable mesh with buses architectures and show how several image processing problems can be solved efficiently on the weakest of these. The specific problems considered are: area and perimeter of components, shrinking and expanding, clustering, and template matching. In many cases, the resulting algorithms are faster than those for other parallel computer architectures.

1.

Introduction

Recently, several researchers have proposed a modification to the well studied mesh architecture in which the interprocessor links are replaced by a reconfigurable bus. The resulting parallel computer architecture is called a reconfigurable mesh with bus (RMB). In all two dimensional Ν χ Ν RMB com2 2 puters, the N processors are located at the N grid points of an Ν χ Ν grid (just as in a traditional mesh computer). However, the traditional linkage between mesh adjacent processors is absent Instead, interprocessor communication takes place via a reconfigurable bus. The RMB family of architectures includes the RMESH, PARBUS, polymorphic torus, and the mesh reconfigurable network (MRN). The architectures have become popular because they are ralatively easy to program and because many problems can be solved very efficiently on them. In fact, it is possible to solve some problems faster on an RMB computer than is theoretically possible on a PRAM computer (See for example: [ 1 0 , 3 4 , 3 5 , 3 9 ] ) . •This research was supported, in part, by the National Science Foundation under grant MIP-9103379.

68

In the RMESH [53, 26, 27, 28] version of an RMB (Fig. 1), the bus is T comprised of 2/v (N-l) segments that correspond to the 2N(N-l) interprocessor links in a traditional mesh. However, each of these segments has a switch on it. The switch on a bus segment may be set in the open or closed position by either one of the two processors at the ends of the segment. With the switch on each 2 segment closed, all N processors are attached to the same bus. As a result, if any one processor writes data to this bus, all remaining processors can read this data from the bus in the next cycle. I.e., it is possible to broadcast data in 0 ( 1 ) time.

mοmom

•Φ o • ο •

ο [] ο

φ

•

φ

φ

ο [] ο [] ο Π

φ

•

•

φ

ο •

ο •

I

I : Processor

Ο : Switch — : Link

φ

ο

•

Fig. 1 A 4 X 4 R M E S H By opening the switches on all vertical bus segments and closing them on all horizontal ones, we form Ν independent buses with each one spanning the processors in a row of the mesh (Fig. 2(a)). Column buses that span all processors in the same column may be formed by closing all switches on column segments and opening all on row segments (Fig. 2(b)). As in other work dealing with the RMESH model, we assume that the the time to broadcast data on a bus or subbus is 0 ( 1 ) . In the exclusive write model, only one processor can write data to a given (sub)bus at any time. In the concurrent write model several processors may simultaneously write to the same (sub)bus. Rules are provided to determine which of the several writers actually succeeds (e.g., arbitrary, maximum, exclusive or, etc.). The PARBUS of [51,52] is also a member of the RMB family. AnNxN PARBUS (Fig. 3) is an Ν χ Ν mesh in which the interprocessor links are bus segments and each processor has the ability to connect together arbitrary subsets of the four bus segments that connect to i t Bus segments that get so connected behave like a single bus. The bus segment interconnections at a p r o c e s sor are done by an internal four port switch. If the up to four bus segments at a processor are labeled Ν (North), Ε (East), W (West), and S (South), then this switch is able to realize any set, A = {Au A 2 } , of connections where A, ç {NJE,W,S}, 1 < i< 2 and the Λ,-'s are disjoint. For example A = {{N,S ] , {E,W}}

69

•

Ο

•

Ο

•

Ο

•

•

Φ

•

•

Ο

Ο

•

•

Ο

Ο

•

Ο

•

Ο

•

•

m •

Ο

•

Ο

•

Ο

•

• Φ

m

m m

φ

Φ

φ

Φ

φ

m

C)

•

•

•

m m

(b) Column buses

(a) Row buses

Fig. 2 Subbuses in an RMESH results in connecting the North and South segments together and the East and West segments together. If this is done in each processor, then we get, simultaneously, disjoint row and column buses. If A = {{N,S },}, then only column buses are formed. Similarly, when A = {{E,W}, φ) only row buses are formed. If A = {(N,SJE,W)/|)}, then all four bus segments are connected. PARBUS algorithms for a variety of applications can be found in [29, 5 2 , 2 3 , 1 0 , 1 1 , 1 2 , 13, 36, 37, 3 8 , 4 9 ] . Observe that in an RMESH the realizable connections are o f t h e f o r m A = {AiJ.Aj c {N,E,W,S}.

Fig. 3 A 4 x 4 PARBUS The polymorphic torus and the mesh reconfigurable network (MRN) are

70

two other members of the RMB family. The polymorphic torus architecture [21, 22] is identical to the PARBUS except that the rows and columns of the underlying mesh wrap around and it possible able to connect together arbitrary subsets of the bus segments that connect to the processor. Specifically, in each row, there is an additional bus segment that connects the rightmost port in the row to the leftmost port in the row and in each column there is a bus segment that connects the lowest port in the column to the topmost one. In an MRN [2], the processor and bus segment arrangement is exactly as for the PARBUS. However the switches internal to processors are able to obtain only the 10 bus configurations given in Fig. 4. Thus an MRN is a restricted PARBUS.

S

S

Fig. 4 Allowable switch configurations in an MRN While we have defined the above reconfigurable bus architectures as square two dimensional meshes, it is easy to see how these may be extended to obtain non square architectures and architectures with more dimensions than two. In this chapter, we shall confine ourselves to image processing algorithms that run on an RMESH. While these same algorithms can be run on a PARBUS, polymorphic torus, and an MRN with no loss in run time, one can often obtain faster algorithms for these other architectures. The reader is referred to the cited references for work on these other architectures. MacKenzie [MACK93] has shown that there are problems that can be solved faster on the other models than on the RMESH. Furthermore, Jang et al. [13] have shown how many (though not all) of the connections supported by the remaining models can be simulated by an RMESH with no loss in run time. However, the simulating RMESH needs 16 processors for each processor in the simulated model and the required I/O configuration must be modified so that data is at just one processor in each 4 x 4 group of RMESH processors. In this chapter, we shall study RMESH algorithms for the following image processing applications: area and perimeter of components, shrinking and expanding, clustering, and template matching. RMESH algorithms for

71

histogramming, histogram modification, and the Hough transform can be found in [17] and [15]. It should be noted that Jang et al. [11] have developed a histogramming algorithm for a PARBUS. They state that the algorithm can be simulated on an RMESH. However, this simulation requires a 16 fold increase in the number of processors and requires that the input be properly configured. To obtain the initial configuration from one in which the Ν χ Ν image resides in an Ν χ Ν block of the 4N χ 4N RMESH requires 0(N) time as the bandwidth available for the rearrangement is only 0(N). This rearrangement time exceeds the total run time of our algorithms which require only an Ν χ Ν RMESH. RMESH algorithms for some other image processing problems can be found in [ 2 6 , 2 7 , 2 8 ] . [34] develops constant time RMESH algorithms for some problems from computational geometry.

2· 2.1.

Data Manipulation Operations Data Diagonalization

In this, a specific row or column of elements is moved to the diagonal positions of the window which contains that row or column. This is illustrated in Fig. 5. This can be accomplished in 0 ( 1 ) time by broadcasting the row (column) along column (row) buses and having the diagonal processors read the data from their bus.

1

1

3

3

5

5 1

3

5

4

2

4

4

2

2 (a) 4th row

(b) 1st column

(c) diagonalize

Fig. 5 Diagonalize 4th row or 1st column elements of a 5x5 window

2.2.

Window Broadcast

The data to be broadcast is initially in the A variable of the PEs in the top left wxw submesh. These PEs have ID (0,0) .. The data is to tile the whole mesh in such a way that A (ij) = A (i mod wj mod w) (A (/,;) denotes register A of the PE with ID ( / J ) ) . This can be done in 0 (w) independent of the size of the RMESH [18].

72

2.3.

Prefix Sum a

2

Assume that N values A 0,Ai,....,Atf -i are initially distributed in the A variables of an NxN RMESH such that A ( / J ) = A ^ + y , 0 < i j , < N. PE (i J ) is to compute a value Swm (i J ) such that iN+j

Sum(iJ)=

£A*,

0 " are variables.

(p PA ( O P E N < s e a t > ) ( W A N T S < p a s s e n g e r > )

—•

( R E M O V E 1 2) ( M A K E RESERVATION < p a s s e n g e r > < s e a t > ) )

Rule PA matches whenever the W M has two W M E s : ( O P E N < s e a t > )

and

(WANTS < p a s s e n g e r > )

where t h e value of " < f l i g h t > " is the same for both. If PA matches and executes, it deletes (i.e., R E M O V E s ) the two W M E s it matched (1 and 2 refer to t h e first and second W M E matched on t h e LHS) and adds (i.e., M A K E s ) t h e following W M E to W M : (RESERVATION < p a s s e n g e r > < s e a t > ) A C E can also be negated, which means t h a t t h e C E is true if and only if no W M E matches it. Rule P B is a rule t h a t will make an airline reservation for an employee only when no regular passenger wants one.

164

(p P B ( O P E N < s e a t > ) ( E M P - W A N T S ) - ( W A N T S ) ( R E M O V E 1 2) ( M A K E RESERVATION < s e a t > ) ) T h e algorithm for a typical serial PS has the following three step loop. M A T C H : T h e LHS of each rule is compared against the entire W M to determine all sequences of W M E s t h a t match the rule. Each such match results in an instantiation, which is a d a t a structure consisting of the rule plus the sequence of W M E s matching the unnegated C E s of its LHS. A single rule can match against multiple sequences of W M E s ; i.e., can have m a n y instantiations. T h e set of all instantiations is called the conflict set. S E L E C T : If the conflict set is empty, the PS halts. Otherwise, exactly one instantiation is selected according to some predefined conflict resolution strategy. A C T : T h e actions on the RHS of the rule for the selected instantiation are performed. Figure 1 shows how rules PA and P B match against a W M and execute. There are two very efficient algorithms for matching, namely R E T E [6] and T R E A T [21, 22]. Both assume t h a t there will be a small number of changes to W M and, hence, to the conflict set. Both keep the results of previous partial matches and use t h a t information to u p d a t e efficiently the set of matching instantiations. T h u s , both algorithms combine the A C T and MATCH steps. In our synchronous parallel model, presented in the next section, we incorporate this combined step. Much of the research on parallel PSs focuses on the match problem. Namely, how can a multiprocessor perform the match more quickly? We will not examine this topic, b u t will assume t h a t a parallel match algorithm is available. We briefly review this literature in Section 8. PSs are considered to be forward chaining rule systems because the rules are used in a forward direction. Namely, a rule is invoked according to its LHS and later executed by performing the actions on its R H S . Forward chaining systems are also called data driven because the d a t a determines the rules t h a t can be executed. This is in contrast to backward chaining rule systems, such as P R O L O G [3].

165

Initial WM =

{ (OPEN 6A 414) (OPEN 8C 414) (WANTS Joe 414) (EMP-WANTS Jane 414)}

Initial Conflict Set ={ (PA (OPEN 6A 414) (WANTS Joe 414)) (PA (OPEN 8C 414) (WANTS Joe 414))} Execute first instantiation. NextWA =

{ (OPEN 8C 414) (EMP-WANTS Jane 414) (RESERVATION Joe 414 6A)}

Next Conflict Set ={ (PB (OPEN 8C 414) (EMP-WANTS Jane 414))}

NextWA =

Ϋ Execute the only instantiation. { (RESERVATION Joe 414 6A) (RESERVATION Jane 414 8C)}

Next Conflict Set ={} Fig. 1. Executing the Airline Rules: PA and PB

1.2.

A Model of Synchronous

Parallel Rule

Execution

We assume t h a t the W M is a set - it does not have duplicate W M E s . O P S 5 and other systems differ in t h a t the W M is a multiset - duplicates are allowed. We discuss the impact of this difference at the end of Section 5. T h e basic loop for our synchronous parallel model is as follows. S E L E C T : T h e processors jointly select a set of instantiations in the conflict set to be executed. T h e instantiations are selected such t h a t their co-execution is serializable. If there are no instantiations then exit. A C T / M A T C H : T h e processors jointly perform the actions on the RHSs of the rules for the selected instantiations and u p d a t e the conflict set for the next cycle. We refer to this model as synchronous same step at the same time.

because all processors execute the

Since we are co-executing instantiations, we place additional requirements on the matcher. Assume t h a t we will co-execute instantiations 1 n 3 3 i ,..., i where each i entails taking actions A\ ,..., A . T h e net effect

166

of co-executing these instantiations must be equivalent to taking all the 3 3 actions in some serial order such t h a t for each i \ A\ executes before A 2) 3 3 A 2 executes before A 3, etc. This mini-serialization problem is considerably easier t o solve t h a n the larger serialization problem t h a t we are examining. To our knowledge, all of the parallel match algorithms, which we will briefly review in Section 8, satisfy it. Moreover, the matcher must be able to handle duplicate actions among the instantiations being co-executed. For example, two different instantiations may each add the same W M E . In order to make the parallel matcher simpler to write, we restrict a set of instantiations such t h a t there is no W M E t h a t is both added and deleted. This could lead to race conditions with uncertain results. We will soon formalize this requirement. We have chosen a syntax t h a t is more amenable to formal analysis t h a n the syntax of 0 P S 5 . (Our implementation, however, accepts O P S 5 syntax.) T h e general form of a rule is as follows. A . unA nA · l ι· · ·^ m A

Λ

aA -^1 ' · ·

aA 'ΐ^ηΑ'

— A is the n a m e of the rule. — Each is a C E with a sign of or "—" and a formula using an 0 P S 5 - l i k e syntax. Negative CEs can use either bound or free variables. T h e bound variables are those t h a t appear in some positive C E . AU other variables are free. Free variables can appear in either CEs or actions, have implicit universal quantification, and have a scope t h a t is limited to the C E or action in which they appear. For example, variable of rule P B in Section 1.1 is free. All other variables in P B are bound. The interpretation of P B is t h a t for a given < s e a t > , and < e m p l o y e e > , there is no < p a s s e n g e r > who WANTS the same . — Each is an action. In this paper, we address only add and delete actions (see [34] for discussion of I / O and CALL actions). Each such action has a sign and a formula. indicates an addition of a W M E and "—" indicates a deletion. Unlike 0 P S 5 , we do not require t h a t a W M E being deleted have been matched by the rule's LHS. Free variables can appear in delete actions, which implies t h a t all matching W M E s are deleted. Free variables cannot appear in add actions. (However, they can appear in positive CALL actions; thus our formal t r e a t m e n t in Section 4 allows them in add actions - again, see [34].) T h e action of modifying a W M E is left out since it is equivalent to deleting, then adding, a W M E . Overall, the functionality of our system includes t h a t of O P S 5 . Our exten-

167

sions beyond 0 P S 5 are allowed for the sake of generality; we do not suggest t h a t they are preferable. l.S.

A Model of Asynchronous

Parallel Rule

Execution

Our asynchronous parallel model is similar to t h a t for synchronous execution, except t h a t each processor performs the basic loop at its own pace. T h u s , at any point in time, any number of processors might be performing the S E L E C T step, and others might be performing t h e A C T / M A T C H step. All other requirements of the synchronous model, particularly those we placed on the matcher, apply to the asynchronous model. We must carefully define the concept of co-execution for the asynchronous model. Let t h e time of execution of an instantiation begin when the system commits t o executing it and end when the system finishes executing its RHS and when all of its effects have been propagated. We say t h a t two instantiations co-execute if their times of execution overlap. 2.

T h e Serialization P r o b l e m

A parallel execution of a set of instantiations is serializable if and only if the result produced by the parallel system could be produced by executing the same instantiations in some serial order. T h e goal of serialization is to ensure t h a t every parallel execution by our system is serializable. There are two factors t h a t can cause non-serializability. T h e first is disabling. One instantiation disables another if executing the first would cause the second t o m a t c h no longer. This occurs if the first one deletes (adds) a W M E t h a t the second one matched positively (negatively). If one co-executes a set of instantiations t h a t contain a cycle of disabling relations, then the result may not be serializable. T h e second factor is clashing. One instantiation clashes with another if executing one would add a W M E t h a t the other would delete. If one co-executes a pair of instantiations t h a t clash, then the result may not be serializable. Figure 2 gives a P M and an initial W M . T h e instantiations t h a t match contain a cycle of disabling relations. T h e figure shows the results of three possible execution p a t h s : (1) executing P i ' s instantiation, (2) executing P2's instantiation, and (3) co-executing both instantiations. T h e third p a t h produces a result t h a t could not be produced by either of the other (serial) execution p a t h s . T h u s , this co-execution is not serializable and should not be allowed.

168

PM=

Pl:+(A), P2: +(B ),

- (C ) -» +(C ), +(D ). - (C ) -> +(C ), +(E ).

execute PI 7w7/a/WM={(Al)(B2)} Both PI & P2 match

|WM={(A1)(B2)(C 1)(D1)) \P2 no longer matches WM={(A1)(B2)(C1) ( E l ) } PI no longer matches

co-execute PI & P2

WM={ (A 1) (B 2) (C 1) (D 1) (C2)(E2)}

PI & P2 cannot be executed serially, as each disables the other. The result of co-executing them is thus unattainable by any serial execution. Fig. 2. A cycle of disabling means that P i and P2 should not co-execute

PM = PI : +(A ), - -CD ), +(E ).

execute PI initialWM= {(A1)(B 1)(D1)} Both PI & P3 match

execute P3

co-execute P1&P3

WM={(A1)(B 1)(C1)(D1)} P3 no longer matches

WM={ (A 1) (Bl) (El)}

execute PI ^

WM={ (A 1) (B 1) (C1)(D1) (El)}

The resulting WM is ambiguous either WM={(A 1) (B 1) (C 1) (D 1) (E 1)} or WM={(A 1) (B 1) (C 1) (E 1)}

The result of co-executing PI & P3 is ambiguous given the clash. Of the possible results, only one is serializable, namely, the one obtained by executing P3 followed by PI. Our parallel model, however, could produce the non-serializable result. Fig. 3. Clashing makes the result of co-executing P i and P3 ambiguous, so they should not co-execute

169

PM = P4: +(A ) -> +(C ), +(D ). P5: +(A ) -> -(C ), - ) and t h a t C = ( O N < x > B2). These two formulas cannot unify because < x > cannot be equal to b o t h BI and B2, which are distinct constants. But indeed, it is possible for this A to disable this C. Imagine t h a t A was in rule R l and a match of R l led to < x > = B 2 . At the same time, C was in R2 and a m a t c h of R2 led to < x > = B l . Since we have two different instantiations, we get two different sets of bindings, and can have two different bindings for < x > . T h e use of IVUnify avoids these mistakes by eliminating all variable overlap. miss any needed synchronizations, it sometimes synchronizes unnecessarily due to this limitation.

173

P r o o f : We begin with the "if" direction by letting the right side be true. Let C be the result of renaming t h e variables in C according to 6'. Note t h a t f Form(C) and Form(C ) m a t c h exactly the same set of ground formulas (i.e., W M E s ) . We form a new substitution list, 6g, as follows. For each mapping of a variable to a constant in 6, 6g has the same mapping. For each mapping of a variable to a variable in 6, 6g m a p s the former variable to a constant in a manner t h a t is consistent with δ (i.e., if two variables m a p to the same third variable in 6, then those two variables m a p to the same constant in 6g). T h e resulting 6g is subsumed by i , and as such, Unify(Form(A)iForm(C),6g). Also, the result of substituting either formula according to 6g results in a , where ground formula. In other words, ^(Form(.4), 6g) = Φ(Εογπι(0 )}69)1 (Form(A),6g) is a ground formula. Let this resulting ground formula be W. \iSgn{A) is " + " , then Sgn{C) is and it is possible for A t o add W and for C to test negatively for W. T h u s , A possibly disables C. Similar arguments apply if Sgn(A) is "—". For the "only if" direction, we let A possibly disable C Assume t h a t Sgn(A) is " + " . T h e n , there exists a W M E W t h a t A can add and C can test negatively for. We rename the variables in C such t h a t there is no overlap 1 with the variables in A. T h i s results in δ and C , as in the preceding. Since A adds W, then there exists a 6a such t h a t Unify(Form(A)> W\δα), and by definition, (Form(A),6a) = W. Similarly, there exists a 6C such t h a t Unify(Form(C), W,8C) and (Form(C'),6c) = W. Now, let 6 be the union of 6a and 6 C . Since A and C share no variables, neither do δα and 6C} so there is no conflict in forming 6. Furthermore, since 6 is the simple union, (Form(A),6) = (Form(C'),6) = W. T h u s , Unify(Form(A),Form(C'),6) l is true and, moreover, lVUmfy(Form(A))Form(C)i6,6 ) is true. Since A adds W while C tests negatively for it, then Sgn(A) φ Sgn(C). Similar arguments apply if we assume t h a t Sgn(A) is "—".• Given the above, we now present a test for whether one instantiation disables another. Theorem

2:

\ßj,k,6,6')

Executing Sgn(Af)

i

A

disables i

B

iff

φ Sgn(Cf )Λ A

P r o o f : First, we note t h a t {Form(Af), i ) results in a formula in A B IVUnifyMFormiAf),i ),^(Form(Cf),i ),6,6') J which all b o u n d variables in Form{Af) are replaced by constants and Β Β all free variables remain. Similarly for φ{Εοτπ\{0 )}% ). Second, A A (Form(A )y t ) is a p a t t e r n t h a t matches against all and only the W M E s A t h a t Af will add or delete when executed with instantiation i . SimiB B larly, (Form(C ), i ) is a p a t t e r n t h a t matches against all W M E s t h a t

174

Action in A - ( P x) - ( Q 1 x) +(P *) + ( Q * y) +(Q ι * ) +(Q ι * )

CE in Β +(Q V ') HQ 2 y) - ( P y) - ( Q * y) - ( Q y ζ) - ( Q y ζ)

Free Vara

y ζ

x, y

A disables Β never never always sometimes sometimes always

Notes 1 1 2

3 4

5

(1) The formulas do not match. (2) 6 = {x/y}

and y is free.

(3) 6 = 0, so it is not "always disables." (4) δ = {l J y ,x j ζ} and neither 1 nor y are free vars. (5) 6 = {l/y1x/z}

and y and χ are free vars.

Table 1. Examples of Types of Disabling Relations

B

B

could have matched C positively or negatively when i was created. T h u s , the intersection of W M E s matching them both is non-empty iff A B A In such cases, i disIVUnify&(Form(Af),i )^(Form(C?),i ),6}6'). B ables i . A B For the reverse direction, we let i disable i . By definition, there must A B be some j and k with Sgn(A ) φ Sgn(C ) and where the W M E s affected by the action overlap with the W M E s affected by the C E . In other words, the above IVUnify must be t r u e . ü We categorize a rule A as either never, always or sometimes rule B.

disabling a

— A never disables Β when no instantiation of A can disable any instantiation of B. — A always disables Β when every instantiation of A disables every instantiation of B. — A sometimes disables Β when neither of the two preceding conditions are true. We say t h a t one rule possibly disables another iff the former rule either always or sometimes disables the latter rule. Table 1 shows some examples. For each, we show an action from rule A, a C E from rule Β and the free variables for both rules A and B. We then show whether A disables Β and provide a brief note as to why. T h e notes are with respect to the tests given in Observation 1 below.

175

O b s e r v a t i o n 1: Tests for each of the three inter-rule are given by the following.

disabling

relations

- A never disables Β iff there is no j and k such that Af

possibly

disables Cf. 1 - A always disables Β iff there exists a j , k, δ and δ such that Sgn(Af)

φ Sgn(C?)

Λ lVUmfy(Foim{Af),

Form(C?),

1

6, δ )

where δ meets the following requirement. Let δ = {xi/yi,..., xn/yn} represent the substitution of xj by yi, etc. The requirement is that for n a all I, 1 < I < f t least one of x\ or y\ is a free variable. Of course, we need to take the variable renaming performed by IVUnify into ac1 / ; count, and so we actually test whether each φ^ (χ^6 ) or φ~*(y/,£ ) is free. - A sometimes disables Β iff the two preceding tests are false.

T h e presence of the "sometimes disables" relationship indicates t h a t a static analysis alone cannot precisely determine when an instantiation of one rule will disable t h a t of another. If we were to perform rule level synchronization (as did I&S and others), which means t h a t we would not perform runtime tests for disabling, then we would be forced to treat "sometimes disables" as "always disables". In other words, if it were possible for one instantiation to disable another, then we would have to assume t h a t it always would. This will yield serializable results, b u t at the expense of many unnecessary synchronizations and reduced speedup. Instead, we will synchronize at the instantiation level, and thus require a precise determination of when one instantiation disables another. Therefore, for each pair of rules, say A and B, we perform a static analysis off-line and compile a function t h a t is true iff an instantiation of one will disable an instantiation of t h e other. If A never disables 2?, the function always returns faise. If A always disables B, the function always returns true. Otherwise, the function compares the values of certain bound variables. We will not give the compilation algorithm here (see [34] for further discussion). However, its essence is to take the substitution list t h a t results from unification (which is performed off-line), formulate tests between the attributes of the two instantiations, t u r n those tests into a function of two arguments (one argument for an instantiation of each rule), and compile it using the s t a n d a r d compiler.

176

4.2.

Clashing A

B

We say t h a t an instantiation i clashes with an instantiation i iff exeA B A cuting b o t h from some W M where both i and i match will have i add B 2 (delete) a W M E t h a t i will delete ( a d d ) . We say t h a t one action possibly clashes with another action iff the first can add (delete) a W M E t h a t t h e second can delete ( a d d ) . T h e test for this condition is similar to t h a t for disabling. Theorem

3:

Sgn(Ax)

Action Α χ possibly clashes with action A2 φ Sgn(A2)

Λ [3ί, δ'] [IVUnifyiFormiA^,

iff

Form(A2),

δ, δ')]

T h e proof is similar to t h a t for Theorem 1. We now define a test for whether one instantiation clashes with another. Theorem

4:

i

A

clashes with i

Γ η , * Λ A ' l i Sgn(Af) Wik ö > >°l[

B

iff B

φ Sgn(A )A A B B lVUnify{{FoTm{Af),i ),{Form(A ),i ),δ,δ')

T h e proof is similar to t h a t for Theorem 2. We categorize a rule A as either never, always^ sometimes, or possibly clashing with a rule Β in a fashion similar to disabling. O b s e r v a t i o n 2: Tests for each of the three inter-rule are given by the following.

clashing

relations

— A never clashes with Β iff there is no j and k such that Af possibly B clashes with A. — A always clashes with Β iff there exists a j , k, δ and δ' such that Sgn{Af)

φ Sgn(A?)

Λ IVUnify(Form(Af),

Form(A%),

δ, δ')

where δ meets the following requirement. Let δ = {x^ jyi,..., xn/yn} represent the substitution of xj by yi, etc. The requirement is that for all I, 1 < I < n, at least one of χι or yi is a free variable. Of course, we need to take the variable renaming performed by IVUnify into ac1 i count, and so we actually test whether each ~~ (χι,δ') or < ^ " ( y / , 5 ' ) is free. — A sometimes clashes with Β iff the two preceding tests are false. Finally, like disabling tests, we can compile a test t h a t is true iff an instantiation of the first rule clashes with an instantiation of the second. We note t h a t the the clash relation is symmetric. 2

O u r production system model is more general than that of OPS5, and so we must consider clashing explicitly. We discuss this further in Section 5.3.

177

5.

Solution t o t h e Serialization P r o b l e m

In this section, we give a formal description of our solution at both the instantiation level and rule levels. In Section 6, we present three algorithms. T h e first two algorithms apply the instantiation level solution; one of the two algorithms runs synchronously and the other asynchronously. T h e third algorithm is from Dan Neiman and takes a different approach to serialization. 5.1.

Solution

that Examines

Instantiations

If our M R P S executes a set of matching instantiations simultaneously, we must guarantee t h a t some serial execution of those instantiations could have produced the same result. P a r t of t h a t problem is in finding a serial execution in which no instantiation is disabled before it is executed. Let J be a set of instantiations t h a t m a t c h in some W M . We define a directed graph, called I D O ( Z ) , t h a t represents t h e instantiation disabling order. For each instantiation t in J , there is a node in IDO(I). For each distinct i1 and i2 in X where i2 disables i1, there is an edge from the node for i1 to the node for %2 (note the reversal of i1 and i2 since the order in 3 IDO is the reverse of the disables r e l a t i o n ) . T h i s comprises all of IDO(X). T h u s , each edge from i1 t o i2 in IDO(X) indicates t h a t i2 disables i1 and, therefore, iJ must be executed before i 2 . We note t h a t there are no self loops in IDO(X), where a self loop is an edge from a node to itself. T h e o r e m 5: The co-execution of a set of instantiations, X, using parallel models is serializable ifIDO(I) is acyclic and no two distinct 4 stantiations in X clash.

our in-

P r o o f : We note t h a t , in our parallel models, the selected instantiations can be executed in any order. If the instantiations in IDO(l) are executed serially according to any topological order of the nodes in I D O ( J ) , no instantiation will be disabled before it is executed. Since IDOÇL) is acyclic, then a topological order exists, and there thus exists a serial order for executing the instantiations in X. Since we have assumed t h a t there are no clashes between distinct instantiations, these instantiations can be 3

FVom here on, we wiU simply write t to refer to a node that represents i. We note that the converse is not true. Assume that IDO(T) is cyclic, so no topological order exists. Assume that we have chosen some serial order in which to execute the members of J . For any cycle within IDO(J), some member of the cycle will be executed first. The execution of that member will disable some other member of the cycle, and so it appears that serialization wiU not work. However, it is possible for some third element of the cycle to re-enable the disabled instantiation before it is executed. 4

178

executed in any order, and every order yields the same net result, given our requirements on the A C T / M A T C H step in Section 1.2. Since the net result is order-independent, the result of co-executing the instantiations in J in parallel is the same as executing t h e m according to the above serial order. • For an example, we return to rule PA from Section 1.1, which makes a reservation for an airline passenger. Imagine we had two instantiations of rule PA t h a t competed for the same seat. (PA ( O P E N 2E 1232) (WANTS Schmölze 1232)) (PA ( O P E N 2E 1232) (WANTS BigWig 1232)) These instantiations do not clash but each disables the other because each deletes the O P E N W M E which led to the match of b o t h . T h u s , their IDO has a cycle and co-executing them violates Theorem 5. T h i s makes sense since co-executing t h e m would lead to (RESERVATION Schmölze 1232 2E) (RESERVATION BigWig 1232 2E) which is a result t h a t could never occur from any serial execution sequence. O u r serialization algorithms are based on Theorem 5. T h e essential idea is to examine the instantiations t h a t currently m a t c h and to select a subset of t h e m whose IDO is acyclic and where no clashing occurs. We will never actually construct an IDO because it is too expensive. Instead, our algorithms select sets of instantiations where there is no clashing and whose I D O , if it were to be constructed, would be acyclic. We note t h a t the selection of such sets of instantiations can be performed incrementally, which is demonstrated by algorithm A2 in Section 6.2. 5.2.

Solution

that Examines

Rules

I&S and others offer serialization algorithms t h a t work at the level of rules, 5 not i n s t a n t i a t i o n s . Therefore, we now define a formal solution at the level of rules; i.e., we identify the rules in a given P M t h a t must be synchronized. Algorithms t h a t work at the rule level perform much of the work for serialization off-line, leaving less work to be done in each cycle. However, this approach offers a less precise solution and obtains less concurrency. 5

This includes the current author; see [34, 35].

179

Let H be a set of rules. We define a directed graph, called RDO(H), t h a t represents the rule disabling order. For each rule in 7£, there is a node in RDO(fl). For each A and Β in ΊΖ where Β possibly disables A, there is an edge from A to Β in RDO(7Z) (note the reversal of A and B). This comprises all of RDO(H). Let Rules(T) be the set of rules t h a t have an instantiation in X. This leads to the following theorem. T h e o r e m 6: The co-execution of a set of instantiations, X, using our parallel models is serializable if: - RDO(Rules(I)) is acyclic except for self loops, — rules that have self loops in RDO(Rules(I)) have only one instantiation in X, - no two distinct rules in RulesfX) can possibly clash, and — rules that possibly clash with themselves have only one instantiation in X. P r o o f : We will show t h a t when X satisfies the above conditions, it also satisfies the conditions of T h e o r e m 5 and thus the parallel execution of X is serializable. Assume X satisfies the above conditions. We first deal with clashing. Since no two rules in Rules(X) can possibly clash, then no two instantiations from X from different rules can clash. Further, since we require t h a t there be only one instantiation for rules t h a t possibly clash with themselves, then no two instantiations of the same rule can clash, and overall, no two instantiations in X can clash. Next, we deal with disabling. We define G(X) as follows. For each instantiation in Î , there is a node in G(X). For each edge in RDO(Rules(X)) from A to B, where A and Β are distinct, we add an edge to G(I) from every instantiation of A to every instantiation of B. Note t h a t G(X) has no self loops. Basically, G(X) represents all possible disablings among distinct elements of X. We will show t h a t G(X) is acyclic and t h a t IDO(X) is a subgraph of it, thereby making IDO(X) acyclic. First, we show by contradiction t h a t G(X) is acyclic. Let i t , . . . , i n , it be a cycle in G(X). Here, η > 1 since G ( J ) has no self loops. By examining the construction of G ( J ) , we note t h a t each adjacent pair of nodes in our cycle comes from distinct rules. T h u s , if G(X) has a cycle, then there must be a cycle in RDO(Rules(X)) t h a t does not include any self loops (i.e., Γ ί , . . . , Γ η , Γ ι where r j is the rule for ι 2 , etc.). But RDO(Rules(X)) is acyclic except for self loops and so we have a contradiction. Therefore, G(X) is acyclic. Now we show t h a t IDO(X) is a subgraph of G(X). First, they have the

180

same set of nodes. Second, each edge from if to ijf in IDO(l) corresponds to ij* disabling if. Given t h a t , rule Β must possibly disable rule A, and so there must be an edge from A to Β in RDO(Rules(I)). T h u s , there must be an edge from if to if? in G ( J ) . In summary, IDO(I) is acyclic, and no two instantiations in J clash, so by T h e o r e m 5, the co-execution of 1 in our parallel models is serializable. • For an example, we again return to rule PA from Section 1.1. Clearly rule PA possibly disables itself - i.e., one instantiation of PA can disable another instantiation of PA - as was shown in the example in Section 5.1. Therefore, given an X t h a t contains at least one instantiation of PA, then RDO(Rules(I)) will have a self loop from the node for PA to itself. According to Theorem 6, rules with such self loops can have only co-execute one instantiation. If there is more t h a n one instantiation of PA in Z , then T h e o r e m 6 does not guarantee their safe co-execution. T h i s example shows clearly why rule level synchronization can oversynchronize. For rule PA, rule level synchronization prohibits the coexecution of any two instantiations of PA because they might disable each other. However, instantiation level synchronization prohibits only when there is an actual cycle of disablings among the instantiations. T h e o r e m 6 shows the correctness of the algorithm of I&S [12] and others who work at the rule level. In this paper, we will not offer any algorithms t h a t operate at the rule level, but will discuss t h e m further in Section 8.

5.3.

Differences

with

OPS5

We noted earlier t h a t we assume W M is a set, whereas O P S 5 allows the W M to be a multi-set. This does not affect the correctness of our work as it applies to O P S 5 , but it can occasionally lead to unnecessary synchronizations. For cases of disabling where the C E is negative, these differences have no impact. However, when a disabling is detected involving a positive C E whose corresponding W M E is duplicated, the disabling action might delete the duplicate W M E and, thus, might not actually disable. In such cases we would synchronize unnecessarily, though still produce a serializable result. For clashing, there is another relevant difference. Since O P S 5 requires t h a t W M E s being deleted be matched on the LHS, whereas we do not, and since O P S 5 treats W M as a multiset, it appears t h a t one can avoid clashing problems via mechanisms other t h a n synchronizing. As such, some M R P S s do not check for clashing (e.g., [24, 25, 23]), though none of these papers address clashing explicitly.

181

Scheduler

Queue for Demons

RD3 Demons

Instantiations to be considered &/or executed

New instantiations

Conflict Set

Fig. 6. Basic Architecture for Ν = 4 · 1 Scheduler and 3 Demons

A l Scheduler Algorithm: (1) If there are no instantiations in the CS then exit. (2) Let M be the m i n i m u m of F χ Ν and the size of the CS. (3) Remove M instantiations from CS and place t h e m in array A from 1 to M. (4) Mark each instantiation in A as in. (5) Schedule each instantiation in A. (6) Wait for quiescence (demon queue empty and all demons idle). (7) Loop to 1. A l D e m o n Algorithm: (1) If demon queue is e m p t y then demon is idle & loop to 1. (2) Demon is busy. Remove an instantiation from queue h call its index ι. (3) For j : = i + 1 to M while A[i] is still marked in do If (A[j] is still marked in) and (A[i] clashes with or disables A[j]) then mark A[i] as out. (4) If A[i] is still marked in then execute it. (5) Loop to 1. Fig. 7. A l : Synchronous algorithm using off-line disabling/clashing tests

182

6.

A l g o r i t h m s t o G u a r a n t e e Serializaibilty

We will present three algorithms t h a t operate at the instantiation level. These algorithms are arguably the three most precise such algorithms in the literature. They are precise because they choose a large set of instantiations to co-execute as compared to other algorithms. T h e y also operate very quickly, as we will show in Section 7. Our goal here is to d e m o n s t r a t e t h a t t h e foregoing theoretical analysis leads to concrete algorithms. In Section 7, we demonstrate the effectiveness of these algorithms, along with their limitations. All three algorithms are targeted for shared-memory machines. One algor i t h m operates synchronously while the other two operate asynchronously. Of the latter two, one synchronizes instantiations using the tests compiled after an off-line analysis, as described in Sections 4 and 5, while the other uses a hybrid scheme t h a t employs locks whenever possible, augmented 6 with tests produced by the match process. All three algorithms use the same basic architecture, which comes from Daniel Neiman's doctoral research [29]. We will explain t h a t architecture now. Given Ν processors, we assign 1 processor to be the scheduler and the remaining Ν — 1 processors to be demons. T h e specific j o b s of the scheduler and demons differ for the different algorithms, b u t their basic jobs are the same. T h e scheduler pulls new instantiations off of the conflict set (CS) and decides whether or not to schedule them. Scheduling an instantiation consists simply of p u t t i n g it on a shared queue. Each demon pulls instantiations off the shared queue - each instantiation goes to exactly one demon - and executes t h e m if the demon decides t h a t execution is appropriate. T h i s basic architecture is shown in Figure 6. T h e differences between the three algorithms are in the processing t h a t each process does to an instantiation, and how each decides whether to schedule a n d / o r to execute the instantiation. 6.1.

Al:

Synchronous

Algorithm

Using Off-Line

Analysis

Figure 7 shows the scheduler's and demons' algorithms for A l . As can be inferred, the scheduler and demons take turns doing work, and each such pair of turns is called a parallel cycle. T h e scheduler only begins when the demons are idle, whereupon it takes M instantiations from the conflict set as they arrive and schedules them, which means simply t h a t it places 6

T h e first algorithm, A l , was first presented as Mi-Greedy in [35]. AU three algorithms were presented and analyzed in [37].

183

t h e m on the execution queue. It limits A/, the number of instantiations considered per parallel cycle, t o be F χ Λί, where F is a constant factor and Ν is the number of processors. We apply this limit because the time t h a t each demon spends testing for disabling and clashing is proportional to the size of M. We experimented with an F of 2, 4, 8 and 1000 (1000 has the same effect as F = oo) and found t h a t F = 2 consistently produced the fastest execution times. All of our results in Section 7 use F = 2. T h e demons take instantiations off of the queue one by one, test t h e m for disabling and clashing, and if appropriate, execute them. Multiple instantiations can be tested as such in parallel because each demon will only m a r k the instantiation it is considering and no other. Moreover, the body of code t h a t executes an instantiation - i.e., performs the resulting A C T and M A T C H work - has been written to allow multiple simultaneous executions (see Section 1.2). Upon examination of the demon algorithm, one can infer t h a t , after the demons finish, the set of instantiations in A t h a t are still marked in meet the requirements of T h e o r e m 3. As a result of line 3, there is no i < j where b o t h A[i] and A[j] are in and where either A[i] clashes with A[j] or A[i] disables A[j]. T h u s , no two in instantiations clash (remember, clashing is symmetric) and there is no cycle of in disabling relations (since we have prevented any forward links of A[i] disabling A[j]). 6.2.

A2: Asynchronous

Algorithm

Using Off-Line

Analysis

A l is synchronous, and as such, wastes a considerable amount of time. Given t h a t different instantiations take different amounts of time to execute, some demons sit idle while waiting for other demons to finish. An asynchronous algorithm eliminates this wasted waiting time (as was argued in [27]), and so we designed A2. Figure 8 shows A2, an asynchronous version of A l . Here, the scheduler simply takes instantiations from the CS and schedules them, thereby performing very little work. T h e scheduler also checks for an empty CS and for quiescence, which signals no further executions. This is similar to A l , however, the scheduler does no waiting. T h e demons also behave similarly to the demons in A l . However, we must carefully define the execution time of an instantiation. A list Ε of the instantiations currently executing is maintained where Ε is a critical resource. Writers t o Ε have unique access but multiple readers can have simultaneous access. An instantiation is said to be executing from the time t h a t a demon removes it from the queue and places it on Ε until the time t h a t the demon either discards it or finishes executing its RHS. In addition,

184

A2 Scheduler Algorithm: (1) If there are instantiations in the CS then go to 2 else if t h e demons are quiescent (i.e., queue empty and all demons idle) then exit else loop to 1. (2) Remove a non-dead instantiation from CS and schedule it. (3) Loop to 1. A2 Demon Algorithm: (1) (2) (3) (4)

If demon queue is empty then demon is idle & loop to 1. Demon is busy. Remove an instantiation from queue & call it I. Mark I as in. Add I to E, a list of executing instantiations. Access to Ε is critical code. Writers have unique access but there can be many readers. (5) For J : = each element in Ε (J φ I) while (I is in) and (I is not dead) do If (J is in) and (J is not dead) and (I clashes with or disables J ) then mark I as out. (6) If (I is in) and (I is not dead) then execute I & remove it from Ε else remove I from Ε and return it to CS if it is not dead. (7) Loop to 1 Fig. 8. A2: Asynchronous algorithm using disabling/clashing tests

and due to the asynchrony of the system, it is possible for an instantiation to become disabled while a demon is testing it in line 5. In t h a t case, the system m a r k s the instantiation as dead and no further processing is performed on it. We note t h a t it is possible for two instantiations t h a t , say, disable each other, to be examined simultaneously by two demons executing line 5, and for each to cause the other to be marked out. In this case, neither is executed and each is returned to the conflict set. T h e scheduler would then schedule both of t h e m again, and it is theoretically possible for the system to repeat this process infinitely. However, we argue t h a t this is extremely unlikely due to the asynchrony of the system. For reasons similar to those for A l , A2 obeys Theorem 3 and yields a serializable result.

185

A3 Scheduler Algorithm: (1) If there are instantiations in the CS then go to 2 else if the demons are quiescent (i.e., queue e m p t y and all demons idle) then exit else loop to 1. (2) Remove a non-dead instantiation from CS. Call it I. (3) Try to acquire locks for I. (4) If successful then schedule I else if I is not dead then return I to CS. (5) Loop to 1. A3 Demon Algorithm: (1) (2) (3) (4)

If demon queue is empty then demon is idle & loop to 1. Demon is busy. Remove an instantiation from queue h call it I. Execute I. Loop to 1. Fig. 9. A3: Asynchronous algorithm using locks

6.3.

A3: Asynchronous

Algorithm

Using Locks

Figure 9 shows A3, which was developed by Daniel Neiman [29, 37], and which is considerably different from A l and A2. A3 is a hybrid algor i t h m . It uses a locking scheme when to prevent non-serializable interferences t h a t are due to positive references to CEs. W h e n interactions are due t o negated CEs, A3 uses tests similar to those used by A2, b u t derived during the match process rather t h a n at compile time. A3 acquires locks in the scheduler so t h a t lock acquisition is a serial process, which eliminates t h e potential for deadlock. By the time an instantiation is scheduled, the system has already determined t h a t it should execute, so the demons perform no decision-making: demons simply take instantiations off the queue and execute t h e m . A3 uses a locking scheme whenever possible because of the low overhead associated with acquiring and testing locks. T h e locking scheme would be sufficient b u t for the use of negative CEs. While it is easy to lock a W M E t h a t is already in the W M , it is not so easy to lock the "non-existence" of a W M E , as implied by a negative C E . However, Neiman has designed an efficient way to use the Rete net [6] to identify precisely the set of W M E s t h a t would match the negative C E . Raschid et al [33] also use locks for serializing, b u t for negative C E s they lock an entire class of W M E s .

186

T h e following discussion assumes some familiarity with R,ete nets [6] and locking schemes. Readers who are not familiar with these topics might wish to skip to the end of this section. In the Rete net, when a W M E is positively matched, a token representing t h a t element is concatenated to a set of tokens being propagated through the network. We can similarly create a pseudo-token corresponding t o a successful m a t c h of a negated element. This token represents a p a t t e r n of the working memory elements t h a t would disable this instantiation. T h i s p a t t e r n is simply the set of tests encountered by the working memory element as it proceeds through the matching process; specifically, t h e inter-element a l p h a tests preceding the N O T node, concatenated t o the tests performed by the N O T node and unified with the positively matched tokens in the instantiation. For example, if we had a rule such as the one shown below, the pseudotoken would have the form ((CLASS = B) ( E L E M E N T ( l ) = W O M B A T )

(ELEMENT(2)=KOALA))

T h u s , any currently executing instantiation t h a t creates an element matching this p a t t e r n would disable an instantiation of PK stimulated by the working memory element (A W O M B A T ) . (P PK

(A < x > ) - ( B < x > KOALA) — (B < x > KOALA))

W M = {(A W O M B A T ) }

W h e n a rule instantiation is created, we thus have two sets of tokens: the W M E s matching the left-hand side (LHS) and negative p a t t e r n tokens. To use the latter, we must also do the following. Before each instantiation is scheduled, we develop a list of all the W M E s t h a t it will add when it is executed. T h i s is reasonable as the formation of elements is usually inexpensive. Immediately before the instantiation is executed, we post all the W M E s it is a b o u t to add onto a global ADD list. We now explain the operation of lock acquisition in detail. (1) Each W M E has a read counter and write flag. Each instantiation has a read and write list. As each instantiation, i, enters the C S , we add to its write list each W M E matched on its LHS t h a t would be modified or removed by i. T h e remaining W M E s matched on F s LHS are placed on its read list. Next, we see if any of the W M E s on the read or write list have their write flag set. If so, we discard i because it will soon be disabled by another instantiation t h a t is

187

already executing. If a W M E on the write list has its read counter > 0, we do not execute i and instead, place it back on the C S . In this way, we do not disable another instantiation t h a t is already executing while giving t another chance later. If i has not been discarded or p u t back on the CS, we proceed. (2) We compare Fs negated p a t t e r n tokens against the list of W M E s on the A D D list. If any match, then t is discarded as it will soon be disabled by an instantiation already executing. Otherwise, we proceed. (3) We now acquire the locks, which amounts to incrementing the read counters for the W M E s on the read list and setting the write flags for the W M E s on the write list. We also post the W M E s to be added to the A D D list. T h e demon also has some extra tasks. After it finishes executing an instantiation i, it removes the elements t h a t i added to the A D D list and decrements the read counters for those W M E s on Ts read list. We note t h a t accessing and modifying the A D D list, read counters and write flags must be performed in critical code. We note t h a t for the reasons explained in Section 5.3, A3 does not need to check for clashing.

6.4-

Discussion

T h e algorithms presented in this section differ from t h a t of Ishida and Stolfo [12] in several i m p o r t a n t ways. (1) Algorithms A l and A2 use unification to compare CEs and actions in developing our off-line tests, which produces more precise tests than comparing W M E class names (though I&S point out t h a t finer grain categories could be used). Note t h a t A 3 , which does not use the off-line tests, does not use unification, b u t the method it uses for comparison produces equivalent precision. (2) We synchronize only for cycles of disabling relations, not for every disabling relation. (3) We synchronize at the instantiation level, not the rule level. In other words, we synchronize instantiations, not rules whose instantiations might disable or clash. (4) Algorithms A2 and A3 operate asynchronously, not synchronously.

188

# of NonI/O Rules Mapper Toni-Waltz Tourney

77 31 17

Avg. φ Synchs per Rule Improvement Over I&S

I&S Synchs Self Distinct 77 31 13

940 440 87

11.5 1.0 1.0 Overall: 1.0

M3-Greedy Synchs Self Distinct 77 31 13

531 215 21 5.9

1.0

2.1 Overall: 1.9

Table 2. Comparison of Number of Rule Synchronizations

7.

Performance Analysis

In order to identify the effectiveness of our algorithms, we begin by measuring the impact of the differences mentioned above between our work and t h a t of I&S. This is followed by a detailed analysis of the three algorithms we presented. In order to make performance measurements, we have used four benchm a r k PS programs. These systems are (1) the M a n h a t t a n Mapper [19], which provides travel schedules for trips through M a n h a t t a n , (2) ToruWaltz, which applies Dave Waltz's constraint propagation algorithm to identify objects from line drawings [43], (3) Tourney, which determines a schedule for t o u r n a m e n t bridge games, and (4) Toru-Waltz-N, a version of Toru-Waltz t h a t was modified by Daniel Neiman so as to obtain greater 7 concurrency. For each of these systems, we only examined rules t h a t did not perform I / O since t h a t is our focus here. 8

T h e implementation is written in Top Level C o m m o n Lisp ( T o p C L ) , which runs on a variety of shared-memory machines. We used a 16 processor Sequent with 128 megabytes of core. T h e basic system, which includes a parallel matcher, was written by Daniel Neiman as part of his doctoral research [27]. Algorithm A3 comes directly from Neiman. Algorithms A l and A2 were implemented by modifying his system.

7

It was modified by Daniel Neiman to increase the available rule concurrency by combining the initialization and processing stages and to aUow rules to be asynchronously triggered. The text of the Toru-Waltz-N benchmark plus a discussion of its implementation and performance can be found in [28]. 8 T o p Level Common Lisp and TopCL are trademarks of Top Level Inc.

189

# of CS's

Avg CS Size

I&S

Ml

M3

340 152 6 166

20.3 58.4 172 83.6

1.2 1.0 1.0 1.1

5.7 11.8 7.2 7.6

2.3 2.2 1.8 2.3

1.0

6.7

2.0

Mapper Toru-Waltz Tourney Overall Average Avg. Improvement over I&S

Table 3. Concurrency Estimates using Number of Co-Executions Available Method I&S Al M3-Greedy

# Cycles 368 20 368

Max. Insts/Cycle 2 124 2

Avg. Insts/Cycle 1.0 18.5 1.0

All methods run against the same example. All fired 369 instantiations total and all gave the same final answer. Table 4. Available Concurrency: Number of Instantiations to Co-Execute for Toru-WaltzN with F = oo

7.1.

Comparing

I&S with our

Algorithms

We begin by showing the increased precision of our approach in contrast to t h a t of I& S, as summarized in Section 6.4. T o accomplish this comparison, 9 we define a modified version of the I&S algorithm t h a t we call M3-Greedy. It is similar to I&S's algorithm except t h a t it does the following in accordance with Sections 4 and 5. — M3-Greedy uses unification t o compare CEs and actions, not simply the W M E class names, and — M3-Greedy synchronizes only for cycles of disabling relations, not for every disabling relation However, M3-Greedy operates a t t h e rule level, as does the algorithm of I&S. Our algorithm A l differs from M3-Greedy by synchronizing at the instantiation level instead of t h e rule level. Table 2 shows the results of a static analysis of I&S and M3-Greedy against the first three benchmark PSs. T h e number of n o n - 1 / 0 rules for each P S is shown. In addition, for each of I&S and M3-Greedy, two numbers are shown. " S e l f is the number of rules t h a t synchronize with themselves; 9

Algorithm M3-Greedy first appeared in [35].

190

nearly all rules do so for both systems. "Distinct" shows the number of pairs of distinct rules t h a t need to be synchronized. For example, the Mapper has 77 n o n - I / O rules, which means t h a t there are 5852 pairs of distinct rules ( 7 7 x 7 6 ) . Of these, 940 must be synchronized according to I&S while 531 need to be synchronized according to M3-Greedy. T h u s , roughly half of the rule-pair synchronizations required by I&S are, in fact, unnecessary. But how does the above static analysis translate into available concurrency during run-time? To estimate concurrency differences, we collected conflict sets from each of the first three benchmark PSs. To do this, we ran each system and collected the instantiations corresponding to n o n - I / O rules t h a t matched in each cycle. In other words, for each conflict set, we discarded the instantiations of any rules performing I / O . If this left the conflict set empty, then we did not count t h a t particular conflict set. We then determined which instantiations in each conflict set could co-execute using the various m e t h o d s . Note t h a t these measurements do not measure run-time. Instead, they measure the number of instantiations t h a t could safely co-execute in the parallel cycles t h a t arise; we examine run-time soon. T h e results are presented in Table 3. This table comes from [35], which analyzed slightly different algorithms. However, M l is very similar to A l and M3 is very similar to M3-Greedy. Also note t h a t 56 conflict sets from Mapper and 4 from Toru-Waltz had cardinality 1, which allowed no possible concurrency. As shown in Table 3, M3 (and thus M3-Greedy) identifies about twice as much concurrency t h a n does I&S. M l (and thus A l ) identifies about 6.7 times as much concurrency. One qualification is needed here, namely, the first three benchmark PSs we have studied were originally written for serial machines. Some efforts were m a d e by the various programs' authors to make the M a n h a t t a n Mapper and Toru-Waltz have more concurrency, b u t they were only moderately successful. Therefore, we have studied Toru-Waltz-Ν in much more detail since it offers a high degree of potential concurrency. Table 4 shows the available concurrency when running the fourth benchmark P S , namely, Toru-Waltz-N. It is clear t h a t b o t h I&S and M3-Greedy performed terribly in t h a t neither obtained any concurrency except for one parallel cycle in which each co-executed 2 instantiations. A l performed well by identifying, at most, 124 instantiations to co-execute and, on average, 18.5 instantiations to co-execute. In terms of execution time, both I&S and M3-Greedy did not offer much competition when executing Toru-Waltz-N. T h e best serial execution time is 12.79 seconds. T h e best execution of A l took 4.57 seconds, which yields

191

a speedup of 2.80 (we discuss run-time tests further in Section 7.3). Both I&S and M3-greedy ran slower t h a n the serial version since they did not identify any noticable concurrency and they incurred overhead over and above t h a t of t h e serial algorithm. 7.2.

Discussion

Regarding

Comparison

with I&S

T h e first clear result is t h a t synchronizing at the rule level is inadequate. In all the programs we studied, large conflict sets were usually due t o m a n y instantiations of a small number of rules. Synchronization at the rule level prevented almost all co-executions. Of the algorithms studied above, only A l was able t o identify and take advantage of a reasonable a m o u n t of concurrency. It also seems clear t h a t using unification and synchronizing only for cycles of disabling relations offers a notable improvement. 7.3.

Comparing

Algorithms

Al,

A2 and A3

We now take a close look a t the three algorithms presented in this paper, namely, A l , A2 and A 3 . Our.goal is two fold. First, we want to understand the advantages and disadvantages of each. Second, we want to see whether or not there is an inherent cost to serialization, and if so, to determine t h a t cost. We will examine only the Toru-Waltz-N benchmark. T h e time for serial execution is 12.79 seconds. In our parallel system, if we t u r n off all checks for seriahzability, except for the locking performed by A 3 , which is very fast, Toru-Waltz-N still produces a correct answer, and the best time we obtain is 1.2 seconds with 15 processors. T h u s , 10.7 is the m a x i m u m possible speedup for this benchmark and our software without full serialization. Any further reductions in speedup are due to the serialization component of our algorithms. Figure 10 shows the speedups attained by the three algorithms. Clearly A3 performed t h e fastest, attaining a m a x i m u m speedup of 5.70 (runtime of 2.24 seconds), followed by A2 with a m a x i m u m speedup of 4.64 (runtime of 2.75 seconds), and followed, finally, by A l , with a m a x i m u m speedup of 2.80 (runtime of 4.57 seconds). However, when A3 was run, additional mechanisms were used, namely, match and action parallelism, which we explain below. We estimate t h a t they reduce the run time by a b o u t 0.5 seconds. S u b t r a c t i n g 0.5 from A2's best time of 2.75 seconds yields 2.25 seconds, or a speedup of 5.68. T h e run times of A2 and A3 are thus very similar. Figure 11 compares the speedup of the estimated A2 (as if it had match and action parallelism) against the actual speedup of A3.

192 6 S

Ρ

5 /

ι ο

1

/

Τ"

•

4

.

——* . — κ —

/

Ρ

/

/

2

3

4

5

6

7

8

— χ —

9

10

-——Χ

11

12

—χ—

13

14

15

Number of Rule Demons X -l> φ

AI: Synchronous algorithm using disables/clashes tests A2: Asynchronous algorithm using disables/clashes tests A3: Asynchronous algorithm using locks

Fig. 10. Speedup for all three algorithms

/

/

/

/ # /

//* // 1

2

3

4

5

6

7

8

9

10

11

12

13

Number of Rule Demons "4" A2: Asynchronous algorithm using disables/clashes tests, φ A3: Asynchronous algorithm using locks. Fig. 11. Speedup for Estimated A2 and Actual A3

14

15

193

We have match parallelism when multiple processors jointly process the matching t h a t results from a single action. We have action parallelism when multiple processors jointly execute all the actions on t h e R H S of a single instantiation. Algorithms A l and A2 have each processor execute its own instantiation, b u t the processors do not collaborate on the m a t c h processing for a single action nor on the action processing for a single instantiation. A3 is actually p a r t of a larger parallel P S developed by Neiman [27], which offers match and action parallelism and which is controlled at the programmer's discretion. It is clear t h a t the speedup for each algorithm tapers off a t around 10 to 12 processors, indicating t h a t these m a x i m u m speedups are close to t h e absolute m a x i m u m s for these algorithms when running Toru-Waltz-N. T h e speedups realized fall quite short of 10.7, which was achieved without full serialization. Serialization thus appears to cut the potential speedup roughly in half for this benchmark. T h e reasons for A l being slowest is simple. A l is performing nearly the same work as A2, b u t has the disadvantage of being synchronous. Given t h a t different instantiations take different a m o u n t s of time to execute, this a m o u n t s t o a waste of processor time as demons sit idle while waiting for other demons t o finish. In addition, when the scheduler is active, all the demons are idle, and vice versa. An asynchronous algorithm eliminates this unnecessary waiting time (as was argued in [27]). Conceptually, A2 and A3 perform the same type of interference checking, although they use very different mechanisms to do so. It is interesting to see t h a t they offer similar potential for speedup. If we take a brief look a t these mechanisms, we see t h a t A2 requires 0(M) t i m e to check interference for each instantiation, where M is the number of instantiations currently co-executing: each instantiation must be checked against all those currently co-executing. It t u r n s out t h a t A 3 , when checking negative CEs, also requires 0(M) time per instantiation since the size of A D D depends on M. However, when A3 checks positive CEs, the time required is not dependent on the size of M and, instead, is constant per rule, which makes 10 this type of check very f a s t . To discover some of the limitations of these algorithms, we broke down the execution times of A2 further. Figure 12 shows a bar chart of the total run times for A2 along with the average time spent by the demons doing their two main tasks: testing for interference and executing instantiations. It is clear t h a t the time spent testing increases with the number of demons. 1 0

I n [27], Neiman argues that this is a good reason for preventing interference only for positive CEs and not for negative CEs, even though this falls short of guaranteeing serializability.

194

S e c ο n d s

3

5

7

9

11

13

Number of Rule Demons •H

Total runtime of overall system

fHH

Avg. time spent testing for interference per demon

[Élll Avg. time spent executing rules per demon Fig. 12. Average Times Used by the Rule Demons This makes sense because the average number of co-executing instantiations increases with the number of demons, so there is more testing to be done by each processor. In fact, this time should increase linearly in the worst case; in our case, it increases very little, probably due to overlap with time t h a t would otherwise be idle. T h e time spent executing rules decreases since there are more demons, and so each one executes fewer instantiations. Therefore, A2 will always be slower than a system t h a t does not guarantee serializability because of this interference checking time. Figure 13 shows the amount of useful work - i.e., work spent executing rules - as opposed to the time spent on serialization. As we can see, the a m o u n t of useful work performed by each processor decreases with increasing number of demons. However, the total amount of useful work performed remains fairly constant. This is good news: our algorithm does not increase the a m o u n t of useful work t h a t needs to be done. Unfortunately, as Figure 14 shows, the interference-checking time - i.e., the synchronization time - grows quickly. As stated above, the synchronization time spent by each processor remains nearly constant in our test runs as the number of demons increase. However, the total a m o u n t of time spent on synchronization increases slightly faster than linearly, and this is bad news.

195

2

3

4

5

6

7

8

9

10

11

12

13

14

15

Number of Rule Demons X Total overall time -J- Average time spent doing useful work per processor φ Total amount of time spent doing useful work (i.e., avg χ Ν) Fig. 13. Time Spent Doing Useful Work for A2

Let us take this p h e n o m e n a to the extreme using some simplifications. Imagine t h a t the time spent executing rules can be spread evenly among any number of processors (which is not the case). As we increase the number of demons, t h e time spent by each performing useful work will approach zero. However, the time spent synchronizing will only increase. T h u s , the synchronization time used by each processor with a small number of demons is an absolute m i n i m u m time t h a t each demon will spend no m a t t e r how m a n y processors are used. T h u s , synchronization time for a small number of demons is an absolute m i n i m u m overall execution time for any number of processors. W i t h 13 demons, the synchronization time is 0.6 seconds. By the above argument, speedup for A2 is limited to about 21 (remember, serial execution time is 12.79 seconds), and to a t t a i n this would require an ability to spread out the useful work t h a t we cannot do as yet. Figure 15 shows our ability to spread around the useful work for A2. Here, we see t h a t as the number of demons increases, so does the average number of co-executing instantiations. However, the number of demons rises faster due to the uneven distribution of available co-executable in-

196 6

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

Number of Rule Demons

φ

Average time spent synchronizing per processor Total amount of time spent synchronizing (i.e., avg χ Ν) Fig. 14. Time Spent Synchronizing for A2

stantiations. Moreover, there is a limit to this, namely, Toru-Waltz-N only offers a certain number of co-executing instantiations at any point in time. In A 3 , the interference checking is done via a locking mechanism. However, this mechanism must run serially in order to avoid deadlocks between instantiations simultaneously a t t e m p t i n g to acquire locks. All locking is thus performed in the scheduler. T h e most time consuming portion of the locking mechanism is the portion t h a t deals with negated tokens. If the cost of checking negated tokens against the A D D list is expensive as compared to the time needed to execute an instantiation, then the benefits of asynchronous execution are lost. In order to form an estimate of the overhead associated with negated tokens, we note t h a t the processing performed when matching each working memory element being asserted against each negated pseudo-token pattern is essentially equivalent to the time of a b e t a node activation within the Rete net (for the check against the A D D list) and two memory node activations (one for each addition or deletion to the A D D list). This approximation is reasonable as the tests contained within the negated pseudo-tokens are derived from the N O T nodes which generated them. T h e b e t a nodes are the most time-consuming component of the pattern matching process and the number of b e t a nodes executed

197

16

Number of Rule Demons Number of demon processors Avg # of instantiations co-executing Fig. 15. Average Number of Co-Executing Instantiations for A2 can be used to create an estimate of relative costs. Using the statistics gathered by G u p t a [10], we note t h a t the average instantiation activates approximately 40 b e t a node and memory operations (of course, the actual figures depend on the size and complexity of the CEs). T h u s the runtime detection of interactions due t o negated tokens may incur costs of as much as 10% of the cost of actually executing the rule for each negated condition in the rule. Because t h e detection of interference must be carried out within a critical region of t h e scheduler, an overhead of this m a g n i t u d e would limit the potential parallelism within the system to a factor of 10 (assuming one negative C E per rule on average, which agrees roughly with 11 the measurements in [10]), exclusive of other scheduling c o s t s . 7.4.

Discussion

regarding Comparison

of Al, A2 and A3

T h e first clear result is t h a t asynchronous execution is much preferred over synchronous execution, as demonstrated by the poor performance of A l . T h e second clear result is t h a t serialization appears to have some inherent 11

We can model a system of this type as an M / M / 8 queue [16].

198

costs. For our software and benchmarks, speedup appears to be limited to approximately one order of magnitude, no m a t t e r how many processors are used. Regarding the relative merits of using off-line compiled tests for synchronization (A2) or a locking scheme combined with tests derived from the R E T E net (A3), there does not appear to be a clear advantage of one over the other. We expected A3 to be slightly slower due to the necessity for deriving the tests for negative interferences at run time, but this seems to be compensated by the low overhead of the lock mechanism used to prevent positive interferences. T h e worst case time grows differently for each method, b u t in our studies, they performed comparably. Better performance is obtained by spreading out the useful work more evenly over more processors. To this end, incorporating match and action parallelism, in addition to the rule parallelism t h a t we have investigated, can only be helpful. 8.

Related Work

T h e first body of research in parallel PSs studied parallel match algorithms for O P S 5 [5]. Since most cpu time in 0 P S 5 is spent in the MATCH step (over 90% according to [4] and over 50% according to [23]), many efforts have tried to make parallel t h a t one step, while leaving the system to continue executing only one rule at a time. G u p t a [9] explores parallel versions of the R E T E algorithm [6], which is the match algorithm used in O P S 5 . Stolfo and Miranker [39, 41], G u p t a [8] and Miranker [21, 22] offer parallel match algorithms for D A D O [40], a massively parallel machine. Oflazer [31] examines the problem of partitioning rules over parallel processors for optimal matching and execution. For most of these works, m a x i m u m speedups realized by these approaches are about ten times the fastest sequential version of O P S 5 , no m a t t e r how many processors are used [9]. However, there are exceptions, such as [15], where theoretical speedups of 23-fold for the MATCH algorithm are reported. A number of other works have studied M R P S s , not all of which guarantee serializability. For example, the system in [2] and BLITZ [26] provide mechanisms to control which rules can co-execute, but otherwise leave this responsibility in the hands of the programmer. Most such works, however, do make the serialization guarantee. We will review them briefly, focusing only on their respective approaches to serialization. T h e first and most i m p o r t a n t work is t h a t of Ishida and Stolfo [12], which we reviewed in Section 3. In earlier work, Moldovan [24, 25] designed RUBIC, which is similar to a term re-writing system (e.g., [11]) in t h a t w h a t

199

is matched on the LHS is replaced by the R H S . Negative tests on the LHS are t h u s not allowed. He uses synchronization at the rule level to guarantee seriahzability, allowing two rules to co-execute iff neither can disable the other, much like t h a t of I&S. Moldovan does not address problems regarding clashes. In later work, Moldovan, Kuo and C h a [18] expand the above model to allow for negative tests on the LHS, and they expand their synchronization criteria to prohibit clashing. However, they still perform synchronization at the rule level. They point out, correctly, t h a t M R P S s can easily produce a serializable b u t wrong answer and offer a sophisticated rule context mechanism to guide the systems towards right answers. In [32], Oshisanwo and Dasiewicz examine some of the trade-offs between run-time and off-line analysis. They also offer a M R P S t h a t leaves more control in the h a n d s of the programmer and t h a t synchronizes at the instantiation level. T h e y define the match-set of an instantiation as the W M E s t h a t it matched positively, and the act-set as the W M E s t h a t it will add, remove or modify. T w o instantiations, A and B, are synchronized if there is any overlap between act-set(A) and act-set(B), between match-set(A) and act-set(B), or between act-set(A) and match-set(B). T h i s criteria oversynchronizes in some cases, which they note. Unfortunately, it also misses some needed synchronizations where a positive action disables a negative C E . For example, they would not synchronize in the example from Figure 2. In an independent effort, performed at the same time as our earlier work on this topic [34], Ishida [13,14] developed an M R P S model t h a t guaranteed seriahzability. Like our work, Ishida compares instantiations, not j u s t rules, and he synchronizes only when cycles of disablings are detected. Our A l algorithm is similar to Ishida's. T h e C R E L system [23, 17] by Miranker et al also relies upon synchronization, primarily at the rule level. They go beyond I&S by synchronizing only when there is a cycle of disabling relations, however, they do not synchronize for clashing. When they compare two CEs a n d / o r actions for possible disablings, they check more than j u s t the type of the W M E affected, b u t it is not clear whether or not they use unification, which yields more precision. Overall, the work is still under development, with much attention being paid to techniques for transforming rules so as to make t h e m run more efficiently. Finally, we note t h a t the serialization problem for database management systems (DBMSs) is similar to t h a t for M R P S s . T h e database in a D B M S is analogous to a W M , an operation is analogous to an action, and a transaction (a partially ordered set of operations) is analogous to an instantiation. Seriahzability would guarantee t h a t , after executing a set of transactions,

200

the final s t a t e of the d a t a b a s e would be one t h a t could have been produced by the same transactions executing in some serial order [1]. A common technique used by many of the DBMS works reviewed in [1], which focuses on distributed database management systems, involves the use of read and write locks. These locks guarantee unique access to certain portions of the d a t a b a s e , which results in some operations being delayed and, sometimes, in certain transactions being aborted. Another approach can be found in [42], where a variety of solutions to seriahzability are offered for DBMSs t h a t are based on graphs similar to our IDO graph. In general, however, there is no DBMS counterpart to a production rule, which leads to the major difference between a DBMS and a P S . Namely, in a DBMS, the transactions to be processed come from outside sources (e.g., from users) while, in a P S , the instantiations are derived entirely from the rules and the s t a t e of the W M . By synchronizing in a PS, we delay execution of instantiations t h a t may, in turn, become disabled and need not execute. Synchronization is less appealing for many DBMS applications since they try to process all transactions. Also, most of these DBMS researchers are less concerned t h a n we are with the performance of memory-resident systems. An especially interesting and related work is [38, 33] who examine the problem of executing PSs over (relational) DBMSs using a locking technique t h a t differs from Neiman's. Given a conflict set, they try to execute all instantiations, where locking forces certain instantiations to be delayed or aborted. By contrast, in our approach t h a t compiles off-line tests, we spend additional time identifying instantiations t h a t can co-execute, b u t afterwards have no delays in their execution. A problem with [38, 33] is t h a t they can run into deadlock. However, this can be prevented by obtaining all locks for an instantiation in advance in a pre-determined order. Our methods are deadlock free. Another difference between our approach and theirs is t h a t we take advantage of off-line analysis, something which is difficult for t h e m to do, and which may not be appropriate in a d a t a b a s e environment. Also, our approach assumes t h a t communication between processors is very fast. For widely distributed processors, the locking approach may be best. Finally, there is the recent dissertation of Daniel Neiman [29]. Neiman argues t h a t full serialization is too costly and, instead, offers a fast locking scheme t h a t prevents most interferences. In particular, he offers the A3 algorithm but without checks for negative CEs. If full serialization is needed, he requires t h a t the PS programmer ensure it. To help in this process, Neiman offers several new control mechanisms t h a t allow a programmer to control rule executions. Overall, his system has been used to implement a

201

variety of P S examples. He produces correct answers at high speeds. An examination of his results shows t h a t his system should be able to provide u p t o two orders of m a g n i t u d e of speedup, as opposed to the one order of m a g n i t u d e t h a t our fully serialized systems seem t o offer to date. 9.

Conclusions

We presented a formal solution to the serialization problem for production systems t h a t execute m a n y instantiations simultaneously. We also presented a variety of algorithms t h a t apply this solution. T h e major improvements we offer over t h a t of Ishida and Stolfo's [12] and others are as follows. (1) We use unification to compare CEs and actions, which produces more precise tests t h a n comparing W M E class names (though I&S point out t h a t finer grain categories could be used). (2) We synchronize only for cycles of disabling relations, not for every disabling relation. (3) We synchronize instantiations, not rules whose instantiations might disable or clash. (4) We execute asynchronously. These improvements increase the concurrency dramatically for the programs we examined. Tests with our initial implementation confirm this in terms of actual speedup. For the Toru-Waltz-N benchmark running with our algorithms, there appears to be an inherent overhead t h a t limits speedup, no m a t t e r how m a n y processors are used. For A 3 , this overhead appears to be at least 10%. For A2, it appears to be at least 5%, and t h a t overhead increases with the number of processors, suggesting a firm limitation to speedup. For A l , we have shown t h a t its performance will always lag behind t h a t of A2. Overall, a speedup of 10 appears to be an absolute limit for A 3 , and probably applies to A2 and A l as well. In addition, we find t h a t our serializable algorithms obtain roughly half the speedup obtained by a similar parallel system t h a t does not guarantee serializability. Further study will be needed to determine whether this overall speedup limitation is due to the Toru-Waltz-N benchmark, our algorithms, the serialization guarantee in general, or a combination of the three. While we have concentrated on the detection of rule interactions in this paper, the overhead analysis is appropriate for any overhead such as control scheduling or heuristic pruning t h a t has to occur within a critical region. We thus feel t h a t this type of research is essential to our understanding of

202

t h e applicability of parallelism to artificial intelligence research in general. Future work of ours will continue to improve and test our algorithms. However, our primary interest will shift to the control problem. T h a t is, how does one design an M R P S system t h a t allows either coarse or fine-grain control of rule execution but t h a t does not incur unacceptable overheads. 10.

Acknowledgments

Many t h a n k s go t o Daniel Neiman for his underlying M R P S system, his A 3 algorithm and his comments on an earlier draft of this paper. Finally, t h a n k s goes to the National Science Foundation who supported this work in p a r t under grant number IRI-8800163. References [1] Philip A. Bernstein and Nathan Goodman. Concurrency Control in Distributed Database Systems. Computing Surveys, 13(2):185-221, June. 1981. [2] Albert Boulanger. The Modification of a Rule-based Diagnostic System for Routinized Parallelism on the Butterfly Parallel Computer. Technical Report 6713, BBN Laboratories Inc., Camb., MA, January 1988. [3] W. F. Clocksin and C. S. Mellish. Programming in Prolog. Springer-Verlag, 1981. [4] C. L. Forgy. On the Efficient Implementation of Production Systems. PhD thesis, Department of Computer Science, Carnegie Mellon University, Pittsburg, PA, 1979. [5] C. L. Forgy. OPS5 User's Manual. Technical Report CMU-CS-81-135, Department of Computer Science, Carnegie Mellon University, 1981. [6] C. L. Forgy. Rete: A Fast Algorithm for the Many Pattern/Many Object Pattern Match Problem. Artificial Intelligence, September 1982. [7] Charles Forgy, Anoop Gupta, Allen Newell, and Robert Wedig. Initial Assessment of Architectures for Production Systems. In Proceedings of the Fourth National Conference on Artificial Intelligence (AAAI-84), Austin, Texas, August 1984. [8] A. Gupta. Implementing OPS5 Production Systems on DADO. Technical Report CMU-CS-84-115, Department of Computer Science, Carnegie Mellon University, December 1983. [9] Anoop Gupta. Parallelism in Production Systems. PhD thesis, Department of Computer Science, Carnegie Mellon University, March 1986. (Also appears as Technical Report CMU-CS-86-122). [10] Anoop Gupta. Parallelism in Production Systems. Morgan Kaufmann Publishers, Inc., Los Altos, CA, 1987. [11] G. Huet. Confluent reductions: Abstract properties and applications to term rewriting systems. J. ACM, 27(4):797-821, Oct. 1980.

203

[12] T. Ishida and S. J. Stolfo. Towards the Parallel Execution of Rules in Production System Programs. In Proceedings of the International Conference on Parallel Processing, 1985. [13] Toru Ishida. Methods and Effectiveness of Parallel Rule Firing. In Proceedings of the 6th IEEE Conference on Artificial Intelligence Applications, March 1990. [14] Toru Ishida. Parallel Firing of Production System Programs. IEEE Transactions on Knowledge and Data Engineering, 3(1):11-17, 1991. [15] Michael A. Kelly and Rudolph E. Seviora. An Evaluation of DRete on CUPID for OPS5 Matching. In Proceedings of the Eleventh International Joint Conference on Artificial Intelligence (IJCAI-89), Detroit, MI, August 1989. [16] Kleinrock and Leonard. Queueing Systems, Volume I: Theory. John Wiley and Sons, 1975. [17] Chin-Ming Kuo, Daniel P. Miranker, and James C. Browne. On the Performance of the CREL System. Journal of Parallel and Distributed Computing, 13(4):424-441, Dec 1991. [18] Steve Kuo, Dan Moldovan, and Seungho Cha. Control in Production Systems with Multiple Rule Firings. Technical Report PKPL 90-10, Department of Electrical Engineering, University of Southern California, Los Angeles, CA, August 1990. [19] M. Lerner and J. Cheng. The Manhattan Mapper Expert Production System. Technical report, Department of Computer Science, Columbia University, May 1983. [20] J. McDermott. R l : A Rule-based Configurer of Computer Systems. Artificial Intelligence, 19(1), 1982. [21] D. P. Miranker. TREAT: A New and Efficient Match Algorithm for AI Production Systems. PhD thesis, Department of Computer Science, Columbia University, 1987. (Also appears as Report No. TR-87-03, Department of Computer Science, University of Texas at Austin, 1987.). [22] Daniel P. Miranker. TREAT: A Better Match Algorithm for AI Production Systems. In Proceedings of the Sixth National Conference on Artificial Intelligence (AAAI-87), pages 42-47, July 1987. [23] Daniel P. Miranker, Chin-Ming Kuo, and James C. Browne. Parallelizing Transformations for a Concurrent Rule Execution Language. In International Conference on Parallel Processing, 1990. [24] Dan I. Moldovan. A Model for Parallel Processing of Production Systems. In Proceedings of IEEE International Conerence. on Systems, Man and Cybernetics, pages 568-573, Atlanta, GA, 1986. IEEE. [25] Dan I. Moldovan. RUBIC: A Multiprocessor for Rule-Based Systems. IEEE Transactions on Systems, Man, and Cybernetics, 19(4):699-706, July/August 1989. [26] Keith Morgan. BLITZ: A Rule-Based System for Massively Parallel Architectures. In Proceedings of 1988 ACM Conference for Lisp and Functional Programming, Snowbird, Utah, July 1988. [27] Daniel Neiman. Control Issues in Parallel Rule-Firing Production Systems.

204

[28]

[29]

[30] [31]

[32]

[33]

[34]

[35]

[36]

[37]

[38]

[39]

[40]

[41]

In Proceedings of the Ninth National Conference on Artificial Intelligence (AAAI-91), pages 310-316, Anaheim, CA, July 1991. Daniel Neiman. UMass Parallel OPS5 Version 2.0, User's Manual and Technical Report. Technical Report COINS T R 92-28, Computer and Information Sciences Department, University of Massachusetts, Amherst, MA, 1992. Daniel E. Neiman. Design and Control of Parallel Rule-Firing Production Systems. PhD thesis, Computer and Information Sciences Department, University of Massachusetts, Amherst, MA, September 1992. N. J. Nilsson. Principles of Artificial Intelligence. Tioga Publishing Company, Palo Alto, CA, 1980. Κ. Oflazer. Partitioning in Parallel Processing of Production Systems. PhD thesis, Department of Computer Science, Carnegie Mellon University, 1987. (Also appears as Tech. Rep. CMU-CS-87-114, March 1987.). A. O. Oshisanwo and P. P. Dasiewicz. A Parallel Model and Architecture for Production Systems. In Proceedings of the 1987 International Conference on Parallel Processing, pages 147-153, University Park, PA, August 1987. Louiqa Raschid, Timos Sellis, and Chih-Chen Lin. Exploiting concurrency in a DBMS Implementation for Production Systems. Technical Report CSTR-2179, Department of Computer Science, University of Maryland, College Park, MD, January 1989. James G. Schmölze. Guaranteeing Serializable Results in Synchronous Parallel Production Systems. Technical Report 89-5, Department of Computer Science, Tufts University, Medford, MA, October 1989. James G. Schmölze. Guaranteeing Serializable Results in Synchronous Parallel Production Systems. lournal of Parallel and Distributed Computing, 13(4):348-365, Dec 1991. James G. Schmölze and Suraj Goel. A Parallel Asynchronous Distributed Production System. In Proceedings of the Eighth National Conference on Artificial Intelligence (AAAI-90), Boston, MA, July 1990. James G. Schmölze and Daniel E. Neiman. Comparison of Three Algorithms for Ensuring Serializable Executions in Parallel Production Systems. In Proceedings of the Tenth National Conference on Artificial Intelligence (AAAI92), San Jose, CA, July 1992. Timos Sellis, Chih-Chen Lin, and Louiqa Raschid. Implementing Large Production Systems in a DBMS Environment: Concepts and Algorithms. In Proceedings of the ACM-SIGMOD International Conference on the Management of Data, pages 404-412, Chicago, IL, 1988. S. J. Stolfo. Five Parallel Algorithms for Production System Execution on the DADO Machine. In Proceedings of the Fourth National Conference on Artificial Intelligence (AAAI-84), 1984. S. J. Stolfo and D. E. Shaw. DADO: A Tree-Structured Machine Architecture for Production Systems. In Proceedings of the Second National Conference on Artificial Intelligence (AAAI-82), 1982. Salvatore J. Stolfo and Daniel P. Miranker. The DADO Production System Machine. Journal of Parallel and Distributed Computing, 3:269-296, 1986.

205

r

[42] Victor Vianu and Gottfried Vossen. Conceptual Le\ el Concurrency Control of Relational Update Transactions. In Proceedings of the Second International Conference on Database Theory, Lecture Notes in Computer Science, Volume 326, 1988. [43] D. L. Waltz. Understanding Line Drawings of Scenes with Shadows. In P. Winston, editor, The Psychology of Computer Vision, pages 19-91. McGraw Hill, New York, NY., 1975.

Parallel Processing for Artificial Intelligence 1 L.N. Kanal, V. Kumar, H. Kitano and C.B. Suttner (Editors) 1994 Elsevier Science B.V.

209

Chapter 9

Parallel A u t o m a t e d T h e o r e m Proving* Christian B. Suttner, J o h a n n Schumann Institut

für

Technische Arcisstr. email:

Informatik Universität

21,

D-80290

{suttner,Schumann}

München München @informatik.tu-muenchen.de

Abstract This paper provides a comprehensive overview of paraUel automated theorem proving, containing a description, analysis, and extensive references, for each approach. Implemented systems as well as sufficiently elaborated proposals are included, and grouped according to a new classification scheme, independent of the underlying calculus. This scheme increases the insight regarding similarities and dissimilarities between systems, and includes useful generalizations of some traditional classifications.

1.

Introduction

A u t o m a t e d theorem proving is easily recognized as a central functionality in the context of a u t o m a t e d reasoning and AL Given a set of axioms, the 1 aim is to prove the validity of a proposed t h e o r e m . However, in contrast to logic programming, where the programmer can direct the execution towards determinism, a u t o m a t e d theorem proving inevitably faces combinatorial explosion. T h e enormous search spaces encountered for difficult problems and the desire to increase the range of problems t h a t can be solved suggest the use of parallel machines to increase the available computing power. The utilization of parallel computation for theorem proving is encouraged by the availability of many options for parallelization. A survey of this potential can be found in [1]. It ranges from fine grain approaches such as unification parallelism to coarse grain parallelism such as running several different provers competitively on the same problem. Given the applicability of *This work was supported by the Deutsche Forschungsgemeinschaft within the Sonderforschungsbereich 342, Teilprojekt A5 (ParaUelization of Inference Systems). 1 Technically, this is often achieved by refuting the negation of the theorem.

210

parallelism and the increasing availability of parallel machines over the last decade, it comes as no surprise t h a t recently an increasing number of parallel a u t o m a t e d theorem provers have been built. T h e purpose of this paper is to provide a description, classification, and analysis of the various a t t e m p t s carried out in t h a t field. T h e survey comprises both implemented systems as well as specific proposals. We assume the reader to be familiar with the basic notions of a u t o m a t e d theorem proving (see, e.g., [2-4]).

Related Work. Previous overviews on various parallel logic-based systems (including a u t o m a t e d theorem provers) can be found in [1, 5, 6]. Each of these also describes some parallel Prolog systems, but none of them is as comprehensive regarding pure theorem proving as this paper. Beyond these summaries and the work discussed later on, there is a large amount of work related to parallel deduction in general. Albeit part of it has significantly influenced several parallelizations presented in this paper, we cannot include its treatment here and therefore merely provide some pointers below. Obviously related to a u t o m a t e d theorem proving are parallel systems for propositional logic (for example [6-8]) and those related to parallel logic programming (e.g., [9, 10]). For an overview on concurrent logic programming languages see [11]; other related overviews are found in [6, 12]. We should also mention approaches towards parallel unification (e.g., [13,14]). There are recent a t t e m p t s to utilize extremely fine grain parallelism for theorem proving by means of connectionist approaches. An overview on this is given in [15]. Finally, there is a large number of related fields, where often similar parallelization techniques are relevant. Those are parallel deductive databases, distributed problem solving (e.g., [16]), and other research dealing with (mainly) uninformed search in a combinatorial space (e.g., [17]). Due to the fast growing number of parallel systems and the difficulty of deciding which should be included, this survey cannot be complete.

In Section 2 we present a classification scheme, for parallel reasoning systems, t h a t will be applied to highlight particular properties of the systems considered. In Sections 3 and 4 parallel theorem provers (implemented and proposed) are described, grouped according to the classification scheme. T h e overview is concluded by Section 5 with a summary, including a historical overview. Some information regarding the problems for which experimental d a t a is shown can be found in the Appendix.

211

2.

C l a s s i f i c a t i o n of P a r a l l e l i z a t i o n

Approaches

Besides supplying a deeper understanding of the general issues, considering classifications is interesting for two reasons: first, a proper classification scheme delineates a range of possible parallelization approaches, and thus gives conceptual help for planned parallelization projects; second, the classification of particular systems according to such a scheme makes it possible to estimate system properties which follow as consequences from the distinctions made by the classification. T h u s , in order to be useful, a classification scheme should lead to an unambiguous classification and should employ concepts which are relevant with respect to interesting system properties. Common Classification Schemes. Classification can be performed at a variety of levels. For example, there is a number of issues regarding the (application independent) implementation level, which leads to well-known distinctions such as message passing vs. shared memory, MIMD vs. SIMD, synchronous vs. asynchronous communication, and so on. A large number of issues regarding the parallelization of logic can be found in [1]. However, here we are interested in a high level characterization which is adequate specifically for parallel deduction systems. In general, the most i m p o r t a n t concept used so far for classification of parallel logic systems has been the distinction between AND- and 0 R parallelism. But this scheme has two drawbacks. First, the distinction is based on the notion of A N D / O R - t r e e s . Although every logical formula can be transformed into disjunctive normal form (and thus a combination of AND and O R ' s ) , the operation of many theorem provers (most notably, nonlinear resolution systems) cannot be described by the processing of an A N D / O R - t r e e . In contrast, such systems are usually set-based in the sense t h a t an inference step extends the set of derived consequences, and this new set forms the basis for the next step. T h e same is true in graph-based systems. No A N D / O R structure of operation can be seen. Second, even in cases where an A N D / O R classification is applicable, it can be ambiguous. An example of this is found in the M G T P / G system [18] (see Sec. 3.1.4). T h e system is described to be OR-parallel, since the underlying model generation calculus is parallelized by searching alternative models concurrently. Since it is sufficient to find at least one model, this is obviously and correctly termed OR-parallelism. However, as soon as this system is used for theorem proving, the viewpoint changes. Then, a refutation proof is searched for, by showing t h a t no model exists. In this case, all possible models need to be checked (for a particular subset of first-order

212

logic this set is finite), and what previously was OR-parallelism now becomes AND-parallelism! T h u s , in the context of theorem proving, the syst e m is more appropriately termed AND-parallel. T h e ambiguity is caused by the difference between the logical concept of an underlying calculus and its realization in a proof procedure. Using an A N D / O R classification does foster such confusion, and can lead to significant misinterpretations if it is used to illustrate system properties. Based on initial ideas described in [19], we will present a classification scheme which supersedes the traditional A N D / O R classification and presents a remedy to the above problems.

A Classification Scheme for Parallel Deduction Systems. We propose a 2 coarse grained classification scheme along two orthogonal distinctions . For a primary distinction regarding parallelization, we consider the relation between the search spaces of parallel workers. Parallelization by search space partitioning is the traditional approach, and is intuitive both with respect to how it can be achieved and what the expected performance might be. However, there are now a number of parallel deduction systems where no partitioning of a common search space is performed. For them, the same problem is given to a number of different systems (or the same system using different search strategies). T h u s , we propose to distinguish between partitioning parallelization and system competition parallelization. Partitioning parallelization is where traditional AND/OR-parallelism (problem decomposition) is located, and this logic-oriented distinction will be generalized to the operational concepts of completeness-based versus correctness-based partitioning. System competition parallelization is based on the idea t h a t using different algorithms or variants of the same algorithm lead to different search spaces, and thus provide potential for speed-up. As a secondary distinction we propose to consider the cooperativeness between parallel processes. T h e distinguishing feature here is t h a t information gathered during the processing of a computational task in the parallel system may or may not be shared with other tasks with the intention of saving work. These distinctions lead to the classification matrix shown in Figure 1, and the individual classes will now be discussed in more detail. 2

A related classification scheme for parallel reasoning systems along three axes is briefly described in [20]. The axes refer to the homogeneity, the separation of the intermediate results, and the synchroneity of a parallel system. Besides the synchronization aspects, these issues can be mapped onto the concepts of partitioning/competition and cooperativeness in our classification, respectively.

213

Partitioning Competition

completeness-based correctness-based different calculi one calculus

uncooperative Sec. 3.1.3-3.1.2 Sec. 3.1.4 -

Sec. 4.1.1

cooperative Sec. 3.2.1-3.2.7 Sec. 3.2.5 Sec. 4.2.1 Sec. 4.2.2-4.2.3

Fig. 1. A matrix representation of the classification taxonomy.

2.1.

Partitioning

Parallelization

T h e idea of partitioning parallelization is to split the search space into parts which are searched in parallel.

Completeness-based Partitioning. For completeness-based partitioning, independent parts of the search space are distributed among workers (e.g. OR-parallelism for A N D / O R - t r e e s , set-split parallelism for set-based c o m p u t a t i o n ) . A solution found by an individual worker usually constitutes a solution to the overall problem. Dropping tasks which should be given to workers in general causes incompleteness of the proof search, and hence the name. Many systems are found in this popular category. Examples of purely OR-parallel systems are P A R T H E N O N , P A R T H E O , and M E T E O R (see Sections 3.1.1 - 3.1.3). M G T P / N , P A R R O T , R O O , and D A R E S (Sections 3.2.1 - 3.2.4) represent different completeness-based partitioning approaches to parallel resolution-style theorem proving. Finally, dcdpparallelism (Section 3.2.5) achieves this in the context of connection graph procedures.

Correctness-based Partitioning. For correctness-based partitioning, the tasks (representing parts of the search space) given to workers are interdependent, and an overall solution is built up from the corresponding partial solutions (e.g. AND-parallelism for A N D / O R - t r e e s ) . We chose this n a m e since, in general, dropping tasks which should be given to workers destroys the correctness of a deduction. Examples from this category are found in the already mentioned M G T P / G system (Section 3.1.4) and the use of AND-parallelism in connection graph inference (Section 3.2.5).

214

2.2.

Competition

Parallelizaüon

Competition parallelization is based on the a t t e m p t to solve the same problem using several different approaches. In particular, no partitioning of a common search space is performed. For this approach to be useful it is necessary t h a t either the quality of solutions obtained by competitors varies or the time required to find a solution has a significant variance among the individual systems. Of course, these variations are relevant only if it is not possible to select the system with the best performance in advance (this is the case for a u t o m a t e d theorem proving as well as many other applications). In competition parallelism, similar as for OR-parallelism, the success of a single system is sufficient for a solution and allows the termination of the computation. Obviously, completeness of the overall computation is assured as long as at least one of the competing systems is deduction complete. T h e basic question for such an approach is how competing systems are obtained. There are two principal choices. One option is to use different calculi based on the fact t h a t each calculus has its particular merits and performs well in certain cases, while it may fail to do so in other cases. T h e other option is to use a single underlying calculus, and competition is achieved by using different search strategies for each competitor. This choice leads to the subdistinction discussed below.

Competition using Different Calculi. Here, different algorithmic approaches are used simultaneously either to improve the solution quality or to obtain a solution earlier. An uncooperative parallel system of this kind is quite easy to build, since simply several different systems are started in parallel. Although this has obvious performance potential for a well-chosen set of competitors, when compared with any of the individual systems, we do not know of such an implementation for a u t o m a t e d theorem proving. Cooperative competition, on the other hand, poses many difficult questions regarding the parallel system design. Not only the selection of the individual systems, but also the amount and type of information and the direction and timing of its exchange, are crucial for the overall performance. As a result, it is hardly possible to predict the performance of such an approach prior to an actual implementation. Recently a set of parallel provers employing cooperative competition based on different calculi have been built (see H P D S , Section 4.2.1).

215

Compétition using a Single Calculus. Here, a set of competing systems is used, all of which are based on the same calculus. T h i s is possible when the inferences demanded by a calculus are controlled by various parameters (e.g., by choosing a particular search strategy or by issuing certain search bounds). Each distinct set of control parameters then leads to a different search behaviour. For many systems, this gives rise to a large number of possible variants. T h e speed-up obtainable with this scheme relies solely on the relation between the structure of the problem search space and the adequacy of the control parameters for achieving optimal exploration. For many types of uninformed search, variations of the control parameters cause significant changes in the run-time. T h u s there is a significant potential for improvement over the average performance of the individual competitors. It should be noted t h a t competition based on different system parameters is often performed manually by researchers, whenever they run many experiments with different parameter settings, trying to find adequate choices which lead to a success. T h e simplest approach for obtaining a set of competing systems is by using different seeds for r a n d o m selection of alternative choices during the search process. Examples for this approach can be found in [21, 22] (a further analysis of their model can be found in [23]; also, see [24] for a presentation in a more general setting and a list of references to early, related work). T h e application of this approach to a u t o m a t e d theorem proving ( R C T H E O , see Section 4.1.1) and a comprehensive discussion of its general properties can be found in [25]. Regarding cooperative competition approaches, a description of the first proposal t h a t we know of is found in [26]. More recently, the t e a m work completion system (see Section 4.2.2) utilizes several different strategies for selecting critical pairs. A parallel computation of several different sets-of-support has been proposed for resolution-based theorem proving (see Section 4.2.3). System Classification. An overview of the systems and approaches surveyed in this paper, together with their classification according to the presented scheme, is given in Figure 2. T h e numbers in parenthesis denote the respective sections. T h e table shows t h a t the classification splits the parallel systems nicely, grouping intuitively similar systems together. Quite naturally, the class of cooperative completeness-based systems is the largest, as this is the traditional approach for parallelization (split the work, and exchange derived information). T h e only class without a system, namely uncooperative competition based on different calculi, is obviously promising from a pragmatic viewpoint, as it allows to obtain the best result for each problem achievable by any of the competing systems. However, natu-

216

rally this class receives little attention by researchers as no interesting new insights can be expected from such an exercise. uncooperative

partitioning

completenessbased

PARTHENON (3.1.1) PARTHEO (3.1.2) METEOR

(3.1.3)

cooperative MGTP/N (3.2.1) PARROT-I/II (3.2.2) ROO

(3.2.3)

[DARES]

(3.2.4)

[dcdp-parallelism]

(3.2.5)

[Clause Diffusion] (3.2.6) [Wivenhoe Model] (3.2.7) partitioning compet.

correctnessbased different calculi

compet.

single calculus

MGTP/G RCTHEO

(3.1.4)

[conn.graph-AND] (3.2.5)

HPDS

(4.2.1)

(4.1.1)

TWC [MSOS1

(4.2.2) (4.2.3)

Fig. 2. A classification of the implemented or proposed parallel automated theorem proving systems which are discussed in the following sections. Simulated systems as well as proposals are embraced by "[""]".

3. 3.1. 3.1.1.

Partitioning-based Parallel T h e o r e m Provers Uncooperative

Schemes

PARTHENON

P A R T H E N O N (PARallel T H E o r e m prover for NON-Horn clauses) [27] is an OR-parallel theorem prover for first order predicate logic. T h e underlying proof calculus is a variant of Model Elimination [3]. Computational Model. P A R T H E N O N exploits OR-parallelism by dynamically executing independent parts of the search tree on different processors. T h e underlying parallel computational model is similar to the SRI-Model [28] for the OR-parallel execution of P R O L O G on architectures with shared memory. Each processor performs a depth-first search (with iterative deepening) on a given subtree of the problem. If a processor has finished executing its subtree, it must look for further work by "stealing" subtrees which are still unexplored. T h e P A R T H E N O N model of computation involves the following improvements over t h e SRI-model: Scheduling. T h e scheduling algorithm (local to each processor) is called whenever a processor cannot find any unexplored alternatives at the current node. T h e scheduler then tries t o find an unexplored node

217

in the tree, looking at parents, children, and siblings of the current node. This entirely distributed scheme of scheduling exhibits a low overhead since in most cases, an unexplored subtree can be found in the "vicinity" of the old one (in less t h a n 3 % , further away t h a n one move, cf. [27]). Furthermore, this scheduling algorithm helps to reduce the time needed for context switching. Variable Binding and Dereferencing. T h e binding of variables is much more complex t h a n in the sequential case, since one variable may be bound to different values in different parts of the search tree. In the SRI model this problem is handled by "binding-arrays", which contain all the variable bindings a particular processor needs. Such binding arrays are kept in the processor's local memory. This allows the most common operations, like dereferencing or binding a variable, to be executed in constant time. In the P A R T H E N O N model, the binding arrays are replaced by a variable stack local to each processor. T h e choice-point stack and the trail are shared. Furthermore, all variable bindings (and not only the conditional ones) are trailed. Trailing of variable bindings is necessary for "undoing" bindings during backtracking steps. W i t h t h a t method for variable binding and dereferencing PART H E N O N is about 20-50% faster t h a n a version using the original SRI-model.

Implementation. P A R T H E N O N runs on various multi-processors, such as an Encore Multimax with 16 processors and 32 MBytes of shared memory, and an IBM R P 3 with 64 R O M P processors. T h e prover is implemented in C using C-threads under the MACH operating system.

Experimental Results. Table 1 shows a selection of results of measurements obtained on the Encore M u l t i m a x and the IBM R P 3 , as presented in [27] (Fig. 21 & 24). T h e execution time is given in seconds.

Assessment. T h e good speed-up figures in Table 1 indicate t h a t the exploitation of OR-parallelism leads to good speed-ups on classical theorem proving problems. In contrast to M E T E O R (cf. Section 3.1.3), PART H E N O N employs an entirely distributed control and distribution of work. T h e scalability of this model has been tested using up to 60 processors. The basic inference component is an interpreter only, but features fast variable dereferencing and low cost variable binding.

218

Multimax, ρ

Problem apabhp ls36 wos4 woslO wos21

1 2886 2273 14000 159 3315

5 645 323 1225 38 661

RP3, ρ =

= 15 292 151 501 20 373

10 1093 — — 152 —

30 647 1122 606 82 1180

60 339 526 265 83 697

Table 1 Execution times in seconds for PARTHENON on an Encore Multimax and an IBM RP3.

3.1.2.

PARTHEO

P A R T H E O [29, 5] is an OR-parallel theorem prover designed for a distributed-memory multi-processor system with message passing as its method of communication. P A R T H E O and R C T H E O (see Section 4.1.1) are based on the SEquential T H E O r e m prover S E T H E O . Underlying System: SETHEO. S E T H E O [30] is a theorem prover for full first order predicate logic in clausal normal form. T h e calculus underlying S E T H E O is Model Elimination [3]. An abstract machine, the S A M , which is an extension of the well known Warren Abstract Machine [31], is used to search for a proof. T h e S A M performs a depth-first search with backtracking. Completeness is obtained with iterative deepening over a variety of bounds (e.g., depth of the search tree, number of inferences in the proof, number of admissible copies of each clause) and their combination. Several powerful methods for reducing the search space are implemented in S E T H E O , enforcing the construction of tableaux with certain restrictions (e.g., regularity). For details see [30]. Computational Model. OR-parallelism is exploited in P A R T H E O by distributing parts of the OR-search tree among the processors. This is accomplished by splitting up the OR-tree into independent tasks, each of which consists of one (open) Model Elimination tableau (i.e., a node of the OR-tree) with a distinguished literal which is to be extended in the next step. New tasks are generated whenever the S A M encounters a branching point in the OR-tree, i.e., when there exists more than one possibility for perform3 ing an extension s t e p . Newly generated tasks are put into the Task-Store in the processor's local memory. Each processor takes a task from its local 3

T h e different possibilities for performing a reduction step are not explored in parallel.

219

task-store and tries to expand it. If the resulting tableau is closed, i.e., a proof has been found, the proof-tree is sent to the host and all other processors are stopped. If the expansion of the tableau fails, a new task has to be fetched from the processor's task store. If no tasks can be found in the local task-store, the processor asks its direct neighbors for work. If one of t h e m has enough work (e.g., more t h a n one task), it gives away some of its tasks (e.g., half of its tasks). T h e entire system is started by placing the initial task into the task store of one processor. Each node of the P A R T H E O system consists of a copy of the S A M , the Local Task Store, and the Communication Unit. T h e Communication Unit is in charge of transferring tasks from one processor to the other. Since the transfer of an entire tableau (i.e., a tree of literals) would cause too much overhead in a message passing architecture, the tasks are sent in a coded way which allows an easy and fast reconstruction of the tableau in the receiving processor. W i t h this coding scheme, only 4 Bytes per inference in the current tableau need to be transferred. Implementation. P A R T H E O has been implemented on a network of 16 Transputers T800 and an Intel Hypercube IPSC32 with 32 nodes (80386). T h e implementation on the network of transputers has been written in 3L parallel C [32] using the communication primitives and light-weight processes of the transputer. T h e network of 16 transputers, connected in a torus-like topology, is controlled by a SUN i386 workstation. This host machine contains the file-system and the user interface. T h e communication part of the P A R T H E O implementation on the Intel hypercube is based on the primitives provided by the MMK system [33, 34]. Asynchronous mail-boxes are used for communication between processes. T h e current implementation of P A R T H E O does not support iterative deepening. Rather, a depth-bound large enough to find a proof for the given formula is given. A detailed description of an entirely distributed scheme for iterative deepening can be found in [29]. Experimental Results. Measurements have been made with many benchmark examples from the area of theorem proving and logic programming. A selection of results obtained with the transputer system, presented in [5], is shown in Table 2. A direct measurement of the processor utilization and the load balance could not be m a d e on the given hardware configuration. Therefore, the task-stealing mechanism of P A R T H E O has been modeled as an extended queueing network. Simulations then have been made using MAOS, a system for modeling, analysis and simulation of extended queueing models

220

[35]. Simulations have been carried through with different parameters of the task stealing model [36, 37] and different number of processors [38]. T h e experiments revealed t h a t the work is being distributed rather quickly among the processors and t h a t a good load balance and processor utilization is obtained most of the time. However, the behavior of the task stealing model, especially on large networks, is stronly influenced by the parameter setting. Problem

wos4 woslO wosl6 ls36 lsl08 lsl21 non-obvious lukasll 10-queens

T ( l ) [sec] 3413.0 262.6 36490.0 1666.0 136.6 44.2 4.5 8.4 370.5

T(16) [sec] 1.84 135.60 5.89 352.00 15.04 10.25 1.84 .66 24.68

s - m i 3 1.85 x l O 1.94 3 6.19 x l O 4.73 9.08 4.32 2.44 12.78 15.01

Sr =

-

1.16 xlO'η .121 2 3.87 x l O .296 .568 .270 .153 .799 .983 2

Table 2 Performance of P A R T H E O on a torus-like network of 16 transputers T800. Times are in seconds.

Assessment. Measurements with P A R T H E O indicate t h a t the exploitation of OR-parallelism seems to be very appropriate for proving theorems, due to the existence of huge search spaces t h a t often contain short proofs. T h e task-stealing model of P A R T H E O , with its coding of tasks, provides an entirely distributed control of the distribution of proof-tasks. Therefore, the system can easily be implemented on parallel hardware with message passing as its communication mechanism. 3.1.3. METEOR M E T E O R [39-41] is a theorem prover for first order predicate logic based on the Model Elimination Calculus [3]. It is capable of running on a sequential machine, as well as on parallel machines and on a network of workstations. In the latter cases OR-parallelism is explored. Computational Model. T h e architecture of M E T E O R is based on the WAM [31]. Instead of compiling the clauses into machine instructions of the WAM, a d a t a structure is created out of the formula. T h e d a t a structure is interpreted by the abstract machine.

221

W h e n M E T E O R is running on parallel hardware or on a network of workstations, it exploits OR-parallelism by executing the alternatives for 4 Model Elimination extension steps in parallel . This corresponds exactly to distributing the OR-search tree among the processors. Work is distributed using a "task stealing" mechanism (similar to, e.g., P A R T H E N O N , Section 3.1.1 or P A R T H E O , Section 3.1.2): whenever a processor has finished its proof task, it tries to get a new proof task from a server or a common task pool. T h e loading of the new proof task into the WAM is performed by a recomputation of the WAM's state. T h e theorem prover M E T E O R further contains a caching and lemmaizing facility in order to avoid redundant explorations of the search tree. Iterative deepening can be performed on the depth of the tableau and the number of inference steps. Further refinements for the iterative deepening, e.g. the roll-back mechanism [42] or weighted depth bounds have been developed and integrated into M E T E O R . Parallel METEOR. This version of M E T E O R uses a concurrent pool 5 (SASS-pool ) for storing the proof tasks (i.e., the choice-points of the WAM). Whenever a processor becomes idle it may "steal" one proof task out of the S ASS pool and execute it. W i t h this model the pool does not migrate the tasks, but makes t h e m available for stealing using a concurrentread, exclusive-write locking protocol. Distributed METEOR. T h e distributed version of M E T E O R runs on a network of UNIX workstations, connected via a local area network. T h e architecture of this version consists of a " M E T E O R server" process and one or more " M E T E O R engines". T h e server communicates via UNIX sockets with the M E T E O R engines, in a star-shaped communication topology. Each M E T E O R engine can be in one of the states query (idle and asking for work), working (performing extension and reduction steps), or must share (the engine sends proof tasks to the server). In the query mode, the engine sends a request for work to the server, which sends back a proof task to be processed. T h e engine from which t h a t task originated is asked to send, one or more new tasks to the server by setting its state to must share. As a consequence, the engine sends all its newly generated proof tasks to the server. If the server detects t h a t all processors are in the query mode and no tasks are available, the bound for iterative deepening is increased and the search starts again. 4

As in PARTHEO (cf. Section 3.1.2), the Model Elimination reduction steps are not tried in parallel. 5 S ASS stands for "Simultaneous Access of Selectively Shared objects".

222

Implementation. METEOR MACH). T h e parallel version (with a Motorola 88000 each). communication and runs on a

is implemented in C (under UNIX and runs on a Butterfly TC2000 with 45 nodes T h e distributed version uses sockets [43] for network of workstations.

Experimental Results. Table 3 summarizes results of measurements for the 6 parallel version on a TC2000, and the distributed version of M E T E O R . T h e results have been taken from [41].

Prob ls36 wosl woslO wos21 wos22

Execution times in seconds Parallel METEOR, ρ — Distributed METEOR, 10 20 35 1 1 5 10 343.2 22.19 9.86 12.59 208.2 21.72 13.58 59.57 8.38 6.3 5.23 38.78 8.16 5.47 42.54 7.26 2.98 2.83 25.63 8.15 5.77 1210 190.79 92.42 43.29 704 185.36 57.35 43571 8139 6309 4494 22565 6585 5259

V

=

20 8.48 3.46 3.46 34.74 3054

Table 3 Execution times of the parallel (shared memory) and distributed (workstation network) versions of M E T E O R .

Assessment. T h e M E T E O R system is a single piece of software, usable on sequential machines, parallel computers and networks of workstation. The prover uses a flexible scheme for iterative deepening to ensure completeness. T h e central control for task stealing (SASS-pool or M E T E O R server), allows to efficiently control the iterative deepening, however, it may become a bottleneck with an higher number of processors. 3.I.4. MGTP/G M G T P [18] is a family of parallel model generation based theorem provers for first order predicate logic. M G T P / G is for ground or range-restricted 7 p r o b l e m s , whereas M G T P / N is for non-ground Horn problems. M G T P / N is described in Section 3.2.1. Both M G T P provers are implemented in KL1 on a distributed memory multiprocessor. Underlying Calculus. T h e calculus underlying M G T P is Model Generation, which was first used in S A T C H M O [44]. Given a formula in clausal 6

T h e execution times for the Parallel METEOR are mean times, those for the distributed METEOR are maximal times. For details see [41]. 7 A clause is range-restricted, if each variable in the clause has at least one occurrence in its antecedent.

223

form, M G T P a t t e m p t s to generate a model for the formula. It starts with an empty set as a model candidate. T h e model candidate is then extended by successively adding instances of consequences of such clauses which have an antecedent which is satisfiable in the model candidate. This calculus has a strong relationship to Model Elimination [3], but since Model Generation starts at positive clauses, it proceeds in a b o t t o m - u p fashion. In the case t h a t all clauses of a formula are range-restricted, matching is sufficient and no full unification is needed. Since KL1 is a committed choice language which only provides fast one-way unification, this feature is of great interest. Therefore, a specialized version of M G T P , namely M G T P / G , has been developed.

Computational Model. For ground or range-restricted problems, the clauses are directly compiled into a set of KL1 clauses. An additional interpreter in KL1 controls the generation and rejection of models. Three methods have been developed in order to avoid redundancies when computing the "conjunctive matching", i.e., when matching the antecedent literals against elements in a model: the R A M S (RAMified Stack) method [45], the M E R C (Multi-Entry Repeated Combination), and the Δ - Μ method. In the first approach, intermediate results are stored to avoid redundancies. T h e M E R C and Δ - Μ method also utilize intermediate matching results to avoid redundancies. These results, however, are not stored, but they are recomputed when needed. Furthermore, a "lazy" model generation has been proposed in [46] to enhance the performance of M G T P . There, model candidates are generated only if they are needed for the satisfiability test ( "generate-and-test" ). T h o u g h there are several opportunities for parallel execution, M G T P / G only exploits parallelism induced by case-splitting when non-Horn clauses are processed. From the viewpoint of the calculus, this corresponds to OR-parallelism. In the context of a u t o m a t e d theorem proving, all cases must be refuted for a proof. Therefore, this kind of parallelism needs to be classified as correctness-based partitioning (see discussion in Section 2). In order to obtain a smooth load balance, processors are allocated in a "bounded parallelism" manner: parallel forking in the proof process can be suppressed if there are currently not enough idle processors available. This is accomplished by distributing an OR-branch (a model candidate augmented with a literal of a disjunctive clause) to a different processor every time case splitting occurs, until reaching a specified level in the proof tree. After having reached the level, the candidates in each processor are expanded by the processor itself without causing further distribution.

224

Implementation. M G T P / G is implemented in KL1 [47] and running on Multi-PSI [47] and P I M / m [48] machines. Experimental Results. Table 4 shows the results of measurements on a Multi-PSI, taken from [18]. Table 5 presents run-times of M G T P / G obtained on a P I M / m machine with 256 processors.

Problem 6-qu.-all 8-qu.-all 10-qu.-all

1 0.65/1 12.54/1 315.5/1

Execution Ti me in Second s/speed-up, ρ 2 4 8 16 0.27/2.44 0.41/1.59 0.19/3.44 0.15/4.22 6.43/1.95 3.34/3.76 1.82/6.91 1.01/12.48 159.9/1.97 79.92/3.95 40.85/7.72 21.8/14.46

Table 4 Performance of M G T P / G on Multi-PSI for the queens problem.

Problem 11-queens-all 14-queens-all bennetl2

T(256) 2.3 128.0 13101.0

Table 5 Execution times (in seconds) of M G T P / G / M E R C / P F on PIM/m-256 with 256 processors.

Assessment. T h e provers M G T P are actually a family of model generation theorem provers which exploit different models of execution. T h e efficient implementation of M G T P / G for ground and range-restricted problems allows M G T P / G to solve very hard problems in this area. 3.2.

Cooperative

Schemes

3.2.1. MGTP/Ν 8 M G T P / N [18] is a parallel theorem prover for arbitrary Horn-formulas , especially suited for non-ground problems. T h e underlying calculus, Model Generation, and the ground version, M G T P / G , are described in Section 3.1.4. 8

M G T P / N does allow splitting only in the ground case. Therefore, MGTP/N is restricted to Horn clause logic.

225

Computational Model. M G T P / N exploits parallelism in the conjunctive 9 matching and subsumption phase of the p r o v e r . Several models of execution have been developed for M G T P / N . Model candidates may be shared among the processors, in the sense t h a t all processors have a copy of the model candidates, whereas only the model-extending candidates are distributed among the processors. In the "Model distribution" version, all model candidates are distributed. Furthermore, a distinction is made if the number of processes can influence the structure of the proof or not. In the "Proof Changing" version (using a ring topology), all processors work in a pipelining manner in order to extend the model. On the other hand, the "Proof Unchanging" version uses model sharing based on lazy model generation. Tasks from a master processor are executed on slave processors which perform generation, subsumption, and satisfiability testing. Generation and subsumption are controlled by the tasks, whereas the testers run in a data-driven mode, i.e., they are activated only when model candidates are to be tested for satisfiability. Implementation. M G T P / N has been implemented in K L / 1 on the MultiPSI [47] and P I M / m [48]. Experimental Results. Table 6 shows the results for several problems, on the P I M / m with 256 processors. According to [18], the "Proof Unchanging" version of M G T P / N produces better speed-up values and a better load-balance for many problems t h a n the "Proof changing" model of execution. Experiments have shown t h a t an almost linear speed-up can be obtained (on problems with condensed detachment [49]) for up to 128 processing elements. Of major importance for the efficiency of M G T P / N is the implementation of the full unification in K L 1 . Experiments show t h a t full unification with occurs check for M G T P / N on P I M / m - 1 P E is about 4 times slower t h a n t h a t of O T T E R (see Section 3.2.3) on a SPARC-II. Taking into consideration t h a t the hardware speed of P I M / m - 1 P E is about 2 times slower t h a n t h a t of a SPARC-II, the overhead for a unification implemented in KL1 is relatively low, compared to one written in C. Assessment. Several different versions of M G T P / N (e.g., proof-changing and proof-unchanging) have been developed and results have been reported. T h e advantages and disadvantages of each architecture, however, are not 9

Unfortunately, the literature available to us did not fully clarify the computational model of MGTP/N. According to discussions with M. Fujita, MGTP/N employs a cooperative (deduced consequences are sent to other processors), completeness-based partitioning scheme (each processor deduces the consequences of a subset of the given clauses) of computation.

226

Problem Det-#44 Det-#49 Det-#60 Det-#67

T(256) 1350.32 2512.42 1051.15 340.28

T(l)/T(256) 231.36 237.76 154.24 166.40

Table 6 Runtime and speed-up of MGTP/N ("Proof Unchanging", model sharing) on PIM/m-256PEs. Times are in seconds.

represented in the measurements. Surprisingly, an implementation of fullunification with occurs check can be implemented quite efficiently in KL1, despite the fact t h a t KL1 only provides one-way matching. 3.2.2.

PARROT-II

P A R R O T - I I (for PARRallel based on hyperresolution.

OTter,

[50]) is a parallel deduction system

Predecessor System: PARROT-I (= POTTER-I). Founded on the work on a parallel propositional theorem prover in [7], the first order theorem prover P A R R O T - I was built [51, 50]. In P A R R O T - I , a master process selects a given clause for each of a number of slaves, and each slave generates the resolvents for its given clause. After a slave has derived all resolvents, it sends the results to the master and requests a new given clause. T h e master process performs forward- and backward-subsumption checks on the incoming clauses and maintains the master version of the clause database, updates of which are sent to the slaves. T h e slave databases are tightly synchronized with the master version. T h e disadvantages of this scheme 10 are substantial memory u s a g e (a copy of the database is stored at each slave) and a severe bottleneck through the master process, which limited the achievable speed-up to at most seven. P A R R O T - I is reported to run on an Encore shared memory computer with 20 processors. In the following, we will describe P A R R O T - I I , an improved successor of P A R R O T - I . Computational Model. T h e basic sequential deduction algorithm underlying P A R R O T - I I was adopted from O T T E R [52]. In P A R R O T - I I , the generation of resolvents is performed by a set of generator processes (see Figure 3). Each generator maintains three d a t a 10

This limits the number of slaves on a shared memory machine. The problem was slightly alleviated by removing the set-of-support from the slave databases and by using a different coding scheme for clauses. For details see [51].

227

structures: - A set-of-support, at least one member of which must participate in a deduction step. - A usable-list, members of which are allowed to be used in a deduction step. - A demodulator-list, members of which specify syntactic transformations to be applied to newly derived clauses. A deduction step consists of selecting a new given clause from the set-ofsupport, moving the given clause from the set-of-support to the usablelist, deriving all resolvents t h a t are immediate descendants of the given clause, simplifying the resolvents according to the demodulator-list, delet11 ing clauses t h a t are too h e a v y , and finally sending the resolvents to a checker process. Each checker process receives clauses from one or more generators, stores a coded representation for those clauses t h a t are not yet contained in its local clause list, and sends those clauses to a subsumer process (thus, duplicates are eliminated by the checker). Each subsumer process receives clauses from one or more checkers, deletes clauses t h a t are forward-subsumed by clauses already stored in the subsumers local clause list, u p d a t e s its clause list with non-subsumed clauses, and sends those to the distributor. T h e distributor receives clauses from one or more subsumers, and performs a final forward-subsumption check (a clause arriving from one subsumer may subsume a clause arriving from another subsumer). T h e distributor updates its set-of-support, selects some clauses for each generator, and sends the complete set-of-support to each generator. It also remembers all clauses sent to generators in a has-been-given list (for a more effective subsumption check). T h e loop closes as each generator moves received clauses either into its set-of-support or to its usable-list, as specified by the distributor. T h e processes and their communication relation can be viewed as a treestructured set of converging pipelines, as shown in Figure 3. Implementation. T h e implementation reported for P A R R O T - I I is written in C and utilizes two Sequent Symmetry machines (shared-memory machines with 24 processors each). Interprocess communication is handled using the p4 macro package [53]. 11

The weight of a clause is a metric on its complexity. Since clauses above a certain complexity are deleted, the system is incomplete.

228

Distributor

Subsumers

Checkers

Generators

Fig. 3. A typical process configuration and the flow of information for P A R R O T II.

Experimental Results. T h e information in Table 7 is adapted from [50], and gives performance results for two problems. Absolute run times of O T T E R for the two problems were reported to be 7 hours for the imp-4 problem and 42 seconds for sam's lemma (on one processor of the parallel machine). T h e exact number of processors utilized is unknown due to multiprogramming usage of the machine, but was estimated to be roughly one processor per process.

#

processes

29 20 16 12 10

N u m b e r of Generators - Checkers - Subsumers

18-6-3 12-4-2 8-4-2 6-2-2 4-2-2

approx. speed-up for i m p - 4

23 15 8 5 5

approx. speed-up for s a m ' s l e m m a

6 4 6 4 3

Table 7 Performance of PARROT-II using different process ensembles compared to OTTER.

It has been observed [50] t h a t P A R R O T - I I produces larger proofs t h a n O T T E R . Furthermore, the sizes of produced proofs and the runtimes varies for different runs.

229

Assessment. Despite removing some of the bottlenecks from initial implementations, the performance of this system is not satisfying (esp. when compared with R O O , see Section 3.2.3). Unfortunately, an elaborate analysis of the system is lacking. 3.2.3. ROO R O O (for Radically Optimized Otter, see [54, 55] for descriptions and experimental evaluation, and [56] for a short s u m m a r y ) is a parallel version of the a u t o m a t e d reasoning system O T T E R . Underlying System: OTTER. O T T E R [52] is an a u t o m a t e d theorem proving system based on resolution. Among the inference rules supported are binary resolution, hyperresolution, and paramodulation. Knuth-Bendix completion can also be performed. T h e implementation is focussed towards efficient handling of large clause databases and redundancy control (subsumption). O T T E R is primarily based on two clause sets: the usable-list and the set-of-support. According to the set-of-support strategy [57], at least one clause from the set-of-support must participate in each inference step, and arbitrary many clauses may participate from the usable-list. T h e search terminates as soon as a proof has been found (i.e., the empty clause has beed derived) or the set-of-support is empty. Until then, the following procedure is continually executed: A clause is selected from the set-ofsupport (called "given clause") and moved to the usable-list. Then all clauses which can be derived in one step from the given clause together with other clauses in the usable-list are generated (including some processing such as demodulation). Those new clauses which are not forward-subsumed by an existing clause are added to the set-of-support, and can be used for a backward-subsumption test eliminating subsumed clauses in the usable-list and the set-of-support. Computational Model. T h e computational model of R O O is based on a parallel algorithm for computing the closure of a set under an operation (see [58]). T h e a d a p t a t i o n of this algorithm to O T T E R led to a scheme where each processor runs a process, each of which temporarily executes one of the following four types of task (see also Fig. 4): Task A (generating clauses): take a clause from the set-of-support as given clause, derive all its immediate descendants, perform forward demodulation and forward-subsumption on each descendant, and add those which are not forward-subsumed to an intermediate clause set (To-be-added list). Task Β ( u p d a t e database): Check a special To-be-deleted list and deallocate clauses where this can be done safely (i.e., it is ensured by time-stamps

230

t h a t all operations involving this clause have ceased). Remove a clause from the To-be-added list, redo all clause processing (forward-demodulation and forward-subsumption) and, if it passed the subsumption test, add it to the set-of-support (as well as to the list M and, if it is an equality, to the list Ν in order to make the clause available for tasks C and D, respectively). Task C (back-subsumption): take a clause from the M list as tester; times t a m p and move back-subsumed clauses into the To-be-deleted list. Task D (back-demodulation): take a clause from the Ν list, use it for backdemodulating clauses in the database, insert new clauses into the intermediate clause set, and time-stamp and move deletable clauses (i.e., parents of the new clauses) into the To-be-deleted list.

Fig. 4. The process structure and data flow in ROO. In order to avoid multiple copies of the same clause, only one process is allowed to perform task Β (update of the master database) at any time. For all other tasks multiple instances are allowed. After a process finishes its current task, the next task to be executed by the process is selected via the following scheme: if the To-be-added list or the To-be-deleted list are non-empty and task Β (updating the database) is not already active, start task B. Otherwise, if the M list is non-empty, start task C (backsubsumption). Otherwise, if the Ν list is non-empty, start task D (backdemodulation). Otherwise, if the set-of-support is non-empty, start task A (generate clauses). T h e To-be-deleted list is introduced in order to avoid the problem of using and deleting a clause by different processes at the same time. Time-

231

s t a m p s are used to avoid the deletion of clauses while some tasks may still be using them in their operations. Implementation. T h e implementation of R O O is basically an extension of O T T E R [52]. It is implemented in C using the p4 macro package [53] for handling the parallelism. R O O runs on shared memory multiprocessors where p4 is available. Some details of the implementation are revealed in [59]. Experimental Results. T h e results given in Tables 8 and 9 are taken from [55], and were obtained on a Intel-386 based Sequent Symmetry with 26 processors and 32 MByte shared memory. Speedups are derived with respect to R O O running on a single processor (ROO-1). sam's lemma Generated Clauses Kept Clauses Memory (K) Seconds | Speedup f3b3 Generated Clauses Kept Clauses Memory (K) Seconds | Speedup

OTTER 5924 134 95 30.7|1.0 OTTER 8616 1435 6642 1057|1.0

ROO-1 5981 134 159 32.0|1.0 ROO-1 8616 1435 6706 1063|1.0

ROO-4 5977 131 220 7.9|4.0 ROO-4 8616 1435 8716 277|3.8

ROO-8 6079 131 344 4.4|7.3 ROO-8 8616 1435 11270 143|7.4

ROO-16 6208 130 592 2.8|11.4 ROO-16 8616 1435 14652 78|13.6

ROO-24 5837 124 840 1.6|20.0 ROO-20 8616 1435 23139 66|16.1

Table 8 Detailed performance of ROO for sam's lemma and for f3b3.

Problem apabhp imp-4 luka5

OTTER 55.5|0.8 29098|1.0 4872|0.7

ROO-1 46.9|1.0 29985|1.0 3597|1.0

ROO-4 13.1|3.6 7180|4.2 869|4.1

ROO-8 5.5|8.5 344118.7 480|7.5

ROO-16 5.7|8.2 1844|16.3 244|14.7

ROO-24 5.4|8.7 1269|23.2 180|20.0

Table 9 Runtime in seconds and speed-up of ROO for three more problems.

Assessment. For this system, a number of detailed experiments have been published. T h e evaluations show a high variability in behaviour for different problems, but nevertheless good performance on average. T h e current model seems adequate for shared memory systems only, and therefore scalability is limited.

232

3.2.1 DARES DARES (for Distributed Automated REasoning System; see [60, 61]) is a parallel a u t o m a t e d theorem prover, for first order logic, based on binary resolution. Underlying Calculus. T h e calculus employed in DARES is binary resolution. Since no factorization is applied, the system is deduction incomplete. Redundancy control (and thereby search space restriction) is performed by tautology deletion and subsumption. T h e search strategy is a saturation level method using a set-of-support strategy and a shortest-clause-first selection. Computational Model. In D A R E S , the set of clauses comprising a proof task is distributed among a set of identical provers. All provers running on a single processor form, together with a process for handling communication, an agent (the active processes compete for the C P U in a round-robin manner). Clauses of the input formula may be distributed redundantly among agents, but not among the provers of a single agent. Due to the distribution of information, typically no prover will be able to solve the problem on it's own in the beginning. Each prover performs inference steps until either the empty clause is derived (a solution is found) or the available clauses are judged to be insufficient for progress towards a proof, either because no more resolvents can be generated, or due to a heuristic assessment. For this, a forward 12 progress test is m a d e at the end of each saturation level. If the test fails, a "knowledge request" is made. For such a request, an agent computes 13 a particular set of l i t e r a l s and sends it to other agents, each of which determines the clauses t h a t unify with some of the literals (those clauses are tagged in order to avoid their consideration in later requests). If no information can be imported after such a request, the conditions for the set are relaxed and the request is repeated with a larger set of literals. Implementation. A simulation of D A R E S has been implemented in SIMU L A C T [62], a distributed testbed facility running on a network of LISP 12

This test fails if no resolvent fulfilled a particular "simplicity criterion" (called proof advancing heuristic) in the current and the last saturation levels and the number of distinct predicate symbols in resolvents of the current level is larger or equal to the number of distinct predicate symbols obtained in the previous saturation level (called monotonie search heuristic). 13 The so-called minimum priority literal set consists of the union of the literals of all clauses which are heuristically determined to have at least a certain likelihood for contributing to a proof, and from which all literals subsumed by others in the set are removed.

233

machines. Simulation Results. In [61], simulation results regarding the effects of varying the number of agents, the dependence of the performance on particular initial clause distributions, and the effect of redundant storage of clauses at different agents are presented. As one of the results it was found t h a t little knowledge redundancy leads to the best results. T h e influence of the number of agents is less clear cut; in the inspected range, from one to ten agents, a slight improvement for more agents was noted. Assessment. This approach is a rare example of a distributed system approach without global view or control. An a t t e m p t is made to solve problems occurring for multiple agent scenarios (such as "Who communicates what with whom when?") for the specific application of a u t o m a t e d theorem proving, and light was shed on the effects of distribution of information. 3.2.5. Parallel Connection Graph Procedures Several proposals have been made regarding parallel connection graph resolution. Underlying Calculus: Connection Graph Procedures. A connection graph is a representation format for first order formulas in clausal form. Initially the graph contains one node for each clause of the original formula. Each literal of a clause is connected to all unifiable literals of opposite sign (complementary literals). If a pair of complementary literals exists within a clause, the connecting pseudo-link represents a link between two copies of the clause. In connection graph procedures, a refutation proof is searched for by link resolution, an inference rule in which a link is selected and the complementary literals are unified. T h e resulting resolvent is added to the graph and the original link is removed. T h e resolvent inherits the unifiable links from its two parent clauses, extended by the appropriate substitutions. This process repeats until the empty clause is derived or no more inferences can be drawn. I m p o r t a n t refinements include the elimination of pure clauses (nodes containing a literal with no links) and tautologies. Resolvents with such properties are simply never added to the graph. Moreover, failing unification in link inheritance may cause a literal to become pure, causing the deletion of its clause. Since all its connecting links are deleted along with a clause, this can trigger a snowball effect which can reduce the size of the graph significantly.

234

de Parallelism, de (for distinct clause) parallelism is based on parallel link resolution where the concurrent resolution of two links of the same clause is disallowed. This approach is found in an early proposal for a parallel connection graph procedure [63], where parallel link resolution is controlled by a central supervisor. Unfortunately, unrestricted dc-parallelism may result in logical inconsistency of the graph [64, 65]. dedp Parallelism, dedp (for distinct clause, disjoint pair, see [66]) parallelism restricts concurrent link resolutions further t h a n dc-parallelism. T h e set of links t h a t can be resolved in parallel (dcdp-links) are the links incident to distinct clauses and edge disjoint pairs. More precisely, the dcdp-links of link L (compatible links) are all existing links in the graph except L itself, except the links of the clauses connected by L, and except the links of the clauses which have links to the two clauses connected by L. Determining a maximal set of dcdp-links is equivalent to optimal graph coloring, a problem known to be NP-hard. An a d a p t a t i o n of a heuristic-based suboptimal graph coloring algorithm for determining a dcdp-link set is described in [66]. Two versions of a prototype implementation of dedp parallelism on an Encore Multimax shared memory multiprocessor machine are reported in [67], but no evaluation is given. AND Parallelism. For AND-parallel link resolution (proposed in [66]), all literals of a selected clause are resolved concurrently (thus, the selected clause is substituted by all resolvents t h a t can be generated from it). AND-parallel link resolution is not refutation complete by itself, but can be used in addition to link resolution to reduce the connection graph (for this, strategies for an adequate selection of nodes for AND-parallel link resolution still need to be developed). Its correctness has been proven in [64, 65]. In [66], a restricted form of AND-parallelism is proposed, in which pseudo-links and bindings of function terms to a constant or a variable are excluded (the latter condition simplifies the issue of consistent unifiers). Precedence Graphs. In [68], precedence graphs are described to avoid logical inconsistencies due to unrestricted concurrent resolution. T h e considerations are based on the Bernstein conditions [69], which are general conditions limiting the access to shared d a t a . T h e conditions avoid inconsistencies t h a t arise through out of order d a t a manipulations due to parallelism. T h e authors analyze reasons for logical inconsistencies, and present a relaxation of the overly restrictive Bernstein conditions to increase the available parallelism. Finally, suggestions for the synchronization of the critical regions enforced by the precedence graphs are made: a

235

lock and withdraw scheme (free all clauses if some of the involved clauses are locked) and a graph partitioning scheme (processors work primarily on individual subgraphs, and synchronization is necessary only at the borders). T h e latter scheme may be extended by a store and reconstruction mechanism for reducing communication. For a more detailed account on the graph partitioning approach, see [70]. Assessment. Albeit considerable effort has been spent in the design of parallel connection graph procedures, so far no experimental results have been published indicating a success. This seems to be due to the substantial overheads incurred for ensuring completeness of the deduction, while maintaining good potential for parallelism at the same time. 3.2.6. Clause Diffusion Clause Diffusion [71-73] provides a general methodology for the parallel execution of calculi, especially those which have contraction or simplification rules. Parallelism is explored at the search level. Basics. T h e execution of a calculus consists of two main parts: an inference mechanism and a search plan. A search plan is needed to navigate through the search space. For example, in a resolution based system the inference mechanism must be capable of generating resolvents while the search plan determines which resolvents are to be generated and used at each step. Many inference mechanisms also contain rules for "contraction", i.e., rules for the deletion of newly generated, but unnecessary, clauses. Examples for such contraction rules are subsumption and tautology elimination. Furthermore, simplification rules, like demodulation, can result in the deletion of clauses. These contraction and simplification rules cause changes in the d a t a base. Therefore, when a t t e m p t i n g parallelization of a calculus, these rules often create a bottleneck, due to the need for exclusive access to the d a t a base. Computational Model. T h e search space which is spanned by the formula and the inference mechanism is partitioned among processes by partitioning the clause set and the (expansion) inference steps onto the nodes of a distributed computing environment. Therefore, as no processor can exploit the whole search space, the processors need to communicate with each other. In this model, information is exchanged via message passing. For a resolution based theorem prover, the input clauses and the newly generated clauses are distributed among the processors. Each processor has its own local database to keep these clauses and executes resolution

236

steps locally. Newly generated clauses are sent together with additional information (e.g., a time-stamp) to all processors in order to ensure completeness of the scheme. In order to avoid redundancies, syntactic rules allow the deletion of arriving clauses in most cases. Implementation. A parallel theorem prover, Aquarius [73], which uses the Clause Diffusion scheme, has been implemented on a network of Sun workstations. O T T E R (cf. Section 3.2.3) has been used as the inference engine. However, no experimental results have been reported. Assessment. T h e Clause Diffusion scheme provides an entirely distributed scheme by distributing the inference rules and the d a t a base. In general, back-subsumption is rather costly, since the data-base must be updated. In this model, no global back-subsumption unit is needed, thus bottle-necks for back-subsumption are avoided. However, much communication is to be expected in such a system, since newly generated clauses are deleted on the arriving processor and not on t h a t processor which created these clauses. Clause Diffusion is a flexible scheme for parallelization, usable for many calculi and other algorithms, e.g., Knuth-Bendix completion. 3.2.7. The Wivenhoe Model T h e Wivenhoe Model [74, 75] is a computational model for proving formulas in first order logic (and the execution of pure Horn clause languages). T h e calculus used in this model is based on the rewriting formalism introduced in [76]. Computational Model. In the Wivenhoe model, the formula (and the calculus) is divided into two parts: Logical relations of the formula essentially represent its propositional structure. They are used to construct refutation plans, which comprise a set of pairs of literals to be unified in order to refute the formula. Structure relations encode the terms belonging to the literals of a refutation plan. T h e structure relations are used to check the validity of a refutation plan. Refutation plans are generated out of the formula by transforming the formula into a set of rewriting rules (see Table 10) and performing a continuous rewriting based on these rules, starting with a start symbol S. In order to obtain a parallel execution of the model, the handling of logical relations and structure relations are separated in a way similar to the DelPhi Model [77, 78]. Refutation plans are generated and simplified

237

logical rule Type ? — L\,..., Ln. query H : —Li,..., Ln. rule unit clause L. OR-branch between literal L and L i , . . ., L r , and connections t{

S N(H) N(L) N(L)

=> => =•

rewriting rule N(Li) * ... * N(Ln) N(L!)*...*N(Ln) c < 1* 7 V ( L 1) + + .. . + i r * W ( L r)

Table 10 The generation of rewriting rules.

by one or more generator processes. T h e refutation plans are checked for their validity by parallel unification, using a directed acyclic graph which is constructed out of the structure relations. T h e performance of this model can be increased substantially by introducing cooperation between the generators of the refutation plans and the processors which perform the unifications. Whenever a unification fails, a failure pattern is constructed which encodes the cause of the failure. This failure p a t t e r n is sent to one of the control units. T h e control unit instructs the refutation plan generators not to deliver any new refutation plans which would cause the same unification failure again. T h e architecture of the Wivenhoe model is shown in Figure 5. Although each of the components can consist of a number of identical processes working in parallel, the highest efficiency, however, can be obtained by doing only the unification in parallel. T h e architecture of the model is designed in such a way t h a t the parallel unification can be supported by a specialized parallel hardware, t h e IFS (Intelligent File Store). T h e I F S / 2 [79], based on the concept of active memory, allows for fast associative search and primitive operations over entire relations. All d a t a in the IFS are kept as tuples of a fixed size. Implementation. A prototypical simulation of this model is reported in [75], b u t no results are given. Measurements on parallel unification, presented in [80] and [74] indicate t h a t the improvement obtained for the parallel unification of the entire refutation plan, over the sequential execution of each individual unification, is rather limited for problems with a regular, recursive structure of terms (e.g., lists). T h e improvements obtained with simple terms or irregular recursive structures, or for very large problems, are significantly better. Assessment. T h e model is of particular interest, because it uses parallel unification. However, no implementation or simulation results for this

238

Fig. 5. Architecture of the Wivenhoe Model.

model are reported. This model is well suited to cooperating with the specific hardware (IFS) for term storage and associative retrieval. 4.

Competition-based Parallel T h e o r e m Provers

4-1. 4.1.1.

Uncooperative

Schemes

RCTHEO

R C T H E O [81, 25] constitutes a parallelization of the sequential theorem prover S E T H E O (see Section 3.1.2). Computational Model. R C T H E O stands for R a n d o m Competition T H E Orem prover, and thereby reflects the two basic ingredients the parallelization is based upon. Competition is used in the sense t h a t the same problem (i.e., the same set of clauses) is given to different prover processes. Each such prover process uses the same sequential theorem prover ( S E T H E O ) , but a different search strategy (single calculus competition). T h a t is, each prover orders the choices occurring at each OR-branch during the search according to a r a n d o m number sequence. Since each prover process uses a different sequence, the search spaces explored by the individual provers diverge soon due to the combinatorial explosion. T h u s R C T H E O exploits a non-decompositional form of OR-parallelism. As there is no exchange of information among the provers, the scheme is uncooperative. R C T H E O preserves the iterative deepening search strategy used by S E T H E O . Obviously, for cases were the bound values have not grown large enough yet, the R C T H E O scheme is fully redundant (i.e., all prover processes do the same work, but in different order). Therefore the random

239

competition is extended to r a n d o m reordering of the literals of a clause being used in an extension step (local AND-reordering). This means t h a t the order in which the subgoals of the current clause will be worked on is r a n d o m . Then, for the same bound values, the sizes of the finite individual search spaces usually vary. As soon as the first prover process detects the unavailability of a proof (given the current bounds), the current deepening level is known to be insufficient and all prover processes are restarted using larger bound values. 14

Implementation. R C T H E O I I [82] is based on S E T H E O V2.63 and implemented in C using a R P C programming library [83]. It runs on a network of 110 H P 9000/720 workstations connected by a LAN. Experimental Results. T h e principle performance of the R C T H E O model can be assessed quite easily in a sequential way, based on individual S E T H E O runs. This, however, does not take into account the overhead due to s t a r t u p , communication, and termination. T h e overhead depends on the number of processors, but is mainly independent of the problem size. For R C T H E O - I I , the overhead reported is in the range of 4 to 6 minutes (see [82]; variations are due to network traffic and processor utilization by other users). Since no detailed evaluation of R C T H E O - I I is available yet, below we present d a t a from [25] based on individual S E T H E O runs (excluding overhead). For Tables 11 and 12, the speed-up is calculated by S(k) = , where 15

E(T\) is the expected number of inferences for S E T H E O using a randomized search strategy, and E(Tk) is the expected number of inferences up to the first solution, running k S E T H E O s in parallel. For a motivation for this definition of speed-up see [25]. No specific evaluation of the combined effect of r a n d o m OR-selection and local AND-reordering is available so far.

Assessment. T h e advantages of this scheme are t h a t it is easy to simulate, easy to implement, and requires little communication. A theoretical speedup prediction (compared to sequential r a n d o m search) is possible if the run-time distribution of tasks is known. This, however, is usually not the 14

RCTHEO I, a predecessor of RCTHEO II, was implemented using UNIX Shell scripts based on rep and rsh, and did not allow local AND-reordering (and thus differed also in its iterative-deepening mode). It is fully subsumed (and outperformed) by RCTHEO II. 15 T h i s number is roughly proportional to the required run time.

240

bound value (-d 7) (-d 50) (-d 7) (-d 7) (-i 19)

Problem non-obvious 10-queens si lucasll ipl

p = 25

p = 50

ρ = 100

p = 200

40 13 13 13 25

58 25 25 40 55

72 50 63 108 112

87 98 240 141 300

p = 400

195 435

550

Table 11 Approximate overhead-free average speed-up performance of randomized ORbranch selection without iterative deepening.

^ ,, Problem si lucasll ipl

bound value (-d 6) (-i 20) (-ι 15)

ρ = 5

ρ = 10

ρ = 20

2.45 1.76 1.61

2.81 1.89 1.74

2.95

Table 12 Overhead-free speed-up performance of local randomized AND-reordering for the no-solutions case using appropriate fixed bounds.

case in practice. For suitable applications good scalability is achieved. However, since no work partitioning is performed, there is a high potential for redundant work. 4-2.

Cooperative

4.2.I.

Η

Schemes

PDS

H P D S (for Heterogeneous Parallel Deduction System, [20]) denotes a family 16 of parallel deduction s y s t e m s . All members are based on a combination of several sequential theorem provers into a single parallel system, by having each of them search for a proof independently (different calculi competition) while at the same time exchanging derived clauses and subsumption information between them (cooperation). Underlying Systems. T h e GLD-component (for Guided Linear Deduction) is a chain-format linear deduction system [84], based on Shostak's Graph Construction procedure [85]. This component performs a Model Elimination style backward-chaining search, using an extended unit preference 16

Parts of this description are based on personal correspondence with G. Sutcliffe.

241

strategy. Albeit t h a t this is not required for completeness, G L D also generates and stores all unit-lemmas and those non-unit lemmas which backsubsume at least one clause. T h e UR-component (for Unit Resulting resolution, see [86]) is complete for Horn problems only and performs a forward-chaining search. T h e HR-component implements positive ordered HyperResolution [87] with factoring. This component also performs a forward-chaining search. All components apply forward- and backward-subsumption on the clauses they derive (GLD does it only for lemmas), and store kept clauses in a common representation format. Also, each component uses iterativedeepening search. T h e type of bound and their increments differ for the individual systems, and each increment depends on the number of new clauses to be processed, using smaller increments as more clauses are available. Computational Model. T h e main idea of the H P D S scheme is to profit from the potential suitability of one of the components for the particular problem at hand (note t h a t it is not known in advance which one performs better), thus running t h e m in competition mode, and at the same time to allow cross-fertilization between the components by exchange of deduced clauses. T h e following different system configurations have been implemented and analyzed: GLD\\ UR-1. G L D | | U R - 1 consists of a combination of the G L D - and the UR-component. In G L D | | U R - 1 , subsumption tests on each derived clause are performed by the creating component. Every non-forward-subsumed new clause is sent to the other components along with associated backsubsumption information. However, since received clauses are not tested for subsumption, a component may end up with two copies of the same clause. GLD\\UR-2. G L D | | U R - 2 consists of a combination of the G L D - and the UR-component. In order to avoid duplicate clauses (and to increase the degree of parallelism), in G L D | | U R - 2 a separate control component is introduced. All derived clauses are passed to the control component untreated, where subsumption tests are performed and from where clauses and subsumption information are distributed to the deduction components (including the creator). T h e evaluation in [88] shows t h a t GLD||UR-2 outperforms G L D | | U R - 1 regarding execution time, but requires an additional processor to achieve this.

242

HPDS. H P D S consists of a combination of the GLD-, the UR-, and the HR-component. Clauses received by a component from another component are buffered (in a shared LINDA tuple space, see below) and added to the components' local set of clauses asynchronously. T h e GLD-component checks the buffer space for clauses after a new center clause is deduced; the UR-component checks it before a new nucleus is chosen; and the HRcomponent checks it before each hyperresolution step. It was found in [20] t h a t this frequent checking is useful for the HR-component, since a given nucleus can be used to derive many hyperresolvents, and checking only before a new nucleus is chosen may unduly delay termination after a proof has been found by another component. For a description of the system architecture and the type of information exchanged, see Figure 6.

kept GLD-lemmas + subsumption info

all deduced, kept clauses + subsumption info

Fig. 6. The flow of information in HPDS. Implementation. H P D S is implemented using Prolog-D-Linda [89, 90], an extension of SICStus Prolog which offers additional Linda-style functionality, and runs on a network of two or respectively three SUN SPARCstations (SUN-OS 4.0.3) connected via Ethernet. T h e GLD||UR-systems were implemented in a Prolog-Linda based on Arity Prolog, running on a network of 386 P C s under DOS. Experimental Results. In Table 13 we present experimental results derived from the d a t a in [20]. For each problem a time limit of 500 seconds was imposed. Originally, 132 test problems were used, for 49 of which none of the systems shown in the table found a proof. Table 13 is based on the remaining 83 problems. T h e average speed-up is determined using only those problems which both the respective system and H P D S could solve.

243

Metric Problems solved Average speed-up

GLD 87% 2.95

UR 67% 1.49

HR 58% 14.4

Naive 96% 1.93

HPDS 87%

Table 13 Performance table comparing HPDS with the individual components (GLD, UR, HR) as stand-alone systems, and against a naive parallel system running the three components concurrently (but on a single machine).

Table 13 reveals t h a t H P D S would perform worse t h a n a parallel version of the naive system using three processors. Within H P D S , the G L D component found the solution (and thereby caused termination of the search) in 56% of the problems, the UR-component in 39%, and the HRcomponent in 5%. This indicates t h a t the HR-component did not contribute sufficiently to be worth its costs. Further experiments have been performed with HPDS-II, which is basically identical to GLD||UR-1 (some improvements to the sequential deduction components have been made). Below we give some results from [91]. Metric Problems solved Average speed-up

GLD 77% 10.0

UR 56% 1.4

Naive 87% 16.9

HPDS-II 83%

Table 14 Similar to Table 13, but for HPDS-II and using a different set of problems. For HPDS-II, the situation is quite different to t h a t for H P D S . Now a parallelized naive system performs much worse t h a n HPDS-II, and the small improvement compared to the UR-component is offset by the small number of problems solved by UR. A comparison between H P D S and HPDS-II shows the sensibility of combining different systems as well as the availability of a significant speed-up potential. As a remark, it was found for all systems t h a t the combination of systems increased the range of problems for which UR-resolution found a proof (i.e., the degree of incompleteness of UR-resolution was reduced). Assessment. T h e H P D S group shows t h a t the idea of combining different calculi has significant potential, but also t h a t the performance of particular combinations is quite unpredictable. T h e design of such systems is very sensitive to minor modifications, and only little parallelism can be utilized. More research in this direction is required to obtain a better intuition for,

2 4 4

and understanding of, this approach. 4.2.2.

Team Work Completion

(TWC)

Team Work Completion [ 9 2 , 9 3 ] denotes a distributed theorem prover for equational logic based on unfailing Knuth-Bendix completion (the associated system has recently been named D I S C O U N T , for D i s t r i b u t e d C o m pletion UsiNg Teamwork). Underlying Calculus. Unfailing Knuth-Bendix completion [ 9 4 , 9 5 ] is a refinement of the Knuth-Bendix method [ 9 6 ] for obtaining complete sets of reductions. In the context of theorem proving, the goal is to prove the equality of two (skolemized) terms, given a set of equations E. Briefly, a reduction ordering is used to orient equations in Ε. Oriented equations are removed from Ε and become members of the set of rules R. Those which cannot be oriented remain in E. T h e rules are used to reduce each term (in Ε and R) until no more reductions are possible, which means for each term a normal form with respect to R has been obtained (E and R are said to be interreduced). T h e idea for this approach is t h a t most equalities can 17 be understood as reductions of complex terms to simpler o n e s . 18 Given R, E, a goal s — t, the set of critical p a i r s C P for R and E, and a reduction ordering. New rules are generated until either no more critical pairs remain (which proves t h a t 5 φ t) or the normal forms for s and / become equal (which completes the proof). Of course, due to the undecidability of the word problem in general, the completion may not terminate. T h e generation of rules works as follows. A critical pair is taken out of CP, both terms are reduced according to R, and compared according to the reduction ordering. If the pair is orientable, a new rule is formed, C P is updated by the critical pairs arising from the rule with R and E, and the rule is added to R. Then R and Ε are interreduced. Otherwise, a new equation is obtained, and C P is updated accordingly. Computational Model. Initially, a supervisor sends the given problem to a set of experts, each of which is a separate prover. T h e operation of T W C can be viewed as a loop of generating and merging information. In each generation phase, the different experts perform unfailing completion as outlined above (single calculus competition). Each expert uses a particular function for choosing critical pairs. Before a new critical pair is 17

T h i s concept is closely related to demodulation as in OTTER (see Section 3.2.3). T w o rules l\[t\ —• r\andl2 —• 7*2 give rise to a critical pair < 1\\χ2\σ,τ\σ > if there is a most general unifier σ such that ta — I2. A rule and an equation lead to the same critical pairs as does the rule with each of the two rules obtainable from the equation by its orientation. 18

245

chosen, an expert checks if it has received a request for a t e a m meeting from the supervisor. If so, the expert switches control to its associated referee, which reports to the supervisor a measure of the quality of the expert plus a selection of "good" rules and equations. Between generation phases t e a m meetings take place (cooperation). At a t e a m meeting, the current supervisor determines (based on the referee reports) the best expert and sends it the rules and equations it received from the referees. T h a t expert then becomes the new supervisor (thereby avoiding a transfer of the complete d a t a sets of the winner to the supervisor). Next, the new supervisor forms the new problem to be given to the experts by orienting the received rules and equations according to its reduction ordering, interreducing the rules and equations, and u p d a t i n g the set of critical pairs. It then chooses the experts and referees to be used, determines when the next t e a m meeting will take place, and sends the new problem to the experts. A sketch of the control flow in the model is given in Figure 7.

Supervisor

Experti

Referee ι

Expert;

Referee;

Expertn

Referee n

Supervisor

Cycle k Fig. 7. A computational cycle in TWC (adapted from Figure 1 in [92]). In addition to the experts described so far, T W C utilizes specialists which 19 report unnecessary critical pairs to the s u p e r v i s o r . These are subsequently removed at the next t e a m meeting. Implementation. T W C has been implemented in C and runs on a network of Sun-4 workstations under UNIX. 19

T h i s happens if the normal forms of the terms of a critical pair are identical with respect to the current rules or the same critical pair occurs repeatedly.

246

Experimental Results. A s u m m a r y of the tables presented in [92] is given in Table 15. More recently, further experiments have been performed on a cluster of Sun ELC machines (see [93]).

Prob.

eql eq2 eq3 eq4 eq5

E2

£i

29.9 115.1 -

94.8 8310.1

40.2 58.4 167.5 29.0

E3

E4

£ i + E2 team

E5

17.1 89.6

-

-

-

63.9

19.2

-

-

9.5 15.0 29.5 25.0 11212.4

best team

members Ει,

Ε2

Ei,E3 E2,

Ε4

Eï,

Es

E\,

Es,

Si

cycles 3 2 2 1 6

sec 9.5 10.6 15.8 9.0 702.1

Table 15 Run-time performance of TWC (times given in seconds). E \ through E& denote six different experts (i.e., strategies for choosing critical pairs), and S\ denotes a specialist for deleting critical pairs. A "?" denotes that no data were given for the respective case. Column 7 ( E \ + E2) shows the results for a standard team, and to the right of it there are the results for the best team. For the best team, the team members are shown on the left, the numbers of cycles in the middle, and the run-time on the right. Note that the number of team members equals the number of processors used.

Assessment. As it is typical for cooperative competition approaches, there is a significant speed-up potential due to competition and cross-fertilization effects, but only a limited (and fairly small) number of processors can be employed. 4.2.3. Multiple Sets-of-Support (MSOS) In this approach [97, 98] for resolution-based theorem proving, a competition between several different sets-of-support for the same problem is performed, avoiding redundant computations by exchange of intermediate results (cooperative single calculus competition). Sei-of-support Strategy. Set-of-support [57] is an important restriction strategy for resolution-style theorem provers. For its utilization, a subset of the clauses of a formula is defined as the set-of-support, and the application of an inference rule is allowed only if at least one clause from the set-of-support participates. Also, all newly derived clauses become members of the set-of-support. While a small set-of-support restricts the search sharply, a set-of-support being too small or unfortunate may cause deductive incompleteness. This is true in particular for i m p o r t a n t resolution refinements such as hyperresolution.

247

Computational Model. T h e idea is to have a number of different sets-ofsupport based on the experience t h a t usually there is a fairly small set-ofsupport which performs very well, but exactly which to choose is not known in advance. Since reasonable choices for sets-of-support are likely to overlap partially (i.e., they share clauses), a straightforward parallel computation will involve a high degree of redundancy. Therefore a generic scheme is proposed which, given a set of disjunct sets-of-support, allows to compute the closure of the set-of-support t h a t is obtained by the union of these setsof-support. In this computation, redundancy is reduced by reusing clauses which have been derived during the computation of the original, disjunct sets-of-support. T h e approach can be combined with different calculi. A completeness proof of the method, independent of the inference rule, is given in [97]. A proposal for an algorithm applying the scheme to binary resolution is found in [98]. Simulation. Using the a u t o m a t e d theorem prover O T T E R [52], experiments have been performed to assess the effects of variation in the set-ofsupport for hyperresolution. These experiments showed t h a t the potential of using different sets-of-support competitively is significant. Also, a simulation assessing the reduction of redundancy was done (for both, see [97]), which showed t h a t indeed a notable portion of redundancy can be avoided. Assessment. While the competition between various sets-of-support certainly can be beneficial, the number of experiments made so far regarding the reduction of redundancy is not yet sufficient for a definite assessment of the effectiveness of this m e t h o d . T h e effectiveness will crucially depend on the communication and control overhead incurred. T h e resulting trade-off between benefit and cost needs to be assessed. 5.

Summary

In this paper, we have given a comprehensive overview of the field of parallel a u t o m a t e d theorem proving. Each approach or system has been described in detail, including available measurements and references. We have focused entirely on theorem provers. For surveys on parallel PROLOG systems and other approaches, see the references given in the introduction. We have tried to make this survey as extensive as possible, but omissions, due to the rapid progress in this field, are possible. Figure 8 shows the historical development in parallel a u t o m a t e d theorem provers. For all the systems described in the text, we give the appropriate year, in which t h a t

248

20

system was d e v e l o p e d . This chart indicates t h a t since about 1988 many new systems have been designed and implemented. [Aquarius] PARROT-I/II PARTHENONPARTHEO [CGs]

I

I

[CGs]

I

I

I

I

I

[CGs]

1

RCTHEO

ROO

METEOR

G L D | | U R - l / 2 HPDS

[Wivenhoe] [DARES]

1

MGTP

1

TWC

1

[MSOS]

1

1981 1982 1983 1984 1985 1986 1987

Year 1988

1989

1990

1991

1992

Fig. 8. History table of approaches to parallel theorem proving, as treated in the text. In order to increase the insight into the approaches, and to emphasize the similarities and dissimilarities between the systems, an abstract classification scheme has been presented. In this scheme, the classification proceeds along the two axes of "Partitioning vs. Competition" and "Cooperative vs. Uncooperative systems". Furthermore, the class of "Partitioning" is subdivided into the classes of completeness-based partitioning and correctnessbased partitioning. T h e well-known type of OR-parallelism belongs to the first subclass, AND-parallelism to the second one. In the class of "Competition" we distinguish between systems where the competing agents operate on the same calculus and those t h a t are based on different calculi. This classification scheme sets up 8 different classes. In Figure 2 we have summarized where the described systems are located in the classification. Our classification scheme seems to be appropriate for the classification of parallel a u t o m a t e d theorem provers. First, it allows systems to be classified independently from the underlying calculus. This is contrast to, e.g., the classification into AND- and OR-parallelism, which is not applicable to set based (e.g., resolution based) theorem provers. Secondly, Figure 2 shows t h a t existing systems fill up almost all classes of the classification scheme, and systems which are intuitively similar to each other belong to the same class. In general, when such a survey is made, one tends to compare the systems with each other on the basis of execution times a n d / o r speed-up values. Such a comparison, however, does not make sense at all. Even the comparison of sequential theorem provers is, due to the different calculi, refinements, and implementation techniques, almost impossible. One can always find benchmark examples for which proofs can be found very 2 0

T h e dates given are the dates of the first major publication (to our knowledge) of that particular system. Therefore, these dates may not reflect precisely the dates of system development completion.

249

quickly with one system, but for which some other system cannot find a proof within reasonable times. T h e comparison of run-times or speed-up values for parallel systems is much more difficult and error prone, since the underlying hardware systems (esp. processors, communication mechanisms, and topology) often differ too much to allow for any kind of reliable statement. A further source of uncertainty comes in with the nondeterministic behavior of many parallel systems. E.g., in an OR-parallel system, extremely varying execution times are obtained if a formula contains more t h a n one proof (which holds for most cases). As shown in [25], in such cases, the definition of speed-up as ί&- is only of little significance. Rather, P

statistical methods need to be employed. Nevertheless, most systems exhibit good to very good processor utilization, compared to many other applications of parallel hardware. This indicates t h a t theorem proving is an ideal candidate for parallelization. Moreover, many of the parallel systems are based on a sophisticated sequential prover implementation, e.g., O T T E R / R O O , S E T H E O / P A R T H E O . This often allows the sophistication of a sequential system to be carried over to the parallel one. Parallel theorem provers have been developed to run on parallel machines with shared memory and with distributed memory, indicating t h a t the models for parallelizing theorem provers are applicable to both types of architectures. Here, it comes into play t h a t almost all approaches exploit a rather coarse-grain parallelism on the clause level. An exploitation of fine-grain parallelism — from AND-parallelism to connectionist theorem provers (cf. [99]) — does not seem to be favorable on existing hardware architectures with relatively few processors. Unfortunatedly, only for very few of the described systems a scalability analysis is available. Acknowledgements. We would like to t h a n k Owen Astrachan, Maria Paola Bonacina, Jörg Denzinger, Wolfgang Ertel, Masayuki Fujita, Ryuzo Hasegawa, Rusty Lusk, Ross Overbeek, Geoff Sutcliffe, and Jiwei Wang for their readiness to answer questions regarding their work and further comments. Moreover, special t h a n k s to Geoff Sutcliffe for his elaborate comments on a draft. Last not least, t h a n k s to Rüdiger Zölch for producing some nice figures.

250

Appendix

Problem apabhp bennetl2 Det-#3 Det-#6 Det-#58 Det-#77 eql - eq5

Ref. [100] [101] [49] [49] [49] [49] [92]

f3b3 ls36

[55] [102]

lsl08 lsl21

[102] [102]

lucasll

[103]

imp4

[103]

ipl

[103]

non-obvious sam's lemma si

[104] [55] [103]

wosl

[105]

wos4 woslO

[105] [105]

wosl 6 wos21 wos22 10-queens

[105] [105] [105] [29]

n-queens-all

[18]

Description Robotics: The "blind hand problem". Quasigroup theory: Idempotent quasigroups of order 12. A two-valued sentential calculus problem. A two-valued sentential calculus problem. A many-valued sentential calculus problem. An equivalential calculus problem. Equality problems. The reference shows the equational theory and the completed rule system for each. Semigroup theory: The size of finite semigroup f3b3. Group theory: The inverse of products equals the product of the inverse. Set theory: Associativity of set intersection Set theory: The difference of a first set and the set which is the difference of the first set and a second set, is the intersection of the two sets. Single axiom imp Ideational propositional calculus: Show that A —• A is a theorem. Single axiom implicational propositional calculus: IC-4 (a specific axiomatization) depends on the 1st Lukasiewicz axiom Single axiom implicational propositional calculus: Similar to s i , but with a significantly larger search space. A theorem on the totality of transitive relations. Lattice theory: A statement about modular lattices. Single axiom implicational propositional calculus: IC-1 (a specific axiomatization) depends on the 1st Lukasiewicz axiom Semigroup theory: In a semi-group with left identity and left inverse, there exists a right identity element. Group theory: Meaning unknown. Group theory: If the square of every element is the identity, then the group is commutative. Group theory: The identity element of a subgroup is unique. Ring theory: X * additiveJdentity = additiveJdentity 1 l Ring theory: Show that χ X y = x~ X y ~ . The queens problem with 10 queens, where the solution lies in the middle of the OR-search tree. The η-queens problem, where all solutions need to be found.

Table 16 Description (taken from the T P T P problem library [106]) and reference for all examples mentioned in the text.

251

References [1] F. Kurfess. Parallelism in Logic - Its Potential for Performance and Program Development. Vieweg Verlag, 1991. [2] C.-L. Chang and R.C.-T. Lee. Symbolic Logic and Mechanical Theorem Proving. Academic Press, 1973. [3] D.W. Loveland. Automated Theorem Proving: a Logical Basis. NorthHolland, 1978. [4] L. Wos, R. Overbeek, Ε. Lusk, and J. Boyle. Automated Reasoning - Introduction and Applications. McGraw-Hill, 1992. [5] J. Schumann. Efficient Theorem Provers based on an Abstract Machine. Dissertation, Institut für Informatik, Technische Universität München, Germany, 1991. [6] J. Schumann. Parallel Theorem Provers - An Overview. In Parallelization in Inference Systems, pages 26-50. Springer LNAI 590, 1992. [7] R.M. Butler and N.T. Karonis. Exploitation of Parallelism in Prototypical Deduction Problems. In Proceedings of the 9th International Conference on Automated Deduction (CADE), pages 333-343. Springer LNCS 310, 1988. [8] W.-T. Chen and M.-Y. Fang. An Efficient Procedure for Theorem Proving in Propositional Logic on Vector Computers. Parallel Computing, 17:983995, 1991. [9] K.A.M. Ali and R. Karlsson. The MUSE Or-parallel Prolog Model and its Performance. In Proceedings of the 1990 North American Conference on Logic Programming. MIT Press, 1990. [10] M.V. Hermenegildo and K.J. Greene. The &-Prolog System: Exploiting Independent And-Parallelism. New Generation Computing, 9:233-256, 1991. [11] E. Shapiro. The Family of Concurrent Logic Programming Languages. ACM Computing Surveys, 21(3):413-510, 1989. [12] W. Bibel, F Kurfess, Κ. Aspetsberger, P. Hintenaus, and J. Schumann. Parallel Inference Machines. In Future Parallel Computers, pages 185-226. Springer LNCS 272, 1986. [13] T. Yamaguchi, Y. Tezuka, and O. Kakusho. Parallel Processing of Resolution. In Proceedings of IJCAI-85, pages 1178-1180, 1985. [14] J. Harland and J. Jaffar. On Parallel Unification for Prolog. New Generation Computing, 5:259-279, 1987. [15] H.-W. Güsgen and S. Hölldobler. Connectionist Inference Systems. In Parallelization in Inference Systems, pages 82-120. Springer LNAI 590, 1992. [16] R.G. Smith. A Framework for Distributed Problem Solving. UMI Research Press, Ann Arbor, 1981. [17] V. Kumar, K. Ramesh, and V.N. Rao. Parallel Best-First Search of StateSpace Graphs: A Summary of Results. In Proceedings of Α Α ΑΙ '88, volume 1, pages 122-127, 1988. [18] M. Fujita, R. Hasegawa, M. Koshimura, and H. Fujita. Model Generation Theorem Provers on a Parallel Inference Machine. In Proceedings of the International Conference on Fifth Generation Computer Systems, pages

252

357-375, 1992. [19] C.B. Suttner. Competition versus Cooperation. Technical Report SFB Bericht 3 4 2 / 1 / 9 2 A (TUM-I9201), Institut für Informatik, Technische Universität München, 1991. [20] G. Sutcliffe. A Heterogeneous Parallel Deduction System. Technical Report ICOT TM-1184, Proceedings of FGCS'92 Workshop W 3 , 1992. [21] V.K. Janakiram, D.P. Agrawal, and R. Mehrotra. Randomized Parallel Algorithms for Prolog Programs and Backtracking Applications. In Int. Conf. on Parallel Processing, pages 278 - 281, 1987. [22] V.K. Janakiram, D.P. Agrawal, and R. Mehrotra. A Randomized Parallel Backtracking Algorithm. IEEE Transactions on Computers, 37(12): 16651676, 1988. [23] Z. Lin. Expected Performance of the Randomized Parallel Backtracking Method. In Logic Programming: Proc. of the North American Conference 1989, pages 677-696. MIT Press, Cambridge, MA, 1989. [24] R. Mehrotra and E.F. Gehringer. Superlinear Speedup through Randomized Algorithms. In International Conference on Parallel Processing, pages 291-300, 1985. [25] W. Ertel. Parallele Suche mit randomisiertem Wettbewerb in Inferenzsystemen, volume 25 of DISK! Infix-Verlag, 1993. [26] Β. Fronhöfer and F. Kurfess. Cooperative Competition: A Modest Proposal Concerning the Use of Multi-Processor Systems for Automated Reasoning. Technical report, Institut für Informatik, Technische Universität München, 1987. [27] S. Bose, E.M. Clarke, D.E. Long, and S. Michaylov. Parthenon: A Parallel Theorem Prover for Non-Horn Clauses. Journal of Automated Reasoning, 8:153-181, 1992. [28] D.H.D. Warren. Parallel Execution Models and Architectures for Prolog, presented at Working Group Architect. ESPRIT 415, 1988. [29] J. Schumann and R. Letz. PARTHEO: A High-Performance Parallel Theorem Prover. In Proceedings of the 10th International Conference on Automated Deduction (CADE), pages 40-56. Springer LNAI 449, 1990. [30] R. Letz, J. Schumann, S. Bayerl, and W. Bibel. SETHEO: A HighPerformance Theorem Prover. Journal of Automated Reasoning, 8(2):183212, 1992. [31] D.H.D. Warren. An Abstract PROLOG Instruction Set. Technical report, SRI, Menlo Park, Ca, USA, 1983. [32] Parallel C - User Guide. 3L Ltd., Livingston, Scotland, 1988. [33] Th. Bemmerl, A. Bode, T h . Ludwig, and S. Tritscher. MMK - Multiprocessor Multitasking Kernel (User's Guide and User's Reference Manual). SFB Bericht Nr. 3 4 2 / 2 6 / 9 0 A , Technische Universität München, 1990. [34] H.-J. Beier, T h . Bemmerl, Α. Bode, H. Ertl, Ο. Hansen, J. Haunerdinger, P. Hofstetter, J. Kremenek, R. Lindhof, Th. Ludwig, P. Luksch, and T h . Treml. T O P S Y S Tools for Parallel Systems. SFB Bericht Nr. 3 4 2 / 9 / 9 0 A , Technische Universität München, 1990.

253

[35] M.R. Jobmann. Leistungsanalyse von Rechen- und Kommunikationssystemen - Konzepte der Modellauswertung und Definition einer Modellierungssprache. Dissertation, Universität Hamburg, Germany, 1991. [36] M.R. Jobmann and J. Schumann. Modelling and Performance Analysis of a Parallel Theorem Prover. In A. Lehmann and F. Lehmann, editors, Messung, Modellierung und Bewertung von Rechensystemen, volume 286 of Informatik Fachberichte, pages 228-243, Neubiberg, September 1991. 6. GI/ITG-Fachtagung, Springer. [37] M.R. Jobmann and J. Schumann. Modelling and Performance Analysis of a Parallel Theorem Prover. In Proceedings of 1992 ACM SIGMETRICS and PERFORMANCE'92 International Conference on Measurement and Modeling of Computer Systems, volume 20(1) of Performance Evaluation Review, pages 259-260, Newport, Rhode Island, U.S.A., June 1992. ACM SIGMETRICS and IFIP W.G. 7.3, ACM. [38] J. Schumann and M. Jobmann. Scalability of a parallel Theorem Prover, a Modelling Approach. Technical report, Technische Universtät München, January 1993. [39] O.L. Astrachan. Investigations in Model Elimination based Theorem Proving. PhD thesis, Dept. of Computer Science, Duke University, USA, December 1992. [40] O.L. Astrachan and D.W. Loveland. METEORs: High Performance Theorem Provers using model elimination. In R.S. Boyer, editor, Automated Reasoning: Essays in Honor of Woody Bledsoe. Kluwer Academic Publishers, 1991. [41] Owen Astrachan. Meteor: Exploring model elimination theorem proving. Technical Report CS-1992-22, Duke University, Durham, NC, USA, 1992. [42] O. L. Astrachan and D. W. Loveland. METEORs: High performance theorem provers using model elimination. Technical Report CS-1991-08, Dept. of CS, Duke University, Durham, North Carolina, 1991. [43] W.R. Stevens. UNIX Network Programming. Prentice Hall, 1990. [44] R. Manthey and F. Bry. SATCHMO: a Theorem Prover implemented in Prolog. In Conference on Automated Deduction (CADE), 1988. [45] H. Fujita and R. Hasegawa. A Model Generation Theorem Prover in KL1 Using a Ramified-S tack Algorithm. Technical Report TR-606, ICOT, Tokyo, Japan, 1990. [46] R. Hasegawa, M. Koshimura, and H. Fujita. Lazy Model generation for Improving the Efficiency of Forward Reasoning Theorem Provers. In Proceed, ings of the International Workshop on Automated Reasoning (IWAR 92), pages 191-202, Tokyo, Japan, 1992. ICOT TR-751. [47] K. Nakajima, Y. Inamura, N. Ichiyoshi, K. Rokusawa, and T. Chikayama. Distributed Implementation of KL1 on the Multi-PSI/V2. In Proceedings of 6th ICLP, 1989. [48] H. Nakashima, K. Nakajima, S. Kondoh, Y. Takeda, and K. Masuda. Architecture and Implementation of PIM/m. In Proceedings of the International Conference on Fifth Generation Computer Systems. ICOT, 1992.

254

[49] W. McCune and L. Wos. Experiments in Automated Deduction with Condensed Detachment. In Proceedings of CADE-11, pages 209-223, Saratoga Springs, USA, 1992. Springer LNAI 607. [50] A. Jindal, R. Overbeek, and W . C . Kabat. Exploitation of Parallel Processing for Implementing High-Performance Deduction Systems. Journal of Automated Reasoning, 8:23-38, 1992. [51] A. Jindal, R. Overbeek, and W. McCune. A Parallel Processing Approach for Implementing High-Performance First-Order Logic Deduction Systems. Technical Report A N L / M C S - T M - 1 3 1 , Argonne National Lab., 1989. [52] W.W. McCune. Otter 2.0 Users Guide. Technical Report A N L - 9 0 / 9 , Argonne National Laboratory, 1990. [53] R. Butler and E. Lusk. Users Guide to the p4 Programming System. Technical Report A N L - 9 2 / 1 7 , Argonne National Laboratory, 1992. [54] E.L. Lusk, W. McCune, and J.K. Slaney. ROO - A Parallel Theorem Prover. Technical Report A N L / M C S - T M - 1 4 9 , Argonne National Laboratory, 1991. [55] E. Lusk and W. McCune. Experiments with ROO, a Parallel Automated Deduction System. In Parallelization in Inference Systems, pages 139-162. Springer LNAI 590, 1992. [56] E.L. Lusk, W . W . McCune, and J. Slaney. ROO: A Parallel Theorem Prover. In Proceedings of CADE-11, pages 731-734. Springer LNAI 607, 1992. [57] L. Wos, D. Carson, and G. Robinson. Efficiency and Completeness of the Set-of-support Strategy in Theorem Proving. Journal of the ACM, 12(2):536-541, 1965. [58] J.K. Slaney and E.L. Lusk. Parallelizing the Closure Computation in Automated Deduction. In Proceedings of the 10. International Conference on Automated Deduction, pages 28-39. Springer LNAI 449, 1990. [59] E.L. Lusk, W . W . McCune, and J.K. Slaney. High-Performance Parallel Theorem Proving for Shared-Memory Multiprocessors. In preparation, 1993. [60] D.J. Macintosh. Distributed Automated Reasoning: The Role of Knowledge in Distributed Problem Solving. PhD thesis, Clarkson University, Potsdam, NY 13699, 1989. [61] S.E. Conry, D.J. Macintosh, and R.A. Meyer. DARES: A Distributed Automated Reasoning System. In Proceedings of A A AI-90, pages 78-85, 1990. [62] D.J. Macintosh and S.E. Conry. SIMULACT: A Generic Tool for Simulating Distributed Systems. In Proceedings of the Eastern Simulation Conference, pages 18-23. The Society for Computer Simulation, 1987. [63] G. Hornung, A. Knapp, and U. Knapp. A Parallel Connection Graph Proof Procedure. In Proceedings of GWAI-81, pages 160-167. Springer LNCS, 1981. [64] R. Loganantharaj. Theoretical and Implementational Aspects of Parallel Link Resolution in Connection Graphs. PhD Thesis, Dept. of Computer Science, Colorado State University, 1985. [65] R. Loganantharaj. Some Theoretical and Implementational Aspects of dcdp-Parallelism in Connection Graph Refutation. Technical Report CS-

255

85-9, Dept. of Computer Science, Colorado State University, 1985. [66] R. Loganantharaj and R.A. Mueller. Parallel Theorem Proving with Connection graphs. In Proceedings of CADES, pages 337-352. Springer LNCS, 1986. [67] R. Loganantharaj. Parallel Link Resolution of Connection Graph Refutation and its Implementation. In Proceedings of the International Conference Parallel Processing, pages 154-157, 1987. [68] J.-Y. Juang, T.-L. Huang, and E. Freeman. Parallelism in ConnectionGraph-Based Logic Inference. In Proceedings of the International Conference on Parallel Processing, pages 1-8, 1988. [69] A.J. Bernstein. Analysis of Programs for Parallel Processing. IEEE Transactions on Computers, 15(5):757-762, 1966. [70] P.D. Cheng and J.Y. Juang. A Parallel Resolution Procedure Based on Connection Graph. In Proceedings of ΑΑΑΙ-8Ί, pages 13-17, 1987. [71] M.P. Bonacina and J. Hsiang. A System for Distributed SimplificationBased Theorem Proving. In B. Fronhofer and G. Wrightson, editors, Parallelization in Inference Systems, number 590 in LNAI, page 370. Springer, 1990. [72] M.P. Bonacina. The Clause-Diffusion Methodology for Distributed Deduction. Technical report, Dept. of Computer Science, State University of New York at Stony Brooks, NY, October 1992. [73] M.P. Bonacina. Distributed Automated Deduction. PhD thesis, Department of Comp. Science, State University of New York at Stony Brooks, NY, USA, December 1992. [74] J. Wang. A Computational Model for Logic Programming Languages and its Supporting Architecture. PhD thesis, University of Essex, England, November 1991. [75] J. Wang. Towards a new Computational Model for Logic Languages. Technical Report CSM-128, Dept. of CS, University of Essex, 1989. [76] C.L. Chang and J.R. Slagle. Using Rewriting Rules for Connection Graphs to Prove theorems. Artificial Intelligence, 12:159-180, 1979. [77] H. Alshawi and D.B. Moran. The Delphi Model and some Preliminary Experiments. In Proc. of 5th Int. Conf. and Symposium, pages 1578-1589. MIT-Press, 1988. [78] W.F. Clocksin. Principles of the DelPhi Parallel Inference Machine. Comp. Journal, 30(5):386-392, 1987. [79] S.H. Lavington and R.A.J. Davies. Active Memory for Managing Persistent Objects. In International Workshop on Computer Architectures to Support Security and Persistance, pages 137-154. Springer, May 1990. [80] J. Wang, A. Marsh, and S. Lavington. Non-WAM Models of Logic Programming and their Support by Novel Parallel Hardware. In Bertram Fronhofer and Graham Wrightson, editors, Parallelization in Inference Systems, volume 590 of LNAI, pages 253-269. Springer, 1992. [81] W. Ertel. OR-Parallel Theorem Proving with Random Competition. In Proceedings of LPAR'92, pages 226-237, St. Petersburg, Russia, 1992.

256

Springer LNAI 624. [82] J. Philipps. R C T H E O II, ein paralleler Theorembeweiser. Fortgeschrittenenpraktikum, Institut für Informatik, Technische Universität München, 1992. [83] J.R. Corbin. The Art of Distributed Applications. Springer-Verlag, 1991. [84] G. Sutcliffe. A Semantically Guided Linear Deduction System. Ph.D. Thesis, Dept. of Computer Science, University of Western Australia, 1992. [85] R.E. Shostak. Refutation Graphs. Artificial Intelligence, 7:51-64, 1976. [86] R. Overbeek, J. McCharen, and L. Wos. Complexity and Related Enhancements for Automated Theorem-Proving Programs. Comp. & Maths, with Appls., 2(1-A):1-16, 1976. [87] J.A. Robinson. Automatic Deduction with Hyperresolution. International Journal of Computer Mathematics, 1:227-234, 1965. [88] G. Sutcliffe. A Parallel Linear & UR-Derivation System. In Informal Proceedings of PPAI-91. Technical Report SFB 3 4 2 / 1 / 9 2 A (TUM-I9201), Technische Universität München, 1991. [89] G. Sutcliffe and J. Pinakis. Prolog-D-Linda: An Embedding of Linda in SICStus Prolog. Technical report, Dept. of Computer Science, University of Western Australia, 1991. [90] G. Sutcliffe. Prolog-D-Linda v2: A New Embedding of Linda in SICStus Prolog. Technical Report T R 9 3 / 6 , Dept. of Computer Science, James Cook University, Townsville, Australia, 1993. [91] G. Sutcliffe. HPDS - A Heterogeneous Parallel Deduction System. Slides for talk, 1992. [92] J. Denzinger. Distributed Knowledge-based Deduction Using the Team Work Method. SEKI Report SR-91-12, Universität Kaiserslautern, 1991. [93] J. Avenhaus and J. Denzinger. Distributed Equational Theorem Proving. In Proceedings of RTA-93, 1993. [94] J. Hsiang and M. Rusinowitch. On Word Problems in Equational Theories. In 14th International Colloquium of Automata, Languages and Programming, pages 5 4 - 7 1 . Springer LNCS 267, 1987. [95] L. Bachmair, N. Dershowitz, and D. Plaisted. Completion Without Failure. In Resolution of Equations in Algebraic Structures, Vol. 2, chapter 1, pages 1 - 30. Academic Press, 1989. [96] D.E. Knuth and P. Bendix. Simple Word Problems in Universal Algebras. In Computational Problems in Abstract Algebra, pages 263-297. Pergamon Press, 1970. [97] C.B. Suttner. Parallel Computation of Multiple Sets-of-Support. In Logic Programming and Automated Reasoning (LPAR'92), pages 238-249. Springer LNAI 624, 1992. [98] M. Grosse and C.B. Suttner. A Parallel Algorithm for Set-of-Support. Technical Report SFB Bericht 3 4 2 / 1 / 9 2 Β, Institut für Informatik, Technische Universität München, 1992. [99] S. Hölldobler. A Connectionist Unification Algorithm. Technical report, International Computer Science Institute, Berkeley, Ca., 1990.

257

[100] G.Α. Wilson and J. Minker. Resolution, Refinements, and Search Strategies: A Comparative Study. IEEE Transactions on Computers, C-25(8), 1976. [101] M. Fujita, J. K. Slaney, and R. Hasegawa. New Results in Mathematics by a Parallel Theorem Prover on the Parallel Inference Machine. June 1992. [102] J.D. Lawrence and J.D. Starkey. Experimental Results of Resolution based Theorem-proving Strategies. Technical report, Computer Science Department, Washington State University, Pullman, 1974. [103] F. Pfenning. Single Axioms in the Implicational Propositional Calculus. In Proceedings of the 9th Int. Conf. on Automated Deduction, pages 710-713. Springer LNCS 310, 1988. [104] J. Pelletier and P. Rudnicki. Non-obviousness. AAR Newsletter 6, pages 4-5, 1986. [105] L. Wos. Unpublished Notes. Argonne National Laboratory, 1965. [106] C.B. Suttner, G. Sutcliffe, and T. Yemenis. The T P T P Problem Library. Technical report, Institut für Informatik, Technische Universität München [Also from Department of Computer Science, James Cook University, Townsville, Australia], 1993.

Parallel Processing for Artificial Intelligence 1 L.N. Kanal, V. Kumar, H. Kitano and C.B. Suttner (Editors) © 1994 Elsevier Science B.V. All rights reserved.

259

Chapter 10 Massive Parallelism in Inference Systems Franz Kurfeß Department of Neural Information Processing, Faculty of Computer University of Ulm, Ulm, Germany

Science,

This contribution concentrates on two aspects of massive parallelism and inference: first, the potential of parallelism in logic is investigated, and then an massively parallel inference system based on connectionist techniques is 1 presented. 1.

P a r a l l e l i s m in L o g i c

T h e exploitation of parallelism in inference systems mostly concentrates on AND/OR-parallelism with Prolog as input language, resolution as underlying calculus and either implicit parallelism or a small set of built-in constructs for explicit parallelism. Whereas the potential degree of parallelism in these approaches seems quite high, its practical exploitation is rather limited, mainly due to the complicated management of the execution environment. T h e potential of parallelism in logic is discussed here with respect to different parts of a logical formula, namely the whole formula, its clauses, the literals of the clauses, the terms, and the symbols (Kurfeß, 1991a). 1.1.

Formula

Level

On the formula level, parallelism occurs mainly in two varieties: one is to work on different relatively large and independent pieces of the formula simultaneously, the pieces being either separate formulae, or something corresponding to modules or objects. T h e other variety is to apply different evaluation mechanisms to one and the same formula; these evaluation mechanisms can be different calculi suitable for different classes of formulae, or different degrees of accuracy ranging from a restriction to the propositional 1

joint work with Steffen Hölldobler, TH Darmstadt

260

structure of the formula (thus neglecting the information contained in the terms), over incomplete unification (weak unification, unification without occur check) to a full treatment of the predicate logic formula. This variety also comprises some reduction techniques, aiming at a smaller search space while maintaining the essential properties of the formula. Another instance of parallelism on the formula level are spanning matings (Bibel, 1987a; Bibel, 1980; Bibel, 1983). A spanning m a t i n g is a set of connections of the formula which - if it is unifiable - represents a solution; thus spanning matings are related to OR-parallelism, but take into account the overall structure of the formula, which to some degree can already be done at compile time. 1.2.

Clause

Level

OR-parallelism (Conery, 1983; Kalé, 1989; Ratcliffe and Robert, 1986; Haridi and Brand, 1988; Szeredi, 1989; Lusk et al., 1988; Warren, 1987; Warren, 1987) occurs on the clause level: alternative definitions of clauses are evaluated at the same time. In the reduction-based evaluation mechanisms, OR-parallelism is exploited at runtime. This creates a considerable overhead for the management of the variable bindings associated with the alternative solution a t t e m p t s . In addition, other reduction techniques may be applied on the clause level. 1.3.

Literal

Level

The literal level contains AND-parallelism and routes. AND-parallelism refers to the simultaneuos evaluation of multiple subgoals in a clause. T h e problem here is t h a t there might be variables appearing in more t h a n one subgoals. All instances of one variable in a clause, however, must have the same value. Thus, if subgoals sharing variables are evaluated in parallel it must be guaranteed that the instantiations of these variables are consistent. On one hand this causes some complications during the execution, since either communication has to take place between processing elements working on subgoals with shared variables, or the bindings of different instances of a shared variable must be checked for consistency after the evaluation of a subgoal. On the other hand, shared variables can be used nicely for synchronization purposes, which is used in a family of ANDparallel logic programming languages with P A R L O G (Clark, 1988; Clark and Gregory, 1983; Clark and Gregory, 1986; Foster and Taylor, 1987; Clark and Gregory, 1984; Gregory, 1987; Gregory et al., 1989; Westphal, 1986), Concurrent P R O L O G (Hellerstein and Shapiro, 1986; Kliger et al., 1988; Shapiro, 1986; Shapiro, 1988; Shapiro and Takeuchi, 1983) nd G H C

261

and its derivatives (Ueda, 1985; Ueda, 1986; Ohki et al., 1987; for New Generation Computer Technology ( I C O T ) , 1992) as main exponents. Routes are based on a statical grouping of connected literals from different clauses (Ibanez, 1988; Ibanez, 1989; Kurfeß, 1991a). In contrast to AND-parallelism, where subprocesses are spawned off dynamically as soon as multiple subgoals are encountered in a clause, routes are identified as far as possible during compile time, and only expanded at run-time if necessary. 1.4-

Term

Level

Of high interest with respect to massive parallelism is the term level; this kind of parallelism is often referred to as unification or term parallelism. Again we can differentiate between two varieties: to treat different terms or pieces of terms simultaneously, or to apply different operations on the same terms. In the first case, pairs of terms can be treated as they emerge during the proof process, or large term structures can be separated into relatively independent substructures (this includes streams, for example). In the second case, unification can be split up into separate phases (singularity, decomposition, occur check). Due to the stepwise evaluation scheme for resolution, the size of terms is rather small, and term parallelism does not look very promising. This is different for approaches like the connection method or similar ones (Wang, 1989; Wang et al., 1990), where the terms of substantial parts of the formula are combined into large term structures, which then can be treated in parallel. 1.5.

Symbol

Level

The application of operations to the components of d a t a structures is the underlying concept for data parallelism (Hillis and Steele, 1986; Waltz and Stanfill, 1988; Fagin, 1991): one and the same operation is executed in a SIMD way on each component of a d a t a structure. This only is useful if there are regular d a t a structures composed of a large number of elements of the same (or a closely related) type. T h e usual representation of logic programming languages like P R O L O G does not lead to a natural exploitation of d a t a parallelism; there are approaches, however, to integrate data-parallel concepts into (Fagin, 1990; Fagin, 1991). 1.6.

Subsymbolic

Level

Applying an alternative paradigm to the representation of symbols, namely distributed representation (Hinton, 1984), leads to another level even below

262

the symbol level. Here the representation of a symbol cannot be localized to one particular place in memory (e.g. one memory cell), but is distributed over an ensemble of cells; one cell, on the other hand, is involved in the representation of more than one symbol. A few inference mechanisms based on connectionist systems have been proposed; some of t h e m aim at a implementation of a logic programming language (or, equivalently, expert system rules) with connectionist methods (Holldobler and Kurfeß, 1991; Pinkas, 1990; Pinkas, 1991c; Pinkas, 1991b; Pinkas, 1991a; Pinkas, 1992; Kuncicky et al., 1992; Lacher et al., 1991a; Kuncicky et al., 1991; Lacher et al., 1991b) whereas others are directed towards inference models not necessarily founded on a formal logic system (Shastri and Ajjanagadde, 1992; Shastri, 1988; Ajjanagadde and Shastri, 1991; Giles et al., 1991) (Barnden, 1989; Lange and W h a r t o n , 1992; Barnden, 1988a; Barnden and Srinivas, 1990) (Lee et al., 1986) (Sun, 1992; Sun, 1991). 2.

Massive Parallelism

A lot of attention has been paid recently to the potential and the use of "massively parallel systems". This term is used in computer architecture for describing computer systems consisting of a large number of relatively simple processing elements, with the Connection Machine, specifically the CM-2 series, as prototypical instance (Hillis, 1985). In a more general context, the term is also used as a theory for describing the behavior of physical systems composed of a large number of relatively simple interacting units, often in connection with neural networks to describe their mode of operation. In this section we will try to point out i m p o r t a n t aspects of massively parallel systems, the emphasis being on its use in the computer architecture field. 2.1.

Characteristics

for Massively

Parallel

Computer

Systems

There seems to be a general understanding about the meaning of "massive parallelism", or "massively parallel systems" in the computing community, although, to our knowledge, a concise definition has not yet been proposed (Hennessy and Patterson, 1990) (Almasi and Gottlieb, 1989) (Bell, 1989) (Douglas and Miranker, 1990; MacLennan, 1989a; Hillis, 1985; Hillis and Steele, 1986; Steele and Hillis, 1986) (Nilsson and Tanaka, 1988; Potter, 1992; Stolfo, 1987; Waltz, 1990; Kalb and Moxley, 1992; Wattenberg, 1992; Aarts and Korst, 1989; Barnden, 1988a; Barnden and Srinivas, 1990; Blevins et al., 1991; Douglas and Miranker, 1990; Frenkel, 1986; Hillyer and Shaw, 1984; Jackoway, 1984; MacLennan, 1989b; Potter, 1985; Uhr, 1985) (Journal of Parallel and Distributed Computing, 1992) (Wavetracer, Inc.,

263

1992) (Stanfill and Waltz, 1988; Denning, 1986; Hillis, 1985; Thinking Machines, 1987; Tucker and Robertson, 1988; Waltz and Stanfill, 1988). At its core lies a computer system which consists of thousands of relatively simple processing elements, with a relatively dense interconnection network, operating in synchrony (SIMD m o d e ) . T h e term "massive parallelism" often is used as a synonym for "data parallelism" as introduced in (Hillis and Steele, 1986; Rose and Steele, 1987). In the following we try to elucidate the characteristics of a massively parallel system with respect to the aspects of Computation, Communication^ Synchronization, and the underlying Memory Model. It is described mainly from a user's point of view, ideally providing a system with an unlimited number of processing elements, each connected with all others. In addition, the user should not have to worry about synchronization or access to memory. Obviously this is an idealized picture, and is in contrast to feasibility, performance, efficiency, and possibly other aspects. 2.2.

Computation

T h e computation aspects of massive parallel system concentrate mainly on: - a large number of processing elements, - virtually infinite for the user, and - relatively simple processing elements. W i t h current technology, tens of thousands of processing elements are feasible, each processing element operating on a single bit. T h e illusion of an infinite number of processing elements can be approached by providing "virtual" processors, implemented through time-sharing of the real units. It is unclear if the simplicity of the processing elements should be considered a fundamental requirement, or if it is a question of trade-off between the number of elements and their complexity. 2.3.

Communication

From a user's point of view, desirable communication properties of a massively parallel system are: - a dense interconnection scheme, - virtually fully connected to the user, with - uniform communication delays. W i t h current computers implemented as electronic devices, the only realistic interconnection method for massively parallel systems seems to be a

264

regular point-to-point serial connection to tens of neighbors. A fully interconnected network is not feasible with this technology, b u t the available density seems to be sufficient for large classes of problems. Automatic routing methods are usually available on the operating system level so t h a t the user may assume full interconnectivity. Communication delays are normally not uniform, but do not seem to impose critical constraints for most applications. Optical technology might offer a substantial increase in interconnection density, but does not seem to be ripe yet for wide application. 2.4-

Synchronization

T h e coordination of activities within a massively parallel system should be supported by - flexible synchronization mechanisms, - transparent to the user, and - problem-specific synchronization methods should be easily available. T h e currently prevalent synchronization method for massively parallel systems is a strict synchronous operation of the processing elements, i.e. in one cycle each processing element executes the same operation, or does nothing. Such a tight synchronization scheme does not seem to be a desirable property, but rather a consequence of available techniques and technologies. This scheme also implies the presence of a single controller, which may lead to problems with future, larger systems. Completely asynchronous operation as in MIMD systems, on the other hand, shifts the burden to the compiler or even the user, which for massive parallelism does not seem to be a viable alternative. T h e usage of the term massive parallelism in the other domain, the description of physical systems, often is connected with a certain amount of self-organization, where the operation of the system is loosely synchronized through the activities of the units itself, without the need for a central controller. Whereas such a behavior certainly would be desirable for a computer as well, it is not clear at present how this could be achieved. One approach is to view a processing element as a finite automaton, which performs an operation according to its internal status and the input it receives. 2.5.

Memory

Model

T h e memory available in a massively parallel system as seen by the user should provide

265

- a single flat memory space, - no memory hierarchy visible to the user (cache, local memory, global memory), and - uniform memory access times. These requirements clearly reflect the currently almost exclusively used way of accessing d a t a through addresses and pointers. Accessing d a t a according to what they represent, i.e. in content-addressable or associative memories does have a number of advantages, especially with operations based on similarities between items in large bodies of information. In particular in connection with neural networks, associative memory access, distributed representation, and sparse encodings are often associated with massive parallelism. At the moment it is not clear if these techniques can be integrated smoothly with conventional memory access, or if they require dedicated architectures. 3.

T h e P o t e n t i a l of M a s s i v e P a r a l l e l i s m for L o g i c

The goal of this section is to check out the potential of massive parallelism for logic. T h e overview table in Figure 1 shows for each level of a logical formula the requirements on computation, communication, synchronization, and memory. The requirements and their corresponding entries in the table are roughly characterized as follows: Computation is low if there are a few operations to be performed in parallel; it is medium for hundreds of simultaneous operations, and high for thousands. Communication is low if at a given time, a few messages are exchanged in the system simultaneously; medium, if it is hundreds, and high for thousands. Synchronization is low if a few synchronizing actions are effective at the same moment, medium for hundreds, and high for thousands. For the memory category several characterizations can be chosen: the number of simultaneously active d a t a elements (i.e. working m e m o r y ) , the number of memory accesses at one point, or the overall number of d a t a elements in main memory at a given m o m e n t . There is an obvious tradeoff in this characterization: similar results can be achieved by a few, very complex operations, or many very simple ones; similar tradeoffes hold for communication, synchronization, and memory requirements. At this point we do not want to detail these tradeoffs further, but just assume some intermediate complexity, e.g. as given by the operations and d a t a structures of a microprocessor or an abstract machine as the WAM (Warren, 1983). It must be noted t h a t the approach here is not a quantitative one: the entries of the table are only estimates, and not based on a detailed analy-

266

sis. Such a detailed analysis would be quite difficult, since the amount of parallelism can depend on the programming style, calculus, internal representation, and many other factors. β

d

•J 1

J S

3

§

b

a

a

I

a

ο

ο

s

Formula /ο κ; modules, objects, calculus

a

S

/oit;

/ou>

low

Clause OR-par., spanning sets

medium

low

low

Literal AND-par., routes

medium

medium

medium

medium

Term unification par., streams

high

high

high

high

Atom data parallelism

high

medium

low

medium

Sub-symbolic connectionist

high

high

low

medium

Fig. 1. The potential of massive parallelism for logic On the formula level, the requirements in all categories are rather low. This is due to the assumption t h a t in most applications only a small number of independent formulae (or tasks, objects, modules, etc.) will be treated simultaneously. It is possible to imagine applications where the number of formulae is much higher, however; this is largely dependent on the organizational style of a large knowledge base, for example. On the clause level, the amount of computation can be considerabley higher than on the formula level, because many clauses may exist expressing alternative rules. The communication requirements on this level still are rather low: alternative clauses represent independent subtrees to be traversed. There is a need for some synchronization, however, which is mainly caused by the necessity to perform backtracking if a branch leads

267

to a dead end. Backtracking can be avoided if all possible alternatives are investigated simultaneously. Although there is some overhead in memory consumption for the storage of binding information in the case t h a t backtracking is necessary, it is not very high. T h e a m o u n t of computation to be performed on the literal level certainly is higher t h a n at the clause level, but for most applications still ranges in the hundreds. It requires a fair a m o u n t of communication since the bindings of identical variables from different literals of the same clause must be the same. T h e same concept of shared variables also causes the necessity of synchronization. There is no exceptional requirement in terms of memory consumption here. The potential for massive parallelism really becomes evident on the term level. Depending on the size and structure of the program of course, thousands of operations can possibly be applied to components of terms, although in the extreme case unification - which is the essential operation on the term level - can be sequential (Yasuura, 1983; Delcher and Kasif, 1988; Delcher and Kasif, 1989). On this level, the computation of variable bindings and checking of term structures necessitate a high a m o u n t of communication and synchronization. Due to the possibly large a m o u n t of d a t a involved, memory requirements can be also quite high. On the symbol level the potential lies in the exploitation of d a t a parallelism (Hillis and Steele, 1986; Succi and Marino, 1991; Fagin, 1990; Fagin, 1991). T h e basic idea is to apply one and the same operation to all components of a d a t a structure like an array. T h e amount of parallelism thus depends on the size of these d a t a structures. Communication is necessary according to the dependencies of the program, whereas synchronization is easy according to the SIMD operation of data-parallel machines. Memory access typically is done locally for each processing element. The potential for parallelism is also quite high on the sub-symbolic level, which is applied in some connectionist models. T h e amount of computation is quite high due to the large amount of processing elements, although the typical operations of these elements are very simple. Communication is also very high, again with the simplicity of the d a t a exchanged. Synchronization basically is not necessary, the design of these systems is based on selforganization. The memory requirements can be about a third higher than conventional systems, which is due to the necessity of sparse encoding to avoid crosstalk between stored concepts (Palm, 1980).

268

4.

CHCL: A Connectionist Inference S y s t e m

The second part describes the design and implementation of C H C L (Hölldobler, 1990c; Hölldobler, 1990a), a connectionist inference system. The operational behavior of C H C L is defined by the connection method (Bibel, 1987b), a calculus which has some advantages over resolution, for example, with respect to parallelism. T h e basic idea here is to identify spanning matings, which represent candidates for alternative solutions to the formula; in order to represent an actual solution, a spanning m a t i n g also has to be unifiable. This evaluation mechanism has been used to derive C H C L as functionally decomposable system, in contrast to symbolic AI systems where symbols as basic tokens are combined via concatenation. The architecture of C H C L is based on networks of simple neuron-like units, interconnected by weighted links. T h e overall system consists of several sub-networks performing the determination of spanning matings, reductions of the search space, and unification. Through its composition from a large number of simple units with weighted interconnections, C H C L exhibits a large degree of parallelism restricted mainly by the inherent properties of the formula under investigation. At the core of the inference system is the unification mechanism (Hölldobler, 1990b; Kohonen et al., 1991), which represents the terms to be unified as a boolean m a t r i x of symbols and their occurrences; initially all those entries of the m a t r i x are true where a symbol occurs at a certain position in one of the terms to be unified. In the course of the computation, more elements become true according to two properties of unification: Singularity, which guarantees t h a t symbols with the same identifier have the same value, and Decomposition, which propagates the unification task to subterms through functions and their arguments. In the example < f(x,x,x) = / ( # ( a ) , y, g(z))y singularity guarantees t h a t x , y , ^ ( a ) and g(z) all have the same value, whereas decomposition propagates the unification task to the subterms g(a) and g(z), thus achieving identical values for a and z. Both singularity and decomposition are implemented through links and auxiliary units between units of the unification network. Singularity links connect different symbols (and subterms) occurring at the same position in different terms, and instances of a symbol occuring at different positions. Decomposition links are only required if subterms must be unified because of a shared variable in their ancestors, and they go from the variables at a "parent" position to the symbols of the respective "child" position(s). T h e operational principle of C H C L is quite simple: it relies on simple boolean units with a threshold, weighted links and the propagation of activation through these links; there is no learning involved. It is also guaranteed t h a t in the worst case the time complexity of the algorithm is linear with the size of the unification

269

problem; the special cases of matching (where only one of the two terms may be changed) and the word problem (are the two terms identical?) are solved in two steps (Hölldobler, 1990a). T h e spatial complexity is not as favorable: the network requires a quadratic number of auxiliary units and a cubic number of links. ?-

object(obj(X,

square,

dark,

circle,

Zl,

large)),

object(obj(a,

circle,

dark,

large)).

object(obj(b,

square,

dark,

medium)).

above(Trajector, RegionAbove),

12))

in(obj(a, striped,

Landmark)

in(Trajector,

rAbove(obj(b, striped,

object(obj(Υ,

above(X,Y).

square,

dark,

:-

rAbove(Landmark,

RegionAbove). medium),

obj(rl,

region,

.

circle,

dark,

large),

obj(rl,

region,

Z3)).

Fig. 2. Dark circle above a dark square and in the striped and dotted regions

4.1.

An Example:

Spatial Relations

Between

Objects

Let's consider the basic operation of C H C L with an example inspired by the Lo project, which aims at building a language learning system for the domain of simple geometric scenes (Feldman et al., 1990). The program in Figure 2 is a description of such a scene; the corresponding matrix, which is a blueprint for a C H C L network, is shown in Figure 3. On the left side are the spanning sets for the program, which are subsets of the set of logical connections. A spanning set represents a solution if it is unifiable;

270

Fig. 3. The representation of the program.

271

for our example program, only t h e second spanning set is unifiable, and only t h e corresponding connections have to be taken into account. These connections direct t h e unification process: the first connection (from the top) connects t h e first literal of t h e first clause with t h e first literal of the second clause, and indicates t h a t the terms of these two literals must be unified. This means t h a t function symbols and constant occuring at the respective positions in t h e two terms must be identical (e.g. o b j , c i r c l e ) , and variables must assume identical values. T h e latter is the case for the variable X, which occurs at the same position ( / l . l ) as the constant a and hence must assume the same value. In the network this is expressed through the activation of the corresponding units, which in t h e figure is shown by an open square; initially set units are shown as filled squares. For the first connection, these are the variable X and the constant a at position / l . l , as well as Zl and d a r k at position / 1 . 3 . 4-2.

Implementation

of

CUCL

C H C L is implemented (Kurfeß, 1991b) using ICSIM (Schmidt, 1990), a connectionist simulator developed at ICSI. This simulator provides a collection of basic modules for the construction of connectionist networks while maintaining a high degree of flexibility through the use of S A T H E R , an object-oriented language and development environment. S A T H E R is a simplified and optimized derivative of E I F F E L , with an emphasis for the practical needs of writing efficient, reusable code. In addition, an implement a t i o n of C H C L on a W A V E T R A C E R

(Wavetracer, Inc., 1991; Wavetracer,

Inc., 1992) is currently under way. References Aarts, Ε. H. and Korst, J. H. (1988/1989). Computations in massively parallel networks based on the Boltzmann machine: A review. Parallel Computing, 9:129-145. Ajjanagadde, V. and Shastri, L. (1991). Computation, 3:121-134.

Rules and variables in neural nets.

Almasi, G. S. and Gottlieb, A. (1989). Highly mings, Redwood City, CA.

Parallel

Computing.

Neural

Benjamin / Cum-

Ballard, D. (1986). Parallel Logical Inference and Energy Minimization. Technical Report TR 142, Computer Science Department, University of Rochester. Barnden, J. (1988a). Simulations of Conposit, a Supra-Connectionist Architecture for Commonsense Reasoning. In 2nd Symposium on the Frontiers of Massively Parallel Computation, Fairfax, VA., Las Cruces. Barnden, J. (August 1988b). The Right of Free Assosiacion: Relative-Position Encoding for Connectionist Data Structures. In Procs. of the Tenth Annual Conference of the Cognitive Science Society.

272

Barnden, J. A. (1989). Neural-net implementation of complex symbol-processing in a mental model approach to syllogistic reasoning, pages 568-573. Barnden, J. A. and Srinivas, K. (1990). Overcoming rule-based rigidity and connectionist limitations through massively-parallel case-based reasoning. Memoranda in Computer and Cognitive Science MCCS-90-187, Computing Research Laboratory, New Mexico State University, Las Cruces, NM. Bell, G. C. (1989). The future of high performance computers in science and engineering. Communications of the ACM, 32(9):1091-1101. Bibel, W. (1980). On Matrices with Connections. Journal Bibel, W. (1983). Matings in Matrices. Communications 852. Bibel, W. (1987a). Automated Theorem Proving. schweig, Wiesbaden, 2nd edition. Bibel, W. (1987b). Automated second edition.

Theorem

of the ACM, 28:633-645. of the ACM, 26, Nr. 11:844-

Artificial Intelligence. Vieweg, Braun-

Proving.

Vieweg, Braunschweig, Wiesbaden,

Blevins, D. W., Davis, E. W., and Reif, J. H. (1991). Processing element and custom chip architecture for the blitzen massively parallel processor. Technical report, MCNC / MCSU / Duke University. Clark, K. and Gregory, S. (1983). PARLOG: parallel programming in logic. Technical report, DOC Reports. Clark, K. and Gregory, S. (1984). Notes on Systems Programming in Parlog. Of the International

Conference

On Fifih

Generation

Computer

Proc. pages

Systems,

299-306. Clark, K. and Gregory, S. (1986). PARLOG: Parallel Programming in Logic. A CM Transactions

on Programming

Languages

Clark, K. L. (1988). Parlog and Its Applications. Engineering, 14:1792-1804. Conery, J. S. (1983).

The AND/OR

Process

1986(8):l-49.

and Systems,

Model

IEEE

Transactions

for Parallel

Programs. PhD thesis, University of California, Irvine. Department of Information and Computer Science.

on

Execution

Software of

Logic

Technical report 204,

Delcher, A. and Kasif, S. (1989). Some results in the complexity of exploiting data dependency in parallel logic programs. Logic Programming, 6:229-241. Delcher, A. L. and Kasif, S. (1988). Efficient Parallel Term Matching. Technical report, Computer Science Department, Johns Hopkins University, Baltimore, MD 21218. Denning, P. (1986). Parallel Computing and Its Evolution. ACM, 29:1163-1169.

Communications

of the

Douglas, C. C. and Miranker, W. L. (1990). Beyond massive parallelism: numerical computation using associative tables. Parallel Computing, 16(1):1—25. Fagin, B. (1990). Data-parallel logic programming. In North Logic

American

Conference

on

Programming.

Fagin, B. (1991). Data-parallel logic programming systems. Technical report, Thayer School of Engineering, Dartmouth College, Hanover, NH 03755.

273

Feldman, J. Α., Lakoff, G., Stolcke, Α., and Weber, S. H. (1990). Miniature language acquisition: A touchstone for cognitive science. Technical Report TR-90-09, International Computer Science Institute, Berkeley, CA 94704-1105. for New Generation Computer Technology (ICOT), I. (1992). Icot free software catalogue. Institute for New Generation Computer Technology (ICOT). Foster, I. and Taylor, S. (1987). Flat Parlog: A Basis for Comparison. Parallel gramming, 16:87-125. Frenkel, Κ. Α. (1986). Evaluating Two Massively Parallel Machines. of the ACM, 29:752-758.

Pro-

Communications

Giles, C. L., Chen, D., Miller, C , Chen, H., Sun, G. Z., and Lee, Y. (1991). Secondorder recurrent neural networks for grammatical inference. In International Joint Conference

on Neural

Networks

(IJCNN-91

) , volume 2, page 273, Seattle, WA.

Goto, Α., Tanaka, H., and Moto-Oka, T. (1984). Highly Parallel Inference Engine: PIE. Goal Rewriting Model and Machine Architecture. New Generation Computing. Goto, A. and Uchida, S. (1987). Towards a High Performance Parallel Inference Machine - The Intermediate Stage Plan for PIM. In Treleaven, P. and Vanneschi, M., editors, Future

Parallel

Computers,

Gregory, S. (1987). Parallel

Implementation.

Berlin. Springer.

Logic Programming

with PARLOG:

The Language

and its

Addison Wesley.

Gregory, S., Foster, I. T., Burt, A. D., and Ringwood, G. Α. (1989). An Abstract Machine for the Implementation of PARLOG on Uniprocessors. New Generation Computing, (6):389-420. Haridi, S. and Brand, P. (1988). ANDORRA Prolog - An Integration of Prolog and Commited Choice languages. In Future Generation Computer Systems, pages 745754, Tokyo. Institute for New Generation Computer Technology (ICOT). Hellerstein, L. and Shapiro, E. (1986). Implementing Parallel Allgorithms in Concurrent Prolog: The Maxflow Experience. Logic Programming, 2:157-184. Hennessy, J. L. and Patterson, D. A. (1990). Computer Approach. Morgan Kaufmann, San Mateo, CA. Hillis, D. W. (1985). The Connection

Machine.

Architecture:

A

Quantitative

MIT Press, Cambridge, MA.

Hillis, W. and Steele, G. (1986). Data Parallel Algorithms. Communications ACM, 29:1170-1183.

of the

Hillyer, B. and Shaw, D. (1984). Execution of Production Systems on a Massively Parallel Machine. Journal

of Parallel

and Distributed

Computing,

submitted.

Hinton, G. E. (1984). Distributed Representations. Technical report, Computer Science Department, Carnegie-Mellon University, Pittsburgh, PA 15213. Hölldobler, S. (1990a). CHCL - a connectionist inference system for horn logic based on the connection method. Technical Report TR-90-042, International Computer Science Institute, Berkeley, CA 94704. Hölldobler, S. (1990b). A structured connectionist unification algorithm. In AAAI '90, pages 587-593. A long version appeared as Technical Report TR-90-012, International Computer Science Institute, Berkeley, CA. Hölldobler, S. (1990c). Towards a connectionist inference system. In Proceedings International

Symposium

on Computational

Intelligence.

of the

274

Hölldobler, S. and Kurfeß, F. (1991). CHCL - A Connectionist Inference System. In Fronhöfer, B. and Wrightson, G., editors, Parallelization in Inference Systems, Lecture Notes in Computer Science. Springer. Ibanez, M. B. (1988). Parallel inferencing in first-order logic based on the connection method. In Artificial

Intelligence:

Methodology,

Systems,

'88. Varna,

Applications

North-Holland. Ibanez, Μ. Β. (1989). Inference parallèle et processus communicants pour les clauses de Horn. Extension au premier ordre par la méthode de connexion. PhD thesis,

I.N.P. de Grenoble, France. Jackoway, G. (1984). Associative Networks on a Massively Parallel Computer. Technical report, Department of Computer Science, Duke University. Journal of Parallel and Distributed Computing (1992). Special issue: Neural computing on massively parallel processors. Journal

of Parallel

and Distributed

Computing,

(3). Kalb, G. and Moxley, R. (1992). Massively

the United States.

Parallel,

Optical,

and Ν euro computing

in

IOS Press, Amsterdam.

Kalé, L. (1988). A Tree Representation for Parallel Problem Solving. In AAAI pages 677-681.

'88,

Kalé, L. V. (1989). The REDUCE OR Process Model for Parallel Execution of Logic Programs. Journal

of Logic

Programming.

Kliger, S., Yardeni, E., Kahn, Κ., and Shapiro, E. (1988). The Language FCP(:,?). In FGCS '88, pages 763-783, Tokyo. Institute for New Generation Computer Technology (ICOT). Kohonen, T., Mäkisara, Κ., Simula, Ο., and Kangas, J., editors (1991). Conference

on Artificial

Neural

Networks

(ICANN-91

International

) , Espoo, Finland. North-

Holland. Kuncicky, D. C , Hruska, S., and Lacher, R. C. (1992). Hybrid systems: The equivalence of expert system and neural network inference. International Journal of Expert Systems.

Kuncicky, D. C , Hruska, S. I., and Lacher, R. C. (1991). Hybrid systems: The equivalence of rule-based expert system and artificial neural network inference. International

Journal

of Expert

Systems.

Kung, C.-H. (1985). High Parallelism and a Proof Procedure. Decision 1:323-331. Kurfeß, F. (1991a). Parallelism

Development.

in Logic

— Its Potential

Support

for Performance

and

Program

Artificial Intelligence. Vieweg Verlag, Wiesbaden.

Kurfeß, F. (1991b). Unification on a connectionist simulator. In International rence

Systems,

on Artificial

Neural

Networks

- ICANN-91,

Confe-

Helsinki, Finland.

Lacher, R. C , Hruska, S. I., and Kucicky, D. C. (1991a). Expert systems: A neural network connection to symbolic reasoning systems. In Fishman, M. B., editor, FLAIRS 9 1 , pages 12-16, St. Petersburg, FL. Florida AI Research Society. Lacher, R. C , Hruska, S. I., and Kuncicky, D. C. (1991b). Backpropagation learning in expert networks. IEEE

Transactions

on Neural

Networks.

275

Lange, T . Ε . a n d W h a r t o n , C. M. ( 1 9 9 2 ) . R E M I N D : Retrieval from episodic m e m o r y by inferencing a n d d i s a m b i g u a t i o n . In B a r n d e n , J. a n d Holyoak, K., editors, Advances in Connectinoist and Neural Computation Theory, v o l u m e II: Analogical Connections. Ablex, Norwood, NJ. Lee, Y., D o o l e n , G., C h e n , H., S u n , G., Maxwell, T . , Lee, H., a n d Giles, C. ( 1 9 8 6 ) . M a c h i n e learning using a higher order correlational network. Physica D, 2 2 ( 1 3):276. Lusk, E., B u t l e r , R., Disz, T . , Olson, R., Overbeek, R., S t e v e n s , R., Warren, D . H. D . , C a l d e r w o o d , Α., Szeredi, P., Haridi, S., B r a n d , P., Carlsson, M., Ciepielewski, Α., a n d H a u s s m a n n , Β . ( 1 9 8 8 ) . T h e A U R O R A OR-Parallel P R O L O G S y s t e m . In International Conference on Fifth Generation Computer Systems, pages 819-830. M a c L e n n a n , B . J. ( 1 9 8 9 a ) . C o n t i n u o u s c o m p u t a t i o n : Taking m a s s i v e parallelism seriously. Technical report, University of Tennessee, C o m p u t e r Science D e p a r t m e n t , Knoxville. M a c L e n n a n , B . J. ( 1 9 8 9 b ) . Outline of a theory of m a s s i v e l y parallel a n a l o g c o m p u t a t i o n . Technical report, University of Tennessee, C o m p u t e r Science D e p a r t m e n t , Knoxville. Nilsson, M. a n d Tanaka, H. ( 1 9 8 8 ) . Massively Parallel I m p l e m e n t a t i o n of Flat G H C on t h e C o n n e c t i o n Machine. In International Conference on Fifth Generation Computer Systems, p a g e s 1031-1040. Ohki, M., Takeuchi, Α., a n d Furukawa, K. (1987). A n Object-oriented L a n g u a g e B a sed o n t h e Parallel Logic P r o g r a m m i n g L a n g u a g e K L l . In Conference on Logic Programming '87, p a g e s 894-909. M I T Press. P a l m , G. ( 1 9 8 0 ) . O n A s s o c i a t i v e M e m o r y . Biological

Cybernetics,

36:19-31.

Pinkas, G. ( 1 9 9 0 ) . C o n n e c t i o n i s t energy m i n i m i z a t i o n a n d logic satisfiability. Technical report, Center for Intelligent C o m p u t i n g S y s t e m s , D e p a r t m e n t of C o m p u t e r Science, W a s h i n g t o n University. Pinkas, G. ( 1 9 9 1 a ) . C o n s t r u c t i n g proofs in s y m m e t r i c networks. In NIPS

'91.

Pinkas, G. ( 1 9 9 1 b ) . E x p r e s s i n g first order logic in s y m m e t r i c c o n n e c t i o n i s t networks. In IJCAI 91 Workshop on Parallel Processing in AI, Sydney, Australia. Pinkas, G. ( 1 9 9 1 c ) . S y m m e t r i c neural networks a n d propositional logic satisfiability. Neural Computation, 3(2):282-291. Pinkas, G. (1992). R e p r e s e n t a t i o n a n d learning of propositional knowledge in s y m m e tric c o n n e c t i o n i s t networks. Technical report, D e p a r t m e n t of C o m p u t e r Science, W a s h i n g t o n University, St. Louis, MO 63130. P o t t e r , J. (1985). The Massively

Parallel

Processor.

M I T Press.

P o t t e r , J. L. ( 1 9 9 2 ) . Associative Computing - A Programming Paradigm for Parallel Computers. P l e n u m P u b l i s h i n g Corporation, N e w York, N Y .

Massively

Ratcliffe, M. a n d R o b e r t , P. ( 1 9 8 6 ) . P E P S y s : A Prolog for Parallel Processing. Technical R e p o r t C A - 1 7 , E u r o p e a n C o m p u t e r Research Center ( E C R C ) , M ü n c h e n , München. Rose, J. a n d Steele, G. ( 1 9 8 7 ) . C*: A n E x t e n d e d C L a n g u a g e for D a t a Parallel Prog r a m m i n g . Technical report, T h i n k i n g Machines Corporation.

276

S c h m i d t , H. W . ( 1 9 9 0 ) . ICSIM: Initial design of an o b j e c t - o r i e n t e d n e t s i m u l a t o r . Technical Report T R - 9 0 - 5 5 , International C o m p u t e r Science I n s t i t u t e , Berkeley, CA 94704. Shapiro, Ε. ( 1 9 8 6 ) . Concurrent Prolog: A Progress R e p o r t . In B i b e l , W . a n d Jorrand, P., editors, Fundamentals of Artificial Intelligence, v o l u m e 232 of Lecture Notes in Computer Science, p a g e s 2 7 7 - 3 1 3 . Springer. Shapiro, E. ( 1 9 8 8 ) . Concurrent

Prolog.

M I T Press.

Shapiro, E. a n d Takeuchi, A. ( 1 9 8 3 ) . O b j e c t - O r i e n t e d P r o g r a m m i n g in Concurrent Prolog. New Generation Computing, (l):25-48. Shastri, L. ( 1 9 8 8 ) . A c o n n e c t i o n i s t a p p r o a c h to k n o w l e d g e r e p r e s e n t a t i o n a n d l i m i t e d inference. Cognitive Science, 1 2 : 3 3 1 - 3 9 2 . Shastri, L. and A j j a n a g a d d e , V. ( 1 9 9 2 ) . From simple a s s o c i a t i o n s to s y s t e m a t i c reasoning: A c o n n e c t i o n i s t representation of rules, variables, a n d d y n a m i c b i n d i n g s using t e m p o r a l synchrony. Behavioral and Brain Sciences. Stanfi.ll, C. and Waltz, D . ( 1 9 8 6 ) . Toward M e m o r y - B a s e d R e a s o n i n g . of the ACM, 2 9 : 1 2 1 3 - 1 2 2 8 .

Communications

Stanfill, C. a n d Waltz, D . ( 1 9 8 8 ) . Artificial intelligence on the c o n n e c t i o n m a c h i n e : A s n a p s h o t . Technical Report G 8 8 - 1 , T h i n k i n g Machines C o r p o r a t i o n , C a m b r i d g e , MA. Steele, G. a n d Hillis, W. ( 1 9 8 6 ) . C o n n e c t i o n Machine Lisp: F i n e - G r a i n e d Parallel S y m bolic P r o c e s s i n g . Technical report, T h i n k i n g M a c h i n e s C o r p o r a t i o n , Technical Report Series. Stolfo, S. ( 1 9 8 7 ) . On the L i m i t a t i o n s of Massively Parallel ( S I M D ) A r c h i t e c t u r e s for Logic P r o g r a m m i n g . In US-Japan AI Symposium. Succi, G. a n d Marino, G. ( 1 9 9 1 ) . D a t a Parallelism in Logic P r o g r a m m i n g . In ICLP Workshop on Parallel Execution of Logic Programs, Paris, France.

91

Sun, R. ( 1 9 9 1 ) . C o n n e c t i o n i s t m o d e l s of rule-based reasoning. Technical report, Brandeis University, C o m p u t e r Science D e p a r t m e n t , W a l t h a m , M A 0 2 2 5 4 . Sun, R. ( 1 9 9 2 ) . A c o n n e c t i o n i s t m o d e l for c o m m o n s e n s e r e a s o n i n g i n c o r p o r a t i n g rules and similarities. Technical report, Honeywell S S D C , M i n n e a p o l i s , M N 5 5 4 1 3 . Swain, M. a n d Cooper, P. ( 1 9 8 8 ) . Parallel Hardware for Constraint Satisfaction. AAAI '88, p a g e s 6 8 2 - 6 8 6 .

In

Szeredi, P. ( 1 9 8 9 ) . Performance Analysis of the Aurora OR-parallel P r o l o g S y s t e m . In North American Conference on Logic Programming. T h i n k i n g Machines ( 1 9 8 7 ) . C o n n e c t i o n Machine M o d e l CM-2 Technical S u m m a r y . Technical R e p o r t H A 8 7 - 4 , T h i n k i n g Machines C o r p o r a t i o n . Tucker, L. W. and R o b e r t s o n , G. G. ( 1 9 8 8 ) . Architecture and A p p l i c a t i o n s of the C o n n e c t i o n Machine. Computer, (8):26-38. Ueda, K. ( 1 9 8 5 ) . G u a r d e d Horn Clauses. Technical R e p o r t T R - 1 0 3 , I n s t i t u t e for New G e n e r a t i o n C o m p u t e r Technology ( I C O T ) . U e d a , K. ( 1 9 8 6 ) . G u a r d e d Horn Clauses: A Parallel Logic P r o g r a m m i n g L a n g u a g e w i t h the C o n c e p t of a Guard. Technical R e p o r t T R - 2 0 8 , I n s t i t u t e for N e w G e n e r a t i o n C o m p u t e r T e c h n o l o g y ( I C O T ) , Tokyo, J a p a n .

277

Uhr, L. (1985). Massively Parallel Multi-Computer Hardware = = Software Structures for Learning. Technical report, New Mexico State University, Las Cruces, NM. Waltz, D. (1987a). Applications of the Connection Machine. Computer,

20:85-100.

Waltz, D. (1987b). The Prospects for Building Truly Intelligent Machines. Technical report, Thinking Machines Corporation. Waltz, D. L. (1990). Massively Parallel AI. In Ninth National Conference on Artificial Intelligence, Boston, MA. American Association for Artificial Intelligence. Waltz, D. L. and Stanfill, C. (1988). Artificial Intelligence Related Research on the Connection Machine. In International

Conference

on Fifth

Generation

Computer

Systems, pages 1010-1024, Tokyo. Institute for New Generation Computer Technology (ICOT). Wang, J. (1989). Towards a New Computational Model for Logic Languages. Technical report, Department of Computer Science, University of Essex, Colchester. Wang, J., Marsh, Α., and Lavington, S. (1990). Non-WAM Models of Logic Programming and their Support by Novel Parallel Hardware. International Workshop on Massively Parallel Inference Systems. Warren, D. (1987). The SRI model for OR-parallel execution of Prolog. In on Logic Programming '87, pages 92-102.

Symposium

Warren, D. H. (1983). An Abstract Prolog Instruction Set. Technical Report 309, SRI International, Artificial Intelligence Center, Menlo Park, California. Wattenberg, U. (1992). Massively

Parallel,

Optical,

and Neurocomputing

in Japan.

IOS

Press, Amsterdam. Wavetracer, Inc. (1991). The multiC Wavetracer, Inc. (1992). Zephyr

Programming

Installation

Language.

and Operation.

Wavetracer, Inc. Wavetracer, Inc.

Westphal, H. (1986). Eine Beurteilung paralleler Modelle für Prolog. In '86, pages 227-240, Berlin. Springer.

Gl-Jakrestagung

Yasuura, H. (1983). On the Parallel Computational Complexity of Unification. Technical Report 27, Institute for New Generation Computer Technology (ICOT), Tokyo.

Parallel Processing for Artificial Intelligence 1 L.N. Kanal, V. Kumar, H. Kitano and C.B. Suttner (Editors) © 1994 Elsevier Science B.V. All rights reserved.

279

Chapter 11 Representing Propositional Logic and Searching for Satisfiability in Connectionist Networks Gadi Pinkas Department of Computer Science and The Center for Optimization and Semantic Control, Box 1045, Washington University, St. Louis, 63130.

The chapter presents methods for representing logic formulas in connectionist networks and for searching for truth assignments that satisfy them. Algorithms are given for transforming any logic formula into a network that performs energy minimization in linear time. The paraüel relaxation process that underlies networks of energy minimization reveals an efficient hill climbing algorithm for satisfiability problems. Experimental results indicate that the parallel implementation of the algorithm will give extremely good average-case performance, even for large-scale, hard satisfiability problems (randomly generated).

1.

Introduction

An almost s t a n d a r d approach in AI going back to [ MacCarthy 68], is to represent an agent's knowledge as a collection of formulas, which can be viewed as a knowledge base. An agent is then said to know a fact, if it is provable from the formulas in the knowledge base. The majority of existing formal AI reasoning systems are based on this logicist view; i.e., the use of logic formulas for representation and the use of a formal proof theory in order to reason about facts or beliefs not mentioned explicitly. While scientists in traditional, symbolic AI were concentrating on development of powerful knowledge representation systems, connectionists were concentrating on powerful learning and adaptation mechanisms. Connectionism was criticized for lacking mechanisms like compositionality and systematicity, which are essential for high level cognitive tasks and are easy for symbolic approaches [ Fodor, Pylyshyn 88]. It is clear t h a t we would like to have systems t h a t have sufficient expressive power, t h a t perform quickly (the brain suggests massive parallelism), and t h a t are capable of learning and adjusting. As [ Hinton 90] pointed out, the u l t i m a t e goal

280

for both scientific approaches is to find efficient learning procedures for representationally powerful systems,... Appreciating the benefits of both the connectionist paradigm and the logicist approach, this chapter tries to build the foundations for a bridge across the two. Dealing with foundations, it concentrates on a logical formalism t h a t is simple and well understood - propositional logic. In this chapter, I shall consider only propositional knowledge; however, the approach can be extended to predicate calculus with nonmonotonic abilities [ Pinkas 91c]. Among the different connectionist models, I choose to consider those with symmetric m a t r i x of weights. This family of models includes Hopfield networks [ Hopfield 82], [ Hopfield 84], Boltzmann machines [ Hinton, Sejnowski 86], harmony theory [ Smolensky 86], mean field theory [ Hinton 89], and other variations. The reasons for using symmetric connectionist networks (SCNs) are the following: (i) Symmetric networks have a natural parallel realization. Implementations exist in analog, digital and optical hardware [ Alspector et al. 88], [ Abu Mustafa, Psaltis 87]. (ii) Symmetric networks can be characterized by energy functions. These functions make it easier to specify the networks' behavior [ Feldman 85]. (iii) Symmetric networks have been used successfully to express and solve (approximate) hard problems [ Hopfield, Tank 85]. (iv) Symmetric networks are capable of representing a large set of asym1 metric networks [ Pinkas 91b]; therefore they are quite powerful and we will not lose expressive power if we restrict ourselves to the sym2 metric case. My purpose is to show that: 1) Propositional logic can be represented efficiently in symmetric networks. 2) T h e algorithm implemented by the network is efficient on average. T h e chapter is organized as follows. Section 2 presents the energy paradigm. Section 3 shows t h a t propositional logic can be represented compactly in SNs. Section 4 provides surprising experimental data, and Section 5 discusses the results and related work. Proofs for the theorems appear in [ Pinkas 92]. 1

I n fact, every non-oscillating network of binary threshold units is representable in SCNs. 2 Sometimes an asymmetric form of a symmetric network will perform better; therefore, for efficiency, we may consider not to restrict ourselves to the symmetric case.

281

2.

The energy paradigm

Finding m i n i m a for quadratic functions is the essence of symmetric connectionist models used for parallel constraint satisfaction [ Hopfield 82], [ Hinton, Sejnowski 86], [ Smolensky 86]. These models are characterized by a recurrent network architecture, a symmetric m a t r i x of weights (with zero diagonal) and a quadratic energy function t h a t should be minimized. Each unit asynchronously computes the gradient of the function and adjusts its activation value, so t h a t energy decreases gradually. T h e network eventually reaches equilibrium, settling on either a local or a global m i n i m u m . [ Hopfield 82] demonstrated t h a t certain complex optimization problems can be stated as constraints t h a t are expressed in quadratic energy functions and be approximated using these kind of networks. There is a direct m a p p i n g between these networks and the quadratic energy functions they minimize. Every quadratic energy function can be translated into a corresponding network and vice versa. Weighted arcs (i.e., pairwise connections) in the network correspond to weighted terms of two variables in the energy function (with opposite sign). Thresholds of units in the network correspond to single-variable terms in the function. Most of the time I shall not distinguish between the function and the network t h a t minimizes it. An example of a network and its energy function is given in Figure 1.

2.1.

High-order

energy

functions

To represent arbitrary logic formulas, a network will need the power of either high-order connections or hidden units. This section defines highorder networks, and shows how to convert t h e m into standard (pair-wise) networks by introducing new hidden units. High-order connectionist networks have sigma-pi units with multiplicative connections [ R u m e l h a r t et al. 86]. It is a common intuition t h a t highorder networks can better express high-order problems [ Sejnowski 86], and can compute functions t h a t are not computable if only second-order connections are allowed [ Williams 86]. In particular, symmetric networks can be easily extended to handle high-order connections. Naturally, such networks may be viewed as minimizing high-order energy functions [ Sejnowski 86]. η A fc-order energy function is a function Ε : { 0 , 1 } —• ΊΖ t h a t can be expressed as sum of products, with product terms of up to k variables. A

282

Fig. 1. A symmetric network that represents the function Ε = —2NT — 2ST —

2WT + 5T + 2RN - WN + W - R + S. k

Ar-order energy function is denoted by: E (xi}...,

xn) — ι

w

^

' ' 'Xik H

iif...,ikXii

1" X ] 17· C Λ,· x Aj. Each Vip can be u p d a t e d using the u p d a t e d Ω^-'s without having to know the values of other ν variables. This feature leads to a faster sequential algorithm as well as fast parallel implementations. An ordered pair < i , p > is used to denote t h a t object o; is assigned label Ip. Throughout this section, such a pair is referred to as an assignment (pair). This assignment of label lp to object Oi is said to be feasible if V; 3q such t h a t Ω?· [p, q] = 1

(4.3)

343

As the computation proceeds, some assignments may become infeasible. T h e n , Ω^-'s are u p d a t e d (by resetting some entries) such t h a t Equation 4.2 is satisfied. This u p d a t e operation can be formulated as follows: if

becomes infeasible, then Vj, q Ω^·[ , it is easy to check if this < i , p > assignment remains feasible by checking this counter variable for 0. Each iteration takes O(nm) time. Since there are at most nm assignment pairs t h a t can be placed on the stack, the total number of iterations executed is upper bounded by nm. Thus, the algorithm in Fig. 4 runs in 2 2 0(n m ) time. T h e time unit corresponds to time needed for performing a simple arithmetic/logic operation. A parallel version of the above algorithm is shown in Fig. 5. T h e architecture consists of nm P E ' s connected to a common bus as shown in Fig. 6.

344

1. 2. 3. 4. 5·

{ Initialization } for i = 1 t o η d o for ρ = 1 t o m d o Vip «- 1; for j = 1 t o η d o WpLfl-O;

6. for g = 1 t o m d o //initialize counter v a l u e / / 7. if (Ω?· [ρ, ϊ] = 1) t h e n Nip [j] , 5 ) ; 11. v

becomes infeasiblel// pushstack(< j , ç >,5); vjq 4 - 0; end; end end end e n d {while}

23. 24. 25. 26. 27. 28. 29.

Fig. 4. A Fast Sequential

Algorithm

345

{ Initialization } 1. p a r a l l e l d o (in P E î p , 1 < 2' < n, 1 < ρ < m) 2. vip «- 1; 3. Sendip = < j,q > ) t h e n Sendip

Architecture

Each P E has the following components: (i) a register for storing Vip, (ii) a memory of size nm bits to store the nm variables, Ω?·[ρ, q], 1 < j < n, 1 < q < m, (iii) η registers for storing counter variables Nip[j]y 1 < j < n, (iv) an Id register which stores < i , p > in P E l p , (v) a register .Send which is set if the assignment < i , p > becomes infeasible and has not been acknowledged by the bus controller, and (vi) a bus interface unit. A block diagram of the P E is shown in Fig. 7. As the computation proceeds, the nm Ω variables are accessed in each P E to u p d a t e the corresponding counter as described in the algorithm in Fig. 4. Therefore, P E î p can determine the feasibility of assignment < i,p > by using these u p d a t e d A^ î p[j]'s, without having to know the values of the υ variables in other P E ' s . Previous parallel designs [7, 5] communicate the values of ν variables among the a n P E ' s . Also, they employ unbounded fan-in gates to perform the ΠΓ=ι d Σ ρ ^ ι operations in Equation 4 . 1 . During each iteration, P E , p , l < 2 < n , l < p < m , sends its identification ( < ζ,ρ > ) onto the bus if the variable Send{p is set to 1. All the P E ' s with their Send variable set are called contending-PE's. T h e bus controller is used to select an Id from all the Id's placed on the bus by the contending-PE's, and then to inform all the P E ' s with the selected Id. An Id selected during an iteration and broadcast to all the P E ' s is said to be acknowledged by the bus controller. Each contending-PE keeps on sending its Id onto the bus until it is acknowledged. It is easy to verify t h a t each iteration takes constant a m o u n t of time, assuming constant time bus operation. Unbounded fan-in logical operations (employed in [7]) are not needed in this design. T h e total time for labeling is 0(nm).

347

Constraint Variable Registers Counter Registers

*

Decoder

η ALU Ν M

Wend

h Bus Interface Unit

to/from bus Fig. 7. Organization of the Ρ Ε Used in the Bus Connected

Architecture

This design offers several advantages compared with all the designs in the literature. T h e number of initially assignable labels for each object is usually much smaller t h a n the total number of labels. T h u s , the time to increment/decrement the counter variables can be very small. Since the basic operation performed is u p d a t i n g counters, the design is suitable for implementation on general purpose parallel machines. Also, the design can be expanded to handle large d a t a size by replacing all the local registers with external memory. Since, in most applications, the range of counter variables is much smaller t h a n m, a relatively small counter can be used to solve a problem instance with large m and η without the need for redesign. Note t h a t earlier designs [7, 5] can only be used to solve problem instances over a small range of m and η due to the hardware needed in their designs for unbounded fan-in operations. Furthermore, our design is suitable for partitioned implementations; when a fixed number of Ρ processors are available, Ρ < nm, this design can be modified to allow each processor to handle more t h a n one assignment. 4.I.3. Bus Implementation There are several choices for implementing the bus system. One possibility is to use a log(nra)-bit reconfigurable bus with a controller. Each P E has the capability to send a signal to the bus to disconnect the remainder

348

of the bus away from the controller. Such a reconfigurable bus has been well studied in [15]. Fig. 8 shows the bus configuration employed by both contending- and non-contending-PE's. Fig. 9 shows the configuration when non-contending- PE

contending- PE

Fig. 8. Bus Configuration Employed by Contending and

Non-contending-PE's

the P E ' s are receiving the acknowledged Id from the controller. Notice t h a t

Fig. 9. Bus Configuration for Id Broadcasting the Id of the contending-PE closest to the controller is acknowledged. An alternate approach is to use a one-bit wired-OR bus. No controller is necessary in this implementation. T h e Id of a P E ranges from 1 to n m ; [log(nm + 1)] bits are used to represent each Id. Each bus operation (lines 19-20 in Fig. 5) is performed over |~log(nra + 1)] steps. Let active P E ' s denote a subset of all contending-PE's. Initially, all contending-PE's are active. During the k-th step, 1 < k < [log ( n m + 1)], each active P E sends the fc-th bit of its Id onto the bus, while each of the other P E ' s sends a "0" onto the bus. T h e P E ' s retrieve the OR-ed value from the bus. An active P E at the beginning of the current step remains active at the end of the current step if and only if the retrieved signal from the bus is less t h a n or equal to the signal it sent to the bus. In the last step of the bus operation at most one P E is active whose Id is the largest Id among all the contending-PE's. T h e [log(nm + 1)] bits retrieved from the bus during the |"log(nm + 1)] steps correspond to the Id of this P E . T h e algorithm terminates when there are no contending-PE's in an iteration. This situation corresponds to receiving "0"s in all the [log(nm-h 1)] steps.

349

T h e above bus feature is available in the Image Understanding Architecture (IUA) [19]. Each iteration is executed in 0 ( l o g ( n m ) ) time. This results in 0 ( n r a l o g ( n r a ) ) total execution time. 4.2.

Pariitioned

Implementations

In this section, two partitioned implementations are developed based on the design in Section 4.1.2. 4.2.Ι. Implementation on a Bus Connected Architecture In the designs in Section 4.1, each of the nm P E ' s is responsible for determining the value of a V{p. Given a fixed number of P E ' s , say Ρ ( < n m ) , we can design a partitioned implementation by assigning each P E to process Vjr distinct v i p s . T h e proposed architecture is shown in Fig. 10. Each

MMi

1\

1\

I\

\1

\1

)

PEi

1\ Bus Controller

!

•· ·

MMj

)

!

P E 2

1\ )1

•· ·

M M m

1

1 )

1

VER

1\

1\

1

\!

)

Fig. 10. Architecture for Partitioned Design Using Bus P E is connected to an external Memory Module (MM) to store all the relevant d a t a . T h e d a t a stored in each MM include V{p variables, and, for each ν,· ρ , the corresponding nm Ω^·[ρ, q] variables and η N{p[j] counter variables. Also, a flag Send{p for V{p is stored in its corresponding memory module. This flag is used to indicate if the infeasibility of this assignment has been acknowledged by the bus controller. Such an acknowledgment initiates u p d a t e of the counters defined in Section 4.1.2. P E ^ , 1 < k < P , has a flag Sdk to indicate if there is any infeasible assignment among its

350

assignments yet to be acknowledged. Thus, Sdk is set to 1 if and only if there is at least one Sendip equal to 1 among all the ^ Senats in MM*. T h e organization of the P E is shown in Fig. 11.

to/from MM k

Decoder

Register

' >

ALU

L _ _ _ l

i

> Sdk

Bus Interface Unit PE*

I

to/from bus

Fig. 11. Organization of the Ρ Ε in the Partitioned Design Using Bus T h e parallel algorithm is described in Fig. 12. T h e three major procedures, Initialize, Id-broadcastl and Update, are given in Fig. 13, Fig. 14 and Fig. 15 respectively. ld(w) (in line 3 as well as in line 12 of Fig. 12) represents the index of the w-th V{p variable stored in the P E . T h e procedure Initialize^', p; i',ρ') initializes the η counter variables for V{p and returns an Id < i' ,ρ' > to the main procedure if this assignment is infeasible. The Read( ) and Write( ) instructions in the algorithms represent the operations executed by the P E ' s to retrieve and modify the contents of the memory modules. At the end of the initialization process (lines 1-7) in Fig. 12, each P E retains the Id of an infeasible assignment, if any. This Id is then placed on the bus for acknowledgment during the first iteration. During each iteration (lines 9-16) in the main procedure, procedure Idbroadcastl executes the bus operation described in Section 4.1.3 for each P E . T h e procedure U p d a t e ( i , p , j , q\i',ρ') performs the u p d a t e operation. This procedure is similar to the operation described in Section 4.1.2 except t h a t the flag Sd is set to 1 in each P E to indicate if there is any infeasible Id to be acknowledged. It is clear t h a t the execution of each Initialize procedure takes O ( n r a )

351

Partition-bus( ) { Initialization } 1. p a r a l l e l d o (in P E * , 1 < * < P) 2. for w = 1 t o ψ d o 3. < i , p > «- Id(u;); 4. Initialize^', p; i',p')\ 5. end; 6. < i , p > , 7. p a r a l l e l e n d ;

8. 9. 10. 11. 12. 13. 14. 15. 16. 17.

{ U p d a t e Iteration } repeat p a r a l l e l d o (in P E * , 1 < k < P) Id-broadcast l(z,p; j , q)\ { execution stops according to a condition in this procedure } f o r u; = 1 t o ^ do < i , p > ) t h e n Sendip 6) i received by each P E . T h e algorithm terminates when the received Id is a nil Id. T h e procedure U p d a t e ( j , q) performs the u p d a t e operation as in lines 20-26 in Fig. 4. It can be easily verified t h a t the time to complete the Initialization step

356

Fig. 19. A Model of a Distributed Memory

1. 2. 3.

System

{ Initialization } parallel d o (in all PEs) < t, ρ > = nil s t o p ; 8. else 9. < ι, ρ > 4— Update(j, q) 10. parallel e n d 11. forever

Fig. 20. A Partitioned Parallel Algorithm

357 2

2

n

is 0( ™ )Tcomp, where Tcomp corresponds to the time required for a basic c o m p u t a t i o n step in a P E . T h e execution time for each iteration can be represented by TcommW

+

0(—)TComp

where T^omm(l) denotes the communication time for selecting and broadcasting an Id to all the P E ' s . Clearly, the performance of this implementation depends on the computation time relative to the communication time. T h e above algorithm can incur excessive communication overhead. This situation can be handled by notifying each processor with more t h a n one infeasible assignment, if available, during each iteration. It has been observed t h a t most P E ' s produce many infeasible assignments, especially in early iterations. This approach can offer a better balance between comp u t a t i o n and communication times by allowing more computation work to be performed during each iteration using a slightly increased a m o u n t of communication. T h e total number of iterations can be expected to be reduced. T h e block diagram shown in Fig. 21 presents a situation where each P E contributes a certain number of Id's to be concatenated with the Id's from other P E ' s . Each shaded box represents an infeasible Id found in the previous iteration (or in the initialization procedure). T h e Id's are concatenated and m a d e available to all the P E ' s to perform the necessary u p d a t e operation during the current iteration. It would be desirable if, during each iteration, each processor can append all its infeasible Id's to the message to be broadcast to all the P E ' s . However, the system overheads incurred in handling variable message size can offset the intended savings in communication time. We chose to balance the communication and computation times using a fixed-size message from each P E , partly due to the fact t h a t most currently available parallel machines support this communication mechanism through system calls. T h e modified algorithm is described in Fig. 22. X (in line 2 as well as in line 8) denotes an array to collect infeasible Ids from the Initialize() or Update() procedures. Let χ denote the number of entries allowed in an X. Note t h a t the number of infeasible assignments available in a P E can be less χ. Y (in line 6) denotes the message containing all X ' s . 5.1.

Implementations

Results

T h e algorithm shown in Fig. 22 was implemented on an i P S C / 8 6 0 . T h e machine has 32 nodes interconnected by a five-dimensional hypercube network in which a modified "wormhole" routing hardware was used.

358

Network

Fig. 21. Broadcasting Multiple Id's

{ Initialization } p a r a l l e l d o (in all PEs) X «- Initialize() parallel end ; { Update Iteration } 4. repeat 5. p a r a l l e l d o (in all PEs) 6. Y · ßq+itd and a tuple a t , / ) G IS(r), define T[r, (i, j , fc,/)] as the set of sequences of trees ( Δ ι , . . . , Δ ? ) such t h a t Tpt Α Δ,· (1 < i < q) and yield(Ai) || 2/2β/ί/(Δ2) II · · · II yield(Ag) = α,+ι · · -ajYcik+i · · a/ (where || stands for string concatenation). T h e following l e m m a gives an upper bound on the level of the index tuples computed by the parsing algorithm. We define size ( θ ) as the number of nodes in tree Θ. L e m m a 1. (1) / / a for every tree Θ (2) / / r = a —• ßi>q then for every

is a grammar node and t is a tuple in IS(a), then G Τ [ α , ΐ ] , level[a,t] < sizefô) + dlen(Q). · is a dotted rule and t is a tuple in IS(r), tree sequence ( Δ ι , . . . , Δ ? ) G T[r,t], level[ryt]
dlen(&) + 1 because a substitution step is needed to derive Θ, given θ ' . Therefore, size(Q) + dlen(&) > + dlen(&)+l. Now,

8ize(&)

level[a, t] < level[p, t] + 1 < size(&)

+ a 7 e n ( 0 ' ) + 1, by the induction hypothesis

dlen(Q\) + cf/en(0 2 ) + 1 because an adjunction step is needed t o derive 0 , given 0 i and 0 2 . Therefore, size(Q) + dlen{ß) > size{ßi) + d / e n ( 0 i ) -h size(Q2) 4- a 7 e n ( 0 2 ) . Now, level[cx, t] < max{/et;e/[a, ^i], level[p, ^ 2 ] } + 1 < m a x { s 2 z e ( 0 i ) + d / e n ( 0 i ) , size(Q2) < s i z e ( 0 i ) + d / e n ( 0 i ) + size(Q2)

1 for i = 1, 2. Case 3: (Completion) Suppose t h a t α is a non-leaf node and tuple t was included in IS(a) via completion. T h a t is, t is in IS(a —• where the ßi's are t h e children of a . Let 0 be a tree in T[o, £ f = 1 Therefore, d / e n ( 0 ) > ( ^ ) f = 1 size(Ai) + dlen(Ai)) + 1.

size(e) size(e)

= + (Σ*=ι

dlen(

404

Now, level[a, t] < level[a —• βι^·, < ^2size(Ai) q · ßq+i}d is a dotted rule and tuple t is in LS(r). Let r' = a —• · /? g,d. Then, there are tuples tfi G IS(r') and ^ G IS(ßq) such t h a t ί = *i 0 £ 2 - Clearly, for any tree sequence ( Δ ι , . . ., Aq) in IS(r), ( Δ ι , . . . , Δ ^ _ ι ) is a tree sequence in T[r',*i] and Δ ? is in T[ßq,t2]. Now, /eve/[r,

< maxj/eve/fr', β\Α · is dotted rule and tuple t is in IS(r), then for every tree sequence ( Δ ι , . . . , Aq) G T[r, Σ ί = ι «^β(Δ,·) < (maxtreesize + l ) n and ]Γ^=ι dlen(Ai) < η because Δ ι , . . . , Aq occur as subtrees of some larger derived tree in Τ[α,*'], which itself has at most (maxtreesize + l ) n nodes and has a derivation sequence of length at most n. From L e m m a 1, it follows t h a t every index tuple is included in its index set after at most (maxtreesize -\-2)n — cn iterations of the m a i n loop. •

405

References [1] A. Abeille, Parsing French with Tree Adjoining Grammar: Some Linguistic Accounts, Proc. 12th International Conference on Computational Linguistics, Budapest, August 1988. [2] A. Abeille, A Lexicalized Tree Adjoining Grammar for French: The General Framework, Technical Report MS-CIS-88-64, Department of Computer and Information Science, University of Pennsylvania, 1988. [3] A. Abeille, Extraction Out of NP in Tree Adjoining Grammars, Papers from the 24th Regional Meeting of the Chicago Linguistic Society, Chicago, Illinois, 1988. [4] A. Abeille, Κ. Bishop, S. Cote, Y. Schabes, and A. K. Joshi, Parsing with Tree Adjoining Grammars: The Current State of Implementation, Technical Report, Department of Computer and Information Science, University of Pennsylvania, August 1988. [5] A. Abeille and Y. Schabes, Parsing Idioms in Tree Adjoining Grammars, Proc. Fourth Conference of the European Chapter of the Association for Computational Linguistics, Manchester, 1989. [6] G. E. Barton, The Computational Difficulty of ID/LP Parsing, Proc. 23rd Annual Meeting of the Assoc. for Comp. Linguistics, July 1985, pp. 76-81. [7] G. E. Barton, R. C. Berwick, and E. S. Ris tad, Computational Complexity and Natural Language, MIT Press: Cambridge, Massachusetts, 1987. [8] R. C. Berwick and A. S. Weinberg, The Grammatical Basis of Linguistic Performance: Language Use and Acquisition, MIT Press: Cambridge, Massachusetts, 1984. [9] A. Borodin and J. E. Hopcroft, Routing, Merging and Sorting on Parallel Models of Computation, Journal of Computer and System Sciences, 30 (1985), pp. 130-145. [10] K. Bishop, S. Cote and A. Abeille, A Lexicalized Tree Adjoining Grammar for English, Technical Report, Department of Computer and Information Science, University of Pennsylvania, 1989. [11] G. Gazdar, E. Klein, G. K. Pulhim, and I. A. Sag, Generalized Phrase Structure Grammar, Blackwell Publishing: Oxford, England, 1985. [12] G. Gazdar and G. K. Pullum, Computational Relevant Properties of Natural Languages and Their Grammars, New Generation Computing, 3 (1985), pp. 273-306. [13] M. D. Harris, Introduction to Natural Language Processing, Reston Publishing Company: Reston, Virginia, 1985. [14] K. S. Jones and Y. Wilks (eds.), Automatic Natural Language Parsing, Ellis Horwood Limited: Chichester, England, 1983. [15] A. K. Joshi, L. S. Levy, and M. Takahashi, Tree Adjunct Grammars, Journal of Computer and System Sciences, 10 (1975), pp. 136-163. [16] A. K. Joshi, How Much Context-Sensitivity is Necessary for Characterizing Structural Descriptions — Tree Adjoining Grammars, in Natural Language Processing — Theoretical, Computational and Psychological Perspectives,

406

[17]

[18] [19]

[20]

[21] [22]

[23] [24] [25]

[26]

[27]

[28]

[29]

[30] [31]

[32]

D. Dowty, L. Karttunen and A. Zwicky (eds.), Cambridge University Press, New York, 1985. C. Kaklamanis, D. Krizanc,and T. Tsantilas, Tight Bounds for Oblivious Routing in the Hypercube, ACM Symposium on Parallel Algorithms and Architectures, 1990, pp. 31-36. S. R. Kosaraju, Speed of Recognition of Context-Free Languages by Array Automata, SIAM Journal on Computing, 4 (1975), pp. 331-340. R. Kaplan and J. Bresnan, Lexical Functional Grammar: A Formal System for Grammatical Representation, in The Mental Representation of Grammatical Relations, J. Bresnan (ed.), MIT Press: Cambridge, Massachusetts, 1983. M. Kay, Parsing in Functional Unification Grammar, in Natural Language Parsing: Psychological, Computational, and Theoretical Perspectives, Cambridge University Press: Cambridge, England, 1985, pp. 251-278. M. King (ed.), Parsing Natural Language, Academic Press: London, England, 1983. A. S. Kroch and A. K. Joshi, The Linguistic Relevance of Tree Adjoining Grammars, Technical Report MS-CIS-85-16, Department of Computer and Information Science, University of Pennsylvania, April 1985. W. G. Lehnert and M. H. Ringle, Strategies for Natural Language Processing, Lawrence Erlbaum Associates: Hillsdale, New Jersey, 1982. F. T. Leighton, Introduction to Parallel Algorithms and Architectures: Trees, Arrays, Hypercubes, Morgan-Kaufmann Publishers, San Mateo, CA, 1992. M. A. Palis and S. Shende, Upper Bounds on Recognition for a Hierarchy of Non-Context-Free Languages, Theoretical Computer Science, 98 (1992), pp. 289-319. M. A. Palis and S. Shende, Sublinear Parallel Time Recognition of Tree Adjoining Languages, Proc. International Conference on Parallel Processing, Chicago, Illinois, August 1989, pp. 202-205. M. A. Palis, S. Shende and D. Wei, An Optimal Linear-Time Parallel Parser for Tree Adjoining Languages, SIAM Journal on Computing, 19:1 (February 1990), pp. 1-31. F. C. N. Pereira and D. H. D. Warren, Definite Clause Grammars for Language Analysis - A Survey of the Formalism and A Comparison with Augmented Transition Networks, Artificial Intelligence, 13 (1980), pp. 231-278. C. Pollard, Generalized Phrase Structure Grammars, Head Grammars, and Natural Languages, Ph.D. Dissertation, Dept. of Linguistics, Stanford University, 1984. W. Ruzzo, Tree-Size Bounded Alternation, Journal of Computer and System Sciences, 22 (1980), pp. 218-235. W. Rytter, The Complexity of Two-Way Pushdown Automata and Recursive Programs, Combinatorial Algorithms on Words, A. Apostolico and Z. Galil (editors), Springer-Verlag, Heidelberg, 1985, pp. 341-356. Y. Schabes and A. K. Joshi, An Earley-type Parsing Algorithm for Tree Adjoining Grammars, Technical Report MS-CIS-88-36, Department of Com-

407

puter and Information Science, University of Pennsylvania, June 1988. [33] Y. Schabes, The TAG System User Manual, Technical Report, Department of Computer and Information Science, University of Pennsylvania, 1989. [34] S. Shende, Computational Feasibility of Some Constrained Formalisms for Natural Language, Ph.D. Dissertation, Department of Computer and Information Science, University of Pennsylvania, 1990. [35] S. M. Shieber, An Introduction to Unification-Based Approaches to Grammar, Lecture Notes No. 4, Center for the Study of Language and Information, Stanford University, 1986. [36] Thinking Machines Corporation, Connection Machine Model CM-2 Technical Summary, May 1989. [37] Thinking Machines Corporation, The Connection Machine CM-5 Technical Summary, October 1991. [38] K. Vijayshanker and A. K. Joshi, Some Computational Properties of Tree Adjoining Grammars, Proc. 23rd Annual Meeting of the Association for Computational Linguistics, Chicago, Illinois, pp. 82-93. [39] D. S. L. Wei, Fast Parallel Routing and Computation on Interconnection Networks, Ph.D. Dissertation, Department of Computer and Information Science, University of Pennsylvania, 1991.

Parallel Processing for Artificial Intelligence 1 L.N. Kanal, V. Kumar, H. Kitano and C.B. Suttner (Editors) © 1994 Elsevier Science B.V. All rights reserved.

409

Chapter 15 Process Trellis and FGP: Software Architectures for Data Filtering and Mining Dr. Michael Factor, Scott J. Fertig and Dr. David H. Gelernter Dr. Michael Factor IBM Israel — Science & Technology Ltd. MATA M — Advanced Technology Center Haifa, 31905 Israel Scott J. Fertig and Dr. David H. Gelernter Yale University Department of Computer Science P.O. Box 2158 New Haven, CT

06520-2158

Software tools for parallel programming used to be a chaotic area, with dozens of contenders promoting different models and methodologies. In the last few years, the field has settled down considerably: there are perhaps half a dozen approaches that are still "in contention." These include message passing systems of various kinds, data parallel languages for synchronous architectures, explicitly-parallel Fortran variants—and (among a few others) Linda[7], which is the system we've used. C-Linda® (the computing language C mated with the coordination language Linda) has been used to date mainly in scientific applications and in graphics (production Linda applications exist in areas like computational fluid dynamics, molecular modeling, radar cross sectioning, seismic simulations, genetic database search, genetic linkage analysis, ray-tracing and others). In our own group, however, we've long been interested in AI applications. The Linda model, based on a shared, associative, object memory, through which processes communicate "anonymously," is appropriate (we believe) to a wide range of significant AI applications. In this article we discuss software architectures for data filtering and mining built using Linda.

1.

Introduction

A software architecture is a virtual machine; it is an applicationindependent frame into which one can plug application-dependent m o d -

410

ules. Production rule systems are one example of a software architecture. T h e rule interpreter is the application-independent frame, and the rules are the application-dependent modules. A software architecture simplifies the creation and maintenance of an application by organizing and structuring the program. T h e F G P Machine is a software architecture for a database-driven expert system t h a t learns. The Process Trellis is a software architecture for realtime, heuristic monitors. These two software architectures, although geared towards very different types of applications, have several commonalities: they are both parallel systems, both are useful for applied AI, and both have been implemented in Linda[7]. Just as one can take advantage of their commonalities to combine two physical machines, producing a tool more powerful t h a n either when taken alone — consider the marriage of fax and computer — we can combine two software architectures, producing a more useful architecture t h a t demon1 strates the best features of each of the elements. Further, as we consider a physical machine to be more powerful the easier it is to combine with other machines, we should judge a software architecture by the same criterion. This article presents an overview of the F G P Machine and the Process Trellis software architectures. T h e descriptions emphasize the properties of the architectures and the systems' parallel nature. We then describe a symbiotic union of the F G P Machine and Process Trellis in the context of a hypothetical super-system for intelligent network monitoring and m a n agement; this system is more powerful than the sum of its parts. First, in the next section we provide an overview of the Linda programming model. While it may not be customary to describe implementation vehicles, the Linda language simplified development of both the Process Trellis and F G P software architectures. More importantly, Linda's intentionally loose coupling among processes will make easier development of a combined Process Trellis and F G P program (as described) for network monitoring. Given the current state of the art in building parallel AI programs, we believe it is appropriate and useful to describe the software tools (languages and software architectures) t h a t have been used successfully. 2.

Linda and the Master/Worker model

Linda consists of a small number of powerful operations t h a t may be integrated into a conventional base language, yielding a dialect t h a t sup1

Others have looked at combining different software architectures. For instance, ABE [10] provided a coUectionof combination methods, e.g., data flow, blackboard, etc., Hexscon [31] combined existing logic controllers with a production rule architecture, and VentPlan [26] combined quantitative and qualitative reasoning.

411

ports parallel programming. T h u s , for example, C and Fortran with the addition of the Linda operations become the parallel programming languages C-Linda and Fortran-Linda. A programming environment based on 2 L i n d a includes a preprocessor, to translate from a Linda parallel language (C-Linda or Fortran-Linda) into the corresponding base language (C or Fortran), possibly with the use of some automatically-generated auxiliary C-Linda routines, and kernel libraries to support the Linda operations at run-time. Portability comes from the consistency of the preprocessor between systems, while efficiency comes from the use of native C and Fortran compilers for the actual generation of object code, and hardware-specific implementations of the kernels. Since Linda has been discussed at length in the literature (e.g., [3-6]), we include only a brief outline here. T h e Linda model is a memory model. Linda uses a virtual, associative, logically-shared memory called tuple space t h a t consists of a collection of logically-ordered sets of d a t a called tuples. There are two kinds of tuples resident in tuple space (TS): process tuples, which are under active evaluation, and d a t a tuples, which are passive. Process tuples (which are all executing simultaneously) accomplish work by generating, using, and consuming d a t a tuples. Once a process tuple has finished executing, it turns into a d a t a tuple (its result), indistinguishable from other d a t a tuples. Linda provides excellent support for a wide variety of approaches to parallel programming. One particular paradigm t h a t has been used frequently with Linda is known as the Master/Worker Model [6]. Typically, this entails the use of distributed d a t a structures and a group of worker processes (not necessarily identical) t h a t examine a n d / o r modify the d a t a structures in parallel under the general supervision of a master process (which may, itself, do work as well). A great strength of the Linda model is its explicit support for distributed d a t a structures, ie., d a t a structures t h a t are uniformly and directly accessible to m a n y processes simultaneously. Any d a t a tuple sitting in Linda tuple space meets this criterion: it is directly accessible — via the Linda operations described below — to any process tuple in t h a t tuple space. Thus, a single tuple constitutes a simple distributed d a t a structure, but it is easy and often useful to build more complicated multi-tuple structures (arrays, queues, or tables, for example) as well. By 2

A commercial software company, Scientific Computing Associates, Inc. (SCIENTIFIC) has developed a number of parallel programming systems based on Linda. SCIENTIFIC supports commercial versions of Linda that now run weU on a broad range of parallel computers, from shared-memory multiprocessors, to distributed-memory machines such as hypercubes, to networks of workstations. The Process Trellis software architecture was developed using a research version of C-Linda at Yale University. The parallel version of the FGP software architecture was developed using SCIENTIFIC's commercial C-Linda system.

412

comparison, message passing systems deal solely with transient d a t a (messages) t h a t exist for only a limited time: between assembly by the sender and disassembly by the receiver. Moreover, the messages are accessible only to two processes and at specific times: the sender before transmission and the receiver after transmission. Another feature of the Linda system is its intentionally loose coupling among processes. Other models (for example, those based on d a t a parallelism) implicitly or explicitly bind processes tightly together. Taken to an extreme, this gives the SIMD model in which all processes perform identical operations in lock step. Even message passing assumes t h a t there is significant underlying synchronization between message senders and receivers. In contrast, Linda processes aspire to know as little as possible about each other — they interact only with the intermediation of d a t a stored in tuple space. As a result, programmers are not forced to think in terms of simultaneities. This simplifies greatly the potentially formidable task of parallel programming, since each individual process can be developed more-or-less independently of the others. Linda's loose interprocess coupling has other advantages as well. Tuple space can be viewed as a long term d a t a memory — deposited tuples remain in tuple space until they are explicitly removed by some process. This means t h a t processes can interact through time as well as space (or machine location), since the producer and consumer of a tuple need never coexist simultaneously. A natural application of this idea arises when parallel computations produce o u t p u t d a t a t h a t must later be used as inputs by completely independent visualization programs or other postprocessors. This sort of interaction is easy to express in Linda, but may well be extremely difficult to express using paradigms like message passing for which the "data" (i.e., messages) have no long term existence. 3.

The F G P Machine

T h e F G P Machine is a software architecture for building expert databases: programs t h a t use a large database of case descriptions to classify, extend and cross-reference a new case. An F G P Machine is defined in terms of a single kind of data-object and three primitive operators: .Fetch, Generalize and Project. These define a virtual machine in terms of which the system is programmed. Using combinations of these operators, the F G P Machine is designed to convert static case descriptions in any domain into expert advice. An F G P Machine is a software architecture t h a t processes input records (either queries or d a t a ) in the context of a case database. Our implementations meet the following goals:

413

- When presented with a new record (whether query or d a t a ) , the F G P Machine "reasons" on t h e basis of specific cases and groups of cases, and therefore is able to cite specific precedents, including precedents t h a t may themselves be incomplete. Hence, both query and d a t a records must be represented as cases. - T h e F G P Machine modifies its behavior on the basis of every new information-providing transaction, and subsumes the functions of a conventional information-ret rie val system. Newly entered d a t a are available for retrieval and inferencing immediately. - T h e F G P Machine serves as a viable learning system, equaling or surpassing the accuracy of other purely data-driven learning programs on a s t a n d a r d classification task. - Unlike statistical analysis packages or connectionist networks, the F G P Machine is designed to interact with a user as a single case is described incrementally. For example, it can generate line-by-line commentary on new cases, possibly offering plausible guesses for the values of attributes not yet described (extensions). - T h e F G P Machine displays not only "quantitative" but "qualitative" expertise. To provide "qualitative expertise", the F G P Machine may need to pursue possibly p r e m a t u r e hypotheses when a t t e m p t i n g to classify a new (and only partially-instantiated) case. Our protocol analysis [18] convinced us t h a t h u m a n experts, when asked to classify a difficult case, suggest extensions and also cite interesting precedents on the basis of a small but evocative degree of attribute overlap. T h e "simulated speculation" heuristic was developed in an effort to approximate this behavior. In the remainder of this section, we discuss the overall design and opera3 tion of the F G P model. Then we move on to discuss the role of parallelism. 3.1.

The FGP Machine

Structure

T h e F G P Machine runs off of a database of unstructured c a s e s , a term we define below. We use the t e r m k n o w l e d g e b a s e to denote any database of such cases t h a t represents some significant portion of experience in a real-world domain. Cases are similar to flat, attribute-vector representations of d a t a , but they are more general. They m a y be incomplete—individual cases need 3

This chapter does not describe the weighting and clustering algorithms implemented to support similarity-based retrieval, nor do we discuss the FGP Machined performance as a learning program. Readers interested in these details are directed to [17] and to [18], or to contact the authors.

414

to have values for only an arbitrary subset of the universe of attributes. Attributes may be multi-valued (as they are in m a n y object-oriented databases). Cases are not assigned to fixed categories. Rather, any attribute in the attribute-universe is a potential category; the program must be willing to direct its inference process at determining the value of any arbitrary attribute for a particular case. This determination may be logically impossible because there is no assumption about consistency of cases (ie two cases may have identical values for every a t t r i b u t e but one.) We adopt this definition of cases because these characteristics are observed in many databases t h a t have been collected for real-world domains. For example, the machine learning archive maintained at the University of California at Irvine contains at least 50 such databases. Not all domains can be so represented; in particular, some domains are better defined by a hierarchy of objects or as a set of if-then rules. However, there are m a n y domains whose objects are naturally represented by cases, and databases of these cases serve as knowledge bases for inferencing. Examples appear in such diverse fields as medicine (radiologist reports, toxin databases), and business (marketing-sales correlation d a t a ) . A case is the basic unit of d a t a t h a t must be represented, operated on and inferred from. W h a t is or is not a case is determined by the domain and not the system architecture. A pnori, cases can neither be abstracted away nor reduced to a set of statistics. Cross referencing to specific precedents is a primary goal, so a just-entered case must be made available immediately and preserved in its raw form as long as possible. T h e F G P Machine provides a framework for similarity-based retrieval of cases related to a given query, a query t h a t itself is presented as a partially instantiated case. One can imagine the F G P Machine's database of cases as a collection of regions in space (cf. the standard vector space text-retrieval model). Each element of the database corresponds to some region. Nearby regions correspond to nearby cases. When presented with an inquiry, the F G P Machine's basic task is to add to the database a new region corresponding to the inquiry. Stationing itself on top of this new region (so to speak), the F G P Machine then looks around and reports the identities of the nearby regions—these will correspond to elements of the database t h a t are nearby to, in other words closely related to, the subject of the inquiry. The F G P Machine can then inspect this list of nearby regions and "generalize"—determine which attributes tend to be shared in common by all or by most of them. It can then guess t h a t these common attributes are likely to hold true for the case being described in the inquiry as well. Having reached whatever conclusions seem reasonable, the F G P Machine simulates "speculation." Temporarily turning aside from the inquiry

415

in hand, it focuses on any "evocative possibilities" t h a t m a y have suggested themselves during the examination of nearby regions. An "evocative possibility" is a d a t u m t h a t might be true, and t h a t would be significant if it 4 were. T h e F G P Machine's interaction with the user (see [18] for an example transcript) represents a combination of fairly safe conclusions, speculation experiments and the subsequent investigation of resulting guesses. 3.2.

Properties

of the FGP

Machine

The F G P Machine differs from others in technical approach, software structure and interface design. Technical approach: our intent was to supplement the conventional, statistical clustering-within-a-feature-space approach with heuristics designed to mimic the behavior of a h u m a n expert. We hypothesized t h a t speculation—the ability to "go out on a limb" and produce meaningful commentary in the absence of any statistically clear-cut solution to a hard problem—was a central capability of h u m a n experts. We implemented this hypothesis in the form of the "simulated speculation" capability. We have proven t h a t the hypothesis is correct—at least in preliminary, fairly smallscale tests—by demonstrating t h a t turning on this capability improves our system's performance.[18] We've also shown t h a t our base system, without simulated speculation, performs as well on a standard classification-system benchmark as the best published results. Software structure. Our system is designed as a virtual machine: a small collection of simple, powerful operators. We can use these operators as a "source language" in which to express a wide range of higher-level, more complex and sophisticated operations. But the implementation need only support the simple operations of the underlying virtual machine. Interface design. Our system (unlike most statistics-based classification systems) is designed (1) to support an interactive give-and-take with the user, and (2) to generate as o u t p u t a line-by-line "commentary" on the input data, which represents a clear and concise presentation of extensions, classifications and case citations. 3.3.

The Role of

Parallelism

We have built a prototype parallel version of the F G P Machine, using Linda. "Running" an F G P Machine is computationally expensive to the extent t h a t it deals with large databases. Because (1) good response time 4

Briefly, the evocativeness of an attribute-value pair with respect to a goal attribute is inversely proportional to the entropy (disorder) of the distribution of values for the goal represented in the group of cases returned by fetch. See [18] for details.

416

is crucial to an interactive system of this sort, and (2) handling massive databases, the larger the better, is exactly the raison d'être of this and all related approaches, it's essential t h a t we attack this problem head-on, and try to make the system run fast even when it is driven by a very large database. Parallelism is the obvious approach. We have been successful in using asynchronous parallelism to solve a series of related problems, and are 5 confident t h a t our techniques will work well for the F G P program as well. Our parallel version runs on (in effect) any MIMD parallel environment, including local area networks of autonomous hosts. This parallel version allows us to overcome the performance limitations t h a t m a d e it impossible, until now, for us to focus our heuristics on large databases; studying the performance of our system on large databases is the crucial next step in a research project t h a t has been promising thus far. A central aspect of our approach is our focus on asynchronous parallelism. Interesting and significant results have been obtained in the use of synchronous parallelism in database-driven expert systems t h a t in some ways resemble ours (notably, the work of Waltz and Stanfill on the Connection Machine[29]). But this previous work doesn't obviate the need for (indeed, only marginally relates to) new research on asynchronous techniques. Asynchronous programs tend to be largely different from synchronous ones. They are also are exceptionally important: asynchronous or MIMD parallel machines are far more widespread today than are synchronous or SIMD platforms. T h e new generation of "parallel PCs" now in production at several large companies hints at an ever-expanding flood of cheap MIMD machines to come. More important still, virtually every significant computing site in the country now owns a MIMD computing platform—recent results from our group and others having demonstrated conclusively t h a t , for some significant applications at least, Ethernetted workstations are a highly powerful and effective parallel-computing environment. [1] We have developed the parallel F G P prototype for just such an environment. T h e computational expense of the F G P program arises mainly at two points. Processing a query requires t h a t we rate the entire database of cases for "closeness to" the query. When we handle a query interactively, we need to repeat this operation every time the user adds a new attribute to the query. Each of these closeness-computations is potentially expensive, because each must be at least linear in cost with the number of cases rated. Note t h a t this computation is independent of the retrieval method: as a first cut, we can ignore all cases t h a t fail to intersect the query on at least experimental validation was underway at the time this chapter was being prepared. Contact the authors for recent reprints.

417

one a t t r i b u t e . We use hashing to determine the candidate cases (to exclude the irrelevant ones). But, having determined the list of candidates, clearly we must examine each one. W h a t we're parallelizing, in other words, isn't an exhaustive search of the database; we're parallelizing the computation t h a t follows the initial database search. T h e F G P program imposes no requirements in terms of the uniformity of cases in the database. It can operate with a wildly heterogeneous databases. In practice, though, case databases are often fairly uniform in the attributes each case includes. W i t h a uniform database, the list of "candidate cases" relative to a query tends to approach the entire database. In tackling the problem of parallelizing a computation t h a t needs to be applied to each element in a potentially large subset of a database, we were guided by research on a similar problem t h a t we've solved successfully. Geneticists sometimes need to search large DNA sequence databases for "close matches" against a newly-discovered target sequence. T h e search requires t h a t a "closeness computation" be performed for each potentially interesting sequence in the database. We have worked with researchers from the Department of Biology and the Medical School at Yale to develop parallel search programs. These C-Linda programs have achieved good speedups relative to sequential codes on typical searches against subsets of Genbank (a large genetic sequence database) on a wide variety of parallel machines: shared memory machines (Apollo DN 10000, Encore Multimax, Sequent S y m m e t r y ) , distributed memory machines (Intel i P S C / 2 ) and LANs (Ethernet-connected Sun Sparcstations). Many versions of this program and their performance are discussed at length in [7]. A closeness comparison for genetic sequences is in m a n y cases more expensive, computationally, than the F G P program's closeness comparisons. In order to achieve good performance, particularly in looselycoupled (expensive-communication) environments such as LANs, the parallel F G P program will accordingly experiment with clumping (grouping many database records into a single unit). Currently, the prototype performs a per-program-run static mapping of the database onto available processors. Both techniques are discussed in[7]. 4.

T h e P r o c e s s Trellis

T h e Process Trellis is a software architecture for building real-iime, heuristic monitors', heterogeneous, large, real-time, parallel and continuously executing programs. T h e Process Trellis organizes a collection of heterogeneous decision processes into an acyclic hierarchical network which mirrors the structure of the domain. Each process is responsible for a semantically meaningful and logically disjoint component of the monitoring task; as long

418

as they obey the Trellis' protocol, they can incorporate any kind of logic appropriate to their part of the computation.[11] is the definitive reference. T h e Trellis imposes a simple and uniform structure on complex, heterogeneous programs; and it does so in such a way t h a t parallelism is inherent in the resulting software. But the Trellis' most i m p o r t a n t property is somewhat idiosyncratic: the Trellis represents a strategy for embodying the "intellectual structure" of a problem domain directly in software. T h e Trellis is implemented in C-Linda, and has been tested in a number of domains; our largest prototype is designed for patient monitoring in the intensive care unit. This application includes more t h a n one hundred separate concurrent, asynchronous processes, and runs effectively on both shared- and distributed-memory parallel computers[16, 8, 27]. [14] discusses 3 new, wide-ranging Trellis applications, in various stages of construction. Our experience leads us to conclude t h a t the Trellis is a promising tool not merely for medical monitoring applications, but for a wide range of problems in applied AI. 4.1.

The Trellis

Structure

The Trellis processes are arranged in a directed acyclic information-flow hierarchy. Information-flow hierarchies describe the flow of information in the analysis or characterization of a domain. In the Process Trellis, one process dominates another in the hierarchy if the lower-level process calculates information t h a t the higher-level process requires. An example will make this clear; figure 1 is a information-flow hierarchy for a generic Process Trellis application.

Fig. 1. A Generic Trellis Hierarchy At the b o t t o m of a Trellis program are raw-data processes t h a t receive inputs from the external world. Above these are data-filter processes which perform tasks such as validating d a t a and converting representations. Still

419

higher are correlation processes which look for simple correlations in the results of the data-filter level. Their o u t p u t s are used by diagnosis level processes in characterizing the domain as a whole. Finally, recommendation processes consider the results of the diagnostic and lower levels in recommending actions. Not all Trellis programs will have all of these levels and additional layers are possible, b u t this sketch captures the essence of a Trellis hierarchy. T h e hierarchy provides guidance in building a program. T h e structure of the application should mirror the structure of the domain; two processes should communicate if there is an information-dependency between the "real" world values they compute. T h e use of an information-flow hierarchy is not unique to the process Trellis. In the blackboard architecture, a collection of knowledge sources communicate through a hierarchical blackboard; each knowledge source, which is responsible for a specific sub-problem, accesses only a subset of the blackboard's levels. For example, Hearsay-II, which performed speech understanding, had blackboard levels corresponding to segments, syllables, words, word-sequences and phrases, amongst others [9]. (However, the Trellis differs from blackboard approaches in most other respects; see [ 1 1 ] ) . Every process in the Trellis hierarchy has a set of inferiors, a set of superiors and a state. T h e value of a process's state, which can be a complex object, depends upon the states of the process's inferiors and upon the state's own prior value; since a process's new state depends upon the prior value of its state, processes are history sensitive. When a process executes, it generates a new state if it has sufficient new information; each process defines for itself in an arbitrary, domaindependent manner what "sufficient new information" means. Whenever any non-empty subset of its inferiors have new states, a process is enabled and can execute. Unlike petri-net or conventional data-flow models, processes do not require inputs from all of their inferiors to be enabled; thus, processes execute with partial information. When a process generates a new state, its superiors are enabled and a t t e m p t to generate new states in turn. Since a process executes whenever any of its inferiors has generated a new state, a process "sees" every state generated by each of its inferiors. To summarize, states flow up the Trellis from inferiors to superiors. Besides passively waiting for a new state from an inferior, a process can actively query any of its inferiors. A query contains no data; it simply causes the lower-level process to a t t e m p t to generate a new state. In fact, a process may not even be able to determine t h a t it has been queried. A queried process executes and a t t e m p t s to generate a new state. If the queried process has insufficient information to generate a new state, it can in turn query its inferiors. A bottom-level process can, when queried, request inputs from a user. To summarize, queries flow down the Trellis

420

from superiors to inferiors. Besides providing a uniform inter-process communication mechanism, the Trellis provides a uniform mechanism for external-world interactions. Significantly, the user and not the program controls when this interaction occurs and with which part of the program it occurs. Probes are an application-independent mechanism for dynamically interacting with any process in a running Trellis program. There are two symmetric probes. A write probe sets the value of any portion of any process's state. A read probe reads any process's state. Probes are either active or passive. An active probe may cause the process to execute; a passive probe cannot. Probes mirror the Trellis' inter-process communication. Write probes are analogous to the upward flow of states, and read probes to the downward flow of queries. Probes have an additional benefit; they can aggregate independent, logically disjoint, and potentially physically dispersed Trellis programs into a single larger program. Processes in the different Trellises are connected via probes; each Trellis treats the other Trellises as the external world. We can use probes to connect processes in different Trellises since processes d o n ' t care where their inputs come from. We have demonstrated the effectiveness of the Trellis with the ICM, a prototype ICU (intensive care unit) monitor [16, 8, 27, 15]. T h e ICM seeks to provide early, systematic detection of evolving trends, taking advantage 6 of asynchronous inputs when available but using primarily on-line d a t a . Currently, the ICM contains over one hundred processes divided between two Trellises. T h e prototype ICM runs off-line but at real-time rates, at a frequency as high as one hundred times a second. 4.2.

Properties

of the

Trellis

T h e Process Trellis aids in the design and construction of real-time, heuristic monitors: heterogeneous, large, real-time, parallel and continuously executing programs. Since processes calculate their states in any way they choose, Trellis programs are by n a t u r e heterogeneous. The inter-process connections in a Trellis program are explicit; this leads to applications t h a t are easy to understand and modify. Modifying a module in a program with a dynamic communication p a t t e r n can indirectly affect any other program module, but the Trellis structure enables us to localize the direct and indirect effects of a program modification (see [11]). This simplifies the building and maintenance of large Trellis applications. 6

T h e ICM is joint work with Drs. Dean Sittig, Aaron Cohn, Stanley Rosenbaum and Perry Miller of the Yale School of Medicine, Department of Anesthesiology.

421

Because processes communicate using a fixed protocol, we can predict the worst-case communication and computation d e m a n d s t h a t new inputs can engender [11, 12]. Using this information, we can schedule the Process Trellis on the processors of a parallel computer, guaranteeing t h a t the program analyzes new inputs as quickly as they enter from the external world. Since Trellis programs are explicitly parallel. Each process can execute independently and concurrently with all other Trellis processes. [12] shows t h a t Trellis programs can be highly efficient. Trellis processes are history sensitive, an essential a t t r i b u t e for a continuously executing program. Finally, probes provide a simple uniform mechanism for dynamic, non-pre-planned interactions between the Trellis and its environment — an especially i m p o r t a n t a t t r i b u t e for a continuously executing program. Using these properties, we can contrast the Trellis to other software architectures; for the sake of concreteness we compare the Trellis to as pro7 duction rule architectures. Production rules are homogeneous; all "knowledge" must be represented by entities of the same type, namely production rules. T h e connections between rules, the modules of the program, is dynamic; this can lead to difficulties in maintaining large rule-bases [28]. As noted by others, e.g., [24], most rule-based systems have unacceptable performance (both in terms of performance and predictability). Finally, while there has been a great deal of work on parallelizing production rule systems [2, 23, 30], it is not yet clear whether reasonable performance gains are possible [22]. 4.3.

Parallelism

and Real-time

Scheduling

All Trellis nodes can execute concurrently; other t h a n explicit communications between processes, there are no inter-process dependencies. In addition, the Trellis, which was designed with parallelism in mind from the start, is based upon the philosophy t h a t there should be sufficient computational resources to execute all relevant processes, whenever they are relevant; thus the architecture does contain inherent bottlenecks t h a t impose an order on the execution of processes. T h u s , the Trellis architecture allows us to use parallelism to meet real-time constraints. Our goal specifically is to minimize the number of processors necessary to guarantee t h a t the program analyzes incoming d a t a within some time bound ( t h a t is, t h a t the Trellis computes the full implications of incoming data within an acceptable interval). In the simplest implementation of a 7

This is not a critique of production rule architectures, as after all production rule systems were not designed to build real-time, heuristic monitors.

422

Trellis program, each Trellis node corresponds to (is implemented by) a separately-scheduled process. But we use a different implementation, more amenable to real-time scheduling. T h e Trellis is executed in a series of sweeps: all nodes are updated for the ith time before any node is updated for the i-f 1st. We create a fixed number of identical worker processes (typically one per available processor — each worker will run full-time, essentially without blocking). We then use a scheduling heuristic to partition Trellis nodes among workers. On the ith iteration, each worker updates the Trellis nodes t h a t have been assigned to it. T h e scheduler (see [11]) presupposes an analytic model of program execution time, which it draws on to predict the behavior of any given node partitioning. We have developed such a model, and shown it to be accurate for small multiprocessors (at least). The model has been tested and remains accurate for synthetic Trellises of up to roughly 20,000 nodes (the largest synthetic programs we can run under the current implementation on available parallel machines). We discuss the model in the context of large parallel machines in [13]. 5.

C o m b i n i n g t h e Trellis a n d F G P p r o g r a m s for R e a l - T i m e D a t a Management

One should judge the power of software architectures (such as the Trellis and the F G P Machine) not only in isolation but also by the degree to which they can be easily integrated with other systems. T h e generic nature of the Trellis' probes enable the Trellis to generate o u t p u t for any system t h a t consumes a stream of d a t a and to get input from any system t h a t is a producer of such streams. T h e simplicity of the F G P Machine's abstract model and the fact t h a t it works off of a regular database make it easy theoretically to integrate the F G P Machine with any other system t h a t is a producer of record-oriented data. Since the F G P Machine's conclusions are themselves database cases, no different from other cases, it is also straightforward in principle, to use an F G P database as input to another system. This section describes a symbiotic union of the F G P Machine with the Process Trellis to produce a system capable of effective real-time d a t a acquisition and analysis. We describe this union in terms of a hypothetical super-system for network monitoring and management, because t h a t is a widely understood context t h a t illustrates the most i m p o r t a n t points. In this system the F G P Machine and the Trellis each act as both a source of d a t a for and a consumer of d a t a from the other. T h e resulting system is more powerful than the sum of its parts.

423

There are three i m p o r t a n t points to keep in mind in reading the next section: - There is nothing unique about the network monitoring domain. We have chosen it for illustration purposes and because it is a familiar domain. - T h e only part of the integration t h a t is not straightforward is the mapping from Trellis state vectors to F G P cases and back again. There is no clear "right" way to perform this mapping; a commitment to a particular mapping has consequences both for what types of d a t a can be filtered (and stored) and how efficiently d a t a can be handled. - Notwithstanding the m a p p i n g problem, we are not describing "vaporware." We already have a good software base from which to describe the proposed integration. T h e Trellis is a functioning software architecture; as we described above, it has been used to build useful systems. T h e F G P Machine works; it has been used effectively in several domains. As both programs are implemented in C-Linda, there are no structural impediments to integration. This is important in three ways. Linda processes communicate through tuple space (as described above); the component processes of the integrated program will be able to communicate through tuple space without major recoding of the computational kernels. Secondly, efficient Linda implementations exist across a wide range of parallel architectures, including networks and powerful multiprocessors. Finally, the Linda developers at S C I E N T I F I C are currently implementing Open Tuple Space (OTS) servers to enable otherwise separate programs to communicate using a global "open" tuple space. W i t h this enhancement to the Linda environment (planned release date Fall 1993), separate modules can be freely attached or detached from an executing parallel program. Besides aiding in debugging (sometimes difficult in parallel environments), O T S servers will facilitate experimentation. For example, modules implementing different filtering algorithms could be compared on the fly without recompilation. Note t h a t it is parallelism and C-Linda t h a t makes it possible for us to envision combining these architectures in a system t h a t requires real-time behavior. T h e computational requirements of the Trellis and of the F G P Machine are both high. But cycles are cheap in parallel environments. Careful coding in Linda and the Trellis scheduler can guarantee t h a t idle processors are kept busy, and t h a t efficient use is m a d e of the cycles available on any deployment platform.

424

6.

An Integrated Program for Network Monitoring

Consider an F G P Machine attached to a database containing information on a network. We assume this d a t a base is continuously updated with new cases representing the status of the network as a whole and of its various components. Clearly, this database can be used to determine static configuration information for the network or to find out its current s t a t u s . In addition, because this is an F G P Machine and not a normal database, the system can also be used to diagnose faults in a network or to suggest changes in the network configuration based upon previous cases t h a t "remind" the system of the current case. F G P daemon decision-procedures can look for problems or unusual situations. For instance, one can easily imagine a F G P query t h a t will be matched when there are a large number of unsuccessful remote login a t t e m p t s for a given machine; such a query could be used to catch an attempted security violation. T h e Process Trellis by itself is an ideal software architecture for building a network monitor. Networks of computers generate large quantities of data—disk usage, network traffic, process status, etc. This d a t a must be processed and converted into a more manageable form, e.g., "Disk usage on node Joyce has increased 10% during the past hour" or "50% of the network messages have parity errors." It then must be correlated with other data, e.g., "Disk usage on node Winston has decreased 10% during the past hour; there has been a large amount of network traffic between Winston and Joyce." Based upon these correlations a diagnosis must be reached, e.g., "J. Smith has moved a large directory from Winston to Joyce." Finally, a recommendation must be made based upon the diagnosis and other information, e.g., "Winston's small disk is 90% full, Joyce's large disk is 10% full, so consider reorganizing the physical disk configuration." We have not described any radically new function; there are many existing systems t h a t provide various pieces. However, the Process Trellis provides a uniform, extensible, clean framework for all system components. T h e process t h a t receives reports (via a permanently attached write probe) on node Winston's disk usage every 5 minutes presents exactly the same interface to the outside world as does the process t h a t decides t h a t the sudden increase in Winston's disk usage does not indicate a problem, since there was a large directory moved from another system. This uniformity of structure, along with the other desirable properties of the Trellis described previously, make the Trellis architecture a perfect match for building a network monitor. T h e real gain in power comes when we consider integrating these two distinct systems. There are several ways the Trellis and the F G P Machine network monitors can be combined.

425

The FGP database can be used to store the configuration of the network. T h e Trellis network monitor can obtain this information from the F G P database by means of an F G P query which always looks for changes to the network configuration and pipes its results via a write probe into the relevant Trellis processes. The Trellis network monitor can be used as a source of data for the FGP database. For the F G P network monitor to be useful, it must have a large reservoir of past cases from which to draw. Read probes permanently attached to interesting Trellis processes can store all of the Trellis processes' conclusions in the database. The Trellis network monitor diagnostic processes can use the FGP database for diagnostic assistance. T h e Trellis process t h a t make diagnoses are black-boxes to the Trellis architecture. As long as the process obeys the Trellis' interface specification, the process can use any form of internal logic. In particular, it is easy to conceive of a Trellis diagnostic process, t h a t has as its inferiors a set of correlation processes. Based upon the states it receives from the correlation processes it generates an F G P query looking for a similar case from the past. Some of the d a t a handled by this hypothetical system must be processed in real-time. While not life-threatening, a rapidly filling disk, or a sick node flooding a network with r a n d o m packets, can quickly cause a computer network to freeze. We could ensure t h a t such a program would run in realtime because: (1) T h e network environment is inherently parallel implying abundant C P U resources, and (2) We can write Trellis and F G P programs in Linda to run efficiently (and in real-time) on networks.

There are several reasons t h a t it is possible for us to envision hooking an F G P Machine to a Process Trellis. - Both architectures have simple interfaces for interacting with the external world. T h e Trellis uses probes and the F G P Machine a database. Many other software architectures for AI tend to ignore interactions with the external world. - Both architectures are amenable to parallelism. T h e Trellis was built as a parallel architecture from the start and a parallel, Linda-based prototype of the F G P Machine now exists. T h e computational requirements of the Trellis and of the F G P Machine are high. A driving principle is t h a t it is better to spend a few cycles looking for an

426

event t h a t is rarely true t h a n it is to save the cycles and fail to notice the event when it is true. These computational requirements can be easily met, but only in a parallel system. - Both systems are implemented using the same coordination language, Linda. When we combine a Trellis program with an F G P Machine we need to coordinate two systems. This task will be simpler if the coordination between the processes of the individuals system can be done using the same mechanisms as the coordination between the systems.

7.

Conclusions

Parallel programming is now a proven and powerful tool, ready to expand its sphere of influence by moving into less traditional but soon to be highly important domains—like database-driven expert systems and realtime heuristic monitors. Applied AI (for its p a r t ) now generally acknowledges the desirability of manipulating massive quantities of information— which is, of course, a computationally expensive proposition. A courtship between parallelism and AI has been predicted for decades, and is now (finally) underway. T h e time is exactly right, from the standpoint of both these technologies, to push the courtship further. We believe t h a t Linda, and Linda-based software architectures such as the Process Trellis and the F G P Machine, will play matchmaking roles.

References [l] Mauricio Arango, Donald Berndt, Nicholas Carriero, David Gelernter, and Douglas Gilmore. Adventures with network linda. Supercomputing Review, 10:42-46, 1990. [2] Guy E. Belooch. CIS: A massively concurrent rule-based system. In Proceedings AAAI-86, Fifth National Conference on Artificial Intelligence, pages 735-741. American Association for Artificial Intelligence, August 1986. [3] R. Bjornson, N. Carriero, D. Gelernter, T. Mattson, and A. Sherman. Experience with Linda: Notes from the bandwagon. Reserach report, Yale University Department of Computer Science, August 1991. [4] Nicholas Carriero. Implementing Tuple Space Machines. PhD thesis, Yale University Department of Computer Science, New Haven, Connecticut, 1987. Department of Computer Science. [5] Nicholas Carriero and David Gelernter. Applications experience with Linda. In Proceedings of the ACM Symposium on Parallel Programming, pages 173187, July 1988. [6] Nicholas Carriero and David Gelernter. How to Write Parallel Programs: A Guide to the Perplexed. ACM Computing Surveys, 21(3), September 1989. Extended version published by MIT Press, Fall 1990.

427

[7] Nicholas Carriero and David Gelernter. How to Write Parallel Programs: A first course. MIT Press, Cambridge, 1990. [8] Aaron I. Cohn, Stanley Rosenbaum, Michael Factor, and Perry L. Miller. DYNASCENE: An approach to computer-based intelligent cardiovascular monitoring using sequential clinical 'Scenes'. Methods of Information in Medicine, 29:122-131, April 1990. Revised version of paper in SCAMC-89. [9] Lee D. Erman, Fredrick H ayes-Roth, Victor R. Lesser, and D. Raj Reddy. The Hearsay-II speech-understanding system: Integrating knowledge to resolve uncertainty. Computing Surveys, 12(2):213-253, June 1980. [10] Lee D. Erman, Jay S. Lark, and Frederick Hayes-Roth. ABE: An environment for engineering intelligent systems. IEEE Transactions on Software Engineering, 14(12):1758-1770, December 1988. [11] Michael Factor. The Process Trellis Software Architecture for Parallel, RealTime Monitors. PhD thesis, Yale University, Department of Computer Science, December 1990. New Haven, CT. [12] Michael Factor. The process trellis software architecture for real-time monitors. In Second ACM SIGPLAN Symposium on Principles & Practice of Parallel Programming (PPoPP), pages 147-155. ACM, SIGPLAN, March 1990. Seattle, WA. (SIGPLAN Notices, 25(3)). [13] Michael Factor and David H. Gelernter. True bigness. Research Report YALEU/DCS/RR-821, Yale University, Department of Computer Science, September 1990. [14] Michael Factor and David H. Gelernter. Case studies in applying the process trellis. Internal research report, Yale University, Department of Computer Science, June 1991. [15] Michael Factor, David H. Gelernter, Craig Kolb, Perry L. Miller, and Dean F. Sittig. Real-time performance, parallelism and program visualization in medical monitoring. IEEE Computer, March 1991. To appear: Revised version of Yale Univeristy Research Report YALEU/DCS/RR-808. [16] Michael Factor, Dean F. Sittig, Aaron I. Cohn, David H. Gelernter, Perry L. Miller, and Stanley Rosenbaum. A parallel software architecture for building intelligent medical monitors. International Journal of Clinical Monitoring and Computing, 7:117-128, 1990. Revised version of paper in SCAMC-89. [17] Scott J. Fertig. FGP: Using statistics to drive an expert database. In Proc. of Interface 91: Computing Science and Statistics, 1991. [18] Scott J. Fertig and David H. Gelernter. FGP: A software architecture for acquiring knowledge from cases. In Proc. of the International Joint Conference on Artificial Intelligence, August 1991. [19] R. Fisher. The use of multiple measurements in taxonomic problems. Annual Eugenics, 7:179-188, 1936. [20] O. Friesen and F. Golshani. Databases in large AI systems. AI Magazine, X(4):17-19, Winter 1989. [21] R. Guha and Douglas. Lenat. Cyc: A mid-term report. AI Magazine, XI(3):32-59, Fall 1990. [22] Anoop Gupta, Charles Forgy, and Allen Newell. High-speed implementations

428

[23]

[24]

[25] [26]

[27]

[28]

[29] [30]

[31]

of rule based systems. ACM Transactions on Computer Systems, 7(2):119146, June 1989. Wilson Harvey, Dirk Kalp, Miliand Tambe, David McKeown, and Allen Newell. The effectiveness of task-level parallelism for high-level vision. In Second ACM SIGPLAN Symposium on Principles & Practice of Parallel Programming (PPoPP), pages 156-167. ACM, SIGPLAN, March 1990. Thomas J. Laffey, Preson A. Cox, James L. Schmidt, Simon M. Kao, and Jackson Y. Read. Real-time knowledge-based systems. Artificial Intelligence Magazine, 9(l):27-45, Spring 1988. G. Piatetsky-Shapiro and W. Frawley, editors. IJCAI-89 Workshop on Knowledge Discovery in Databases, 1989. Geoffrey Rutledge, George Thomsen, Ingo Beinlich, Brad Farr, Lewis Sheiner, and Lawrence Fagan. Combining qualitative and quantitative copmutation in a ventilator therapy planner. In Proceedings of SCAMC-89, pages 315-319. Symposium for Computer Applications in Medical Care, November 1989. Washington, DC. Dean F. Sittig and Michael Factor. Physiologic trend detection and artifact rejection: A parallel implementation of a multi-state Kaiman filtering algorithm. Computer Methods in Programs in Biomedicine, 31:1-10, 1990. Revised version of paper in SCAMC-89. Elliot Soloway, Judy Bâchant, and Keith Jensen. Assessing the maintainability of XCON-in-RIME: Coping with the problems of a VERY large rulebase. In Proceedings AAAI-87, Sixth National Conference on Artificial Intelligence, pages 824-829. American Association for Artificial Intelligence, July 1987. C. Stanfill and D. Waltz. Toward memory-based reasoning. Communications of the ACM, 29(12):1213-1228, December 1986. Milind Tambe, Dirk Kalp, Anoop Gupta, Charles Forgy, Brian Milnes, and Allen Newell. Soar/PSM-E: Investigating match parallelism in a learning production system. In Proceedings of the ACM Symposium on Parallel Programming (PPEALS), pages 146-160. ACM, SIGPLAN, July 1988. New Haven, CT. (SIGPLAN Notices, 23(9)). M. Lattimer Wright, Milton W. Green, Gudrun Fiefl, and Perry F. Cross. An expert system for real-time control. IEEE Software, 3(2): 16-24, March 1986.