Visual Information and Information Systems: 8th International Conference, VISUAL 2005, Amsterdam, The Netherlands, July 5, 2005, Revised Selected Papers (Lecture Notes in Computer Science, 3736) 3540304886, 9783540304883

Table of contents :
Frontmatter
Unsupervised Color Film Restoration Using Adaptive Color Equalization
Grey-Scale Image Colorization by Local Correlation Based Optimization Algorithm
Face Recognition Using Modular Bilinear Discriminant Analysis
Computer Vision Architecture for Real-Time Face and Hand Detection and Tracking
Video Spatio-temporal Signatures Using Polynomial Transforms
Motion Trajectory Clustering for Video Retrieval Using Spatio-temporal Approximations
Interactive Animation to Visually Explore Time Series of Satellite Imagery
An OpenGIS{\textregistered}-Based Approach to Define Continuous Field Data Within a Visual Environment
Wayfinding Choreme Maps
An Approach to Perceptual Shape Matching
Interactive Volume Visualization Techniques for Subsurface Data
Compressed Domain Image Retrieval Using JPEG2000 and Gaussian Mixture Models
Indexing and Retrieving Oil Paintings Using Style Information
Semi-automatic Feature-Adaptive Relevance Feedback (SA-FR-RF) for Content-Based Image Retrieval
A Visual Query Language for Uncertain Spatial and Temporal Data
Surveying the Reality of Semantic Image Retrieval
Too Much or Too Little: Visual Considerations of Public Engagement Tools in Environment Impact Assessments
Active Landmarks in Indoor Environments
Image Annotation for Adaptive Enhancement of Uncalibrated Color Images
Automatic Redeye Removal for Smart Enhancement of Photos of Unknown Origin
Analysis of Multiresolution Representations for Compression and Local Description of Images
Presenting a Large Urban Area in a Virtual Maquette: An Integrated 3D Model with a `Tangible User Interface'
Perceptual Image Retrieval
Analyzing Shortest and Fastest Paths with GIS and Determining Algorithm Running Time
Multimodal Data Fusion for Video Scene Segmentation
Backmatter

Citation preview

Lecture Notes in Computer Science Commenced Publication in 1973 Founding and Former Series Editors: Gerhard Goos, Juris Hartmanis, and Jan van Leeuwen

Editorial Board David Hutchison Lancaster University, UK Takeo Kanade Carnegie Mellon University, Pittsburgh, PA, USA Josef Kittler University of Surrey, Guildford, UK Jon M. Kleinberg Cornell University, Ithaca, NY, USA Friedemann Mattern ETH Zurich, Switzerland John C. Mitchell Stanford University, CA, USA Moni Naor Weizmann Institute of Science, Rehovot, Israel Oscar Nierstrasz University of Bern, Switzerland C. Pandu Rangan Indian Institute of Technology, Madras, India Bernhard Steffen University of Dortmund, Germany Madhu Sudan Massachusetts Institute of Technology, MA, USA Demetri Terzopoulos New York University, NY, USA Doug Tygar University of California, Berkeley, CA, USA Moshe Y. Vardi Rice University, Houston, TX, USA Gerhard Weikum Max-Planck Institute of Computer Science, Saarbruecken, Germany

3736

Stéphane Bres Robert Laurini (Eds.)

Visual Information and Information Systems 8th International Conference, VISUAL 2005 Amsterdam, The Netherlands, July 5, 2005 Revised Selected Papers

13

Volume Editors Stéphane Bres Robert Laurini LIRIS - INSA de Lyon Bât. Jules Verne, 17 av. Jean Capelle 69621 Villeurbanne, France E-mail: {stephane.bres,robert.laurini}@insa-lyon.fr

Library of Congress Control Number: 2005937928 CR Subject Classification (1998): I.4, I.5, I.3, H.3, H.5, H.2 LNCS Sublibrary: SL 6 – Image Processing, Computer Vision, Pattern Recognition, and Graphics ISSN ISBN-10 ISBN-13

0302-9743 3-540-30488-6 Springer Berlin Heidelberg New York 978-3-540-30488-3 Springer Berlin Heidelberg New York

This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, re-use of illustrations, recitation, broadcasting, reproduction on microfilms or in any other way, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer. Violations are liable to prosecution under the German Copyright Law. Springer is a part of Springer Science+Business Media springer.com © Springer-Verlag Berlin Heidelberg 2006 Printed in Germany Typesetting: Camera-ready by author, data conversion by Scientific Publishing Services, Chennai, India Printed on acid-free paper SPIN: 11590064 06/3142 543210

Preface

Visual Information Systems on the Move

Following the success of previous International Conferences of VISual Information Systems held in Melbourne, San Diego, Amsterdam, Lyon, Taiwan, Miami, and San Francisco, the 8th International Conference on VISual Information Systems held in Amsterdam dealt with a variety of aspects, from visual systems of multimedia information, to systems of visual information such as image databases. Handling of visual information is boosted by the rapid increase of hardware and Internet capabilities. Now, advances in sensors have turned all kinds of information into digital form. Technology for visual information systems is more urgently needed than ever before. What is needed are new computational methods to index, compress, retrieve and discover pictorial information, new algorithms for the archival of and access to very large amounts of digital images and videos, and new systems with friendly visual interfaces. Visual information processing, features extraction and aggregation at semantic level and content-based retrieval, and the study of user intention in query processing will continue to be areas of great interest. As digital content becomes widespread, issues of delivery and consumption of multimedia content were also topics of this workshop. Be on the move…

June 2005

Stéphane Bres Robert Laurini

VISual2005 Conference Organization

Conference Chair Robert Laurini, INSA of Lyon, France

Program Chair Robert Laurini, INSA of Lyon, France

Program Committee Josef Bigun, University of Halmstad, Sweden Stéphane Bres, INSA de Lyon, France Sebastien Caquard, Carleton University, Canada S.K. Chang, University of Pittsburgh, USA Alberto Del Bimbo, University of Florence, Italy David Forsyth, University of California at Berkeley, USA Borko Fuhrt, Florida Atlantic University, USA Theo Gevers, University of Amsterdam, The Netherlands William Grosky, University of Michigan, USA Jesse Jin, Newcastle University, Australia Inald Lagendijk, Delft University of Technology, The Netherlands Denis Laurendeau, Laval University, Canada Clement Leung, Victoria University of Technology, Australia Michael Lew, University of Leiden, The Netherlands Chang-Tien Lu, Virginia Institute of Technology, USA Simon Lucas, University of Essex, UK Keith Nesbitt, Charles Stuart University, Australia Fernando Pereira, Institute of Telecommunications, Portugal Catherine Plaisant, University of Maryland, USA Simone Santini, University of California, San Diego, USA Raimondo Schettini, University of Milano-Bicocca, Italy Timothy Shih, Tamkang University, Taiwan David Sol, University of the Americas at Puebla, Mexico H. Lilian Tang, University of Surrey, UK Karl Tombre, LORIA, France Giuliana Vitiello, University of Salerno, Italy Marcel Worring, University of Amsterdam, The Netherlands

VIII

Organization

Organizing Committee Stéphane Bres, INSA de Lyon, France Theo Gevers, University of Amsterdam, The Netherlands Nies Huijsmans, University of Amsterdam, The Netherlands Robert Laurini, INSA of Lyon, France

Table of Contents Unsupervised Color Film Restoration Using Adaptive Color Equalization A. Rizzi, C. Gatta, C. Slanzi, G. Ciocca, R. Schettini . . . . . . . . . . . . . .

1

Grey-Scale Image Colorization by Local Correlation Based Optimization Algorithm Dongdong Nie, Lizhuang Ma, Shuangjiu Xiao, XueZhong Xiao . . . . . . .

13

Face Recognition Using Modular Bilinear Discriminant Analysis Muriel Visani, Christophe Garcia, Jean-Michel Jolion . . . . . . . . . . . . . .

24

Computer Vision Architecture for Real-Time Face and Hand Detection and Tracking D. Gonz´ alez-Ortega, F. J. D´ıaz-Pernas, J. F. D´ıez-Higuera, M. Mart´ınez-Zarzuela, D. Boto-Giralda . . . . . . . . . . . . . . . . . . . . . . . . . . .

35

Video Spatio-temporal Signatures Using Polynomial Transforms Carlos Joel Rivero-Moreno, St´ephane Bres . . . . . . . . . . . . . . . . . . . . . . . .

50

Motion Trajectory Clustering for Video Retrieval Using Spatio-temporal Approximations Shehzad Khalid, Andrew Naftel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

60

Interactive Animation to Visually Explore Time Series of Satellite Imagery Connie A. Blok . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

71

R -Based Approach to Define Continuous Field Data An OpenGIS Within a Visual Environment Luca Paolino, Monica Sebillo, Genoveffa Tortora, Giuliana Vitiello . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

83

Wayfinding Choreme Maps Alexander Klippel, Kai-Florian Richter, Stefan Hansen . . . . . . . . . . . . .

94

An Approach to Perceptual Shape Matching Xiaoyi Jiang, Sergej Lewin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Interactive Volume Visualization Techniques for Subsurface Data Timo Ropinski, Klaus Hinrichs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121

X

Table of Contents

Compressed Domain Image Retrieval Using JPEG2000 and Gaussian Mixture Models Alexandra Teynor, Wolfgang M¨ uller, Wolfgang Kowarschick . . . . . . . . . 132 Indexing and Retrieving Oil Paintings Using Style Information Yan Yan, Jesse S. Jin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Semi-automatic Feature-Adaptive Relevance Feedback (SA-FR-RF) for Content-Based Image Retrieval Anelia Grigorova, Francesco G.B. De Natale . . . . . . . . . . . . . . . . . . . . . . 153 A Visual Query Language for Uncertain Spatial and Temporal Data Karin Silvervarg, Erland Jungert . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Surveying the Reality of Semantic Image Retrieval Peter G.B. Enser, Christine J. Sandom, Paul H. Lewis . . . . . . . . . . . . . 177 Too Much or Too Little: Visual Considerations of Public Engagement Tools in Environment Impact Assessments Ann Shuk-Han Mak, Poh-Chin Lai, Richard Kim-Hung Kwong, Sharon Tsui-Shan Leung . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 Active Landmarks in Indoor Environments Beatrix Brunner-Friedrich, Verena Radoczky . . . . . . . . . . . . . . . . . . . . . . 203 Image Annotation for Adaptive Enhancement of Uncalibrated Color Images Claudio Cusano, Francesca Gasparini, Raimondo Schettini . . . . . . . . . . 216 Automatic Redeye Removal for Smart Enhancement of Photos of Unknown Origin Francesca Gasparini, Raimondo Schettini . . . . . . . . . . . . . . . . . . . . . . . . . 226 Analysis of Multiresolution Representations for Compression and Local Description of Images Fran¸cois Tonnin, Patrick Gros, Christine Guillemot . . . . . . . . . . . . . . . . 234 Presenting a Large Urban Area in a Virtual Maquette: An Integrated 3D Model with a ‘Tangible User Interface’ Irene Pleizier, Evert Meijer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 Perceptual Image Retrieval Noureddine Abbadeni . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259

Table of Contents

XI

Analyzing Shortest and Fastest Paths with GIS and Determining Algorithm Running Time Turan Erden, Mehmet Zeki Coskun . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 Multimodal Data Fusion for Video Scene Segmentation Vyacheslav Parshin, Aliaksandr Paradzinets, Liming Chen . . . . . . . . . . 279 Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291

Unsupervised Color Film Restoration Using Adaptive Color Equalization A. Rizzi1, C. Gatta1, C. Slanzi1, G. Ciocca2, and R. Schettini2 1

Dipartimento di Tecnologie dell’Informazione, Università degli studi di Milano - Polo di Crema, via Bramante, 65, 26013 Crema (CR), Italy {rizzi,gatta}@dti.unimi.it, [email protected] 2 DISCo (Dipartimento di Informatica, Sistemistica e Comunicazione), Edificio 7, Università degli Studi di Milano-Bicocca, Via Bicocca degli Arcimboldi 8, 20126 Milano, Italy {ciocca, schettini}@disco.unimib.it

Abstract. Chemical processing of celluloid based cinematic film, becomes unstable with time, unless they are stored at low temperatures. Some defects, such as bleaching on color movies, are difficult to solve using photochemical restoration methods. In these cases, a digital restoration tool can be a very convenient solution. Unfortunately, for old movies color and dynamic range digital restoration is usually dependent on the skill of trained technicians who are able to control the parameters through color adjustment, and may be different for a sequence or group of frames. This leads to a long and frustrating restoration process. As an alternative solution, we present in this paper, an innovative technique based on a model of human color perception:, to correct color and dynamic range with no need of user supervision and with a very limited number of parameters. The method is combined with a technique that is able to split the movie into different shots and to select representative frames (key frames) from each shot. By default, key frames are used to set the color correction method parameters that are then applied to the whole shot. Due to the robustness of the color correction method the setting used for the key frame is used successfully for all the frames of the same shot.

1 Introduction Movie chemical materials are the result of a chemically unstable process, and is subject to fading with time. This fading is irreversible and in several cases photochemical restoration of faded prints is risky and not always possible. In these cases, digital color restoration can solve the problem. In this paper, we propose a technique for color digital restoration of faded movies based on a perceptual approach, inspired by some adaptation mechanisms of the human visual system (HVS), in particular, lightness constancy and color constancy. The lightness constancy adaptation enables perception of the scene regardless of changes in the mean luminance intensity and the color constancy adaptation enables perception of a scene regardless of changes in color of the illuminant. S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 1 – 12, 2005. © Springer-Verlag Berlin Heidelberg 2005

2

A. Rizzi et al.

Restoring film fading and/or color bleaching can be seen as a problem of chromatic noise removal, such as color constancy mechanisms [1][2]. Consequently an algorithm is chosen for digital images that performs unsupervised enhancement, called ACE (Automatic Color Equalization) [3][4]. It provides experimental evidence in an automatic correction of the color balance of an image. Although the number of ACE parameters is very small and their tuning not critical, their setting can vary widely according to the image content and to the kind of final rendering chosen by the film director (e.g. low or high key, artistic color distortion, etc..).

2 Towards Unsupervised Restoring Parameters Tuning To implement a standard tuning procedure, we need to extract a set of still images (key frames) that summarize the video content in a rapid and compact way. Different methods can be used to select key frames. In general these methods assume that the video has already been segmented into shots by a shot detection algorithm, and extract the key-frames from within each shot detected. One of the possible approaches to key frame selection is to choose the first frame in the shot as the key frame [5]. Ueda et al [6] and Rui et al. [7] use the first and last frames of each shot. Other approaches include time sampling of shots at regular intervals. As an alternative approach [7], the video is time sampled regardless of shot boundaries. In [8][9] the entire shot is compacted into a small number of frames, grouping consecutive frames together, or taking frames from a predefined location within the shot. Other approaches, such as [10][11], compute the differences between consecutive frames in a shot using color histograms, or other visual descriptions, to measure the visual complexity of the shot; the key frames are selected by analyzing the values obtained. In [12][13] the frames are classified in clusters, and the key frames are selected from the larger clusters, or by hierarchical clustering reduction. The drawbacks to most of these approaches is the number of representative frames must be fixed in some a priori method, for example, depending on the length of the video shots. This cannot guarantee that the selected frames will not be highly correlated. It is also difficult to set a suitable interval of time, or frames: large intervals mean a large number of frames will be chosen, while small intervals may not capture enough representative frames, or those chosen may not be in the right places. We apply here a new algorithm that dynamically selects a variable number of keyframes depending on the shot’s visual content and complexity. After the extraction of key frames, these images are used as a set for the parameter tuning of ACE, the chosen algorithm for color correction. By default the key frames are used to set the color correction method parameters, which are then applied to the whole shot. Due to the robustness of the color correction method the setting used for the key frames is used successfully for all the frames of the same shot.

3 ACE: Automatic Color Equalization ACE is an algorithm for unsupervised enhancement of digital images. It is based on a computational approach that merges the "Gray World" and "White Patch" equalization mechanisms, while taking into account the spatial distribution of color

Unsupervised Color Film Restoration Using Adaptive Color Equalization

3

information. Inspired by some adaptation mechanisms of the human visual system, ACE is able to adapt to widely varying lighting conditions, and to extract visual information from the environment efficiently. The implementation of ACE follows the scheme as shown in Fig. 1: first stage: chromatic spatial adaptation (responsible for color correction); and second stage: dynamic tone reproduction scaling, to configure the output range, and implement accurate tone mapping. No user supervision, no statistics and no data preparation are required to run the algorithm.

Fig. 1. ACE basic scheme

In Fig. 1 I is the input image, R is an intermediate result and O is the output image; subscript c denotes the chromatic channel. The first stage, the Chromatic/Spatial adaptation, produces an output image R in which every pixel is recomputed according to the image content, approximating the visual appearance of the image. Each pixel p of the output image R is computed separately for each chromatic channel c as shown in equation (1).

r (I ( p ) − I ( j ) ) d ( p, j ) j∈Im, j ≠ p R( p) = Ymax ¦ j∈Im, j ≠ p d ( p, j )

¦

Fig. 2 displays the used r( ) function.

Fig. 2. r( ) function

(1)

4

A. Rizzi et al.

The second stage maps the intermediate pixels array R into the final output image O. In this stage, a balance between gray world and white patch is added, scaling linearly the values in Rc with the following formula

Oc ( p) = round[127.5 + s c Rc ( p)]

(2)

where sc is the slope of the segment [(mc,0),(Mc,255)], with

M c = max[ Rc ( p )] p

mc = min[ Rc ( p )]

(3)

p

using Mc as white reference and the zero value in Rc as an estimate for the medium gray reference point to compute the slope sc. A more detailed description of the algorithm can be found in [3][4]. The application of ACE for movie restoration, is not a straight forward process; several aspects have been modified or introduced in order to fulfill the technical needs of the film restoration field.

4 Color Frame Restoration The principal characteristic of ACE is its local data driven color correction; ACE is able to adapt to unknown chromatic dominants, to solve the color constancy problem and to perform an image dynamic data driven stretching. Moreover, ACE algorithm is unsupervised and needs little involvement of the user. These properties make it suitable for film restoration, a problem in which usually there is no reference color to compare the results of the filtering, subjectivity is used to determine the pleasantness and naturalness of the final image. Faded movie images are dull, have poor saturation and an overall color cast. This is due to the bleaching of one or two chromatic layers of the film. Since it is necessary to deal with lost chromatic information, restoring the color of faded movies is more complex than a simple color balance. The technique, presented here, is not just an application of ACE on movie images, but an enhancement of ACE principles to meet the requirements of digital film restoration practice. ACE is used to remove possible color casts, to balance colors and to correct contrast of every single frame of the movie. This preliminary tool does not use any inter-frame correlation to improve its performance. This will be a subject for future research. In this instance, ACE parameters have to be properly tuned and new functions have been added to achieve image naturalness, to preserve the natural histogram shape and to add new functions for the restoration process. These new functions can obtain satisfactory results even though the input frame is excessively corrupted. The new functions are: • Keep Original Gray (KOG): This function is devised to relax the GW mechanism in the second stage. Instead of centering the chromatic channels around the medium gray, “keep original gray” function preserves the original

Unsupervised Color Film Restoration Using Adaptive Color Equalization

5

mean values (independently in R, G and B channels). This results in histograms more similar in shape with the original . Relaxing GW mechanism in the second stage does not affect the ACE color constancy property, achieved in the first stage. This prevents to much modification of a low or high key images. This function is also important for the fade-in and fade-out sequences. • Keep Original Color Cast (KOC):In some instances, to create a more artistic quality, film directors use unnatural color in a sequence, (e.g. “Nosferatu”, 1922 directed by F.W. Murnau1). Even though the first stage of ACE removes the color cast, an estimation can first be undertaken of the color cast and then replace back the color cast at the second stage. • Keep Original Dynamic Range (KODR): Sometimes film directors use a limited dynamic range of the film to obtain specific visual effects and low or high key pictures. In these cases, the use of KODR respects the original intention of the director. This function can also be used to manage frames that are excessively corrupted. In Summary, ACE parameters can be tuned to achieve : • • • •

The SLOPE of the r( ) function: greater the slope the more the final contrast. KOG: retains the original mean lightness of the frame. KOC: retains the original color cast. KODR: retains the original dynamic range.

All these parameters need to be set according to the characteristics of each single shot. To complete a restoration of the entire movie, a set of representative frames need to be extracted.

5 Choosing the Representative Frames The video is segmented into shots (a continuous sequence of frames taken over a short period of time) by detecting abrupt changes and fades between them, since these are more common than other editing effects. For abrupt changes, a threshold-based algorithm is implemented, coupled with a frame difference measurement computed from histograms and textures descriptors. To detect fades a modified version of the algorithm proposed by Fernando et al. [14] is implemented. The results obtained by these algorithms are submitted for evaluation to a decision module, which gives the final response. This copes with conflicting results, or with groups of frames that are not meaningful, such as those between the end of a fade-out and the start of a fade-in, by increasing the robustness of the detection phase. A gradual transition detection algorithm is currently being developed; it will be integrated in a similar manner. The key-frames selection algorithm proposed, dynamically selects the representative frames by analyzing the complexity of the events depicted in the shot. The frame difference values initially obtained are used to construct a cumulative 1

“[…] Many scenes featuring Graf Orlock were filmed during the day, and when viewed in black and white, this becomes extremely obvious. This potential blooper??is corrected when the "official" versions of the movie are tinted blue to represent night. […]” from Internet Film Database (http://www.imdb.com/title/tt0013442/).

6

A. Rizzi et al.

graph that describes how the frames’ visual content changes over the entire shot, an indication of the shot’s complexity: sharp slopes indicate significant changes in the visual content due to a moving object, camera motion or the registration of highly dynamic event. These cases are considered interesting “event points” that must be taken into account in selecting the key frames to include in the final shot summary. Event points are identified in the cumulative graph of contiguous frame differences by selecting those points at the sharpest curve of the graph (curvature points). The representative frames are those corresponding to the mid points between each pair of consecutive curvature points. In more detail: Three different descriptors are computed: a color histogram, an edge direction histogram, and wavelet statistics. The use of various visual descriptors provides a more precise representation of the frame and captures small variations between the frames in a shot. The color histogram used is composed of 64 bins determined by sampling groups of meaningful colors in the HSV color space [15]. The edge direction histogram is composed of 72 bins corresponding to intervals of 2.5 degrees. Two Sobel filters are applied to obtain the gradient of the horizontal and the vertical edges of the luminance frame image. These values are used to compute the gradient of each pixel and those pixels that exhibit a gradient over a predefined threshold are taken to compute the gradient angle and then the histogram. Multiresolution wavelet analysis can provide information about the overall texture of the image at different levels of detail. At each step in the multiresolution wavelet analysis four sub-images (or sub-bands) are obtained with the application of a low-pass filter (L) and high-pass filter (H) in the four possible combinations of LL, LH, HL and HH. These bands correspond to a smoothed version of the original image (the LL band) and the three coefficient matrices of details (the LH, HL and HH bands). We apply the multiresolution wavelet analysis on the luminance frame image, using three-step Daubechies multiresolution wavelet expansion to produce ten sub-bands. Two energy features, the mean and the variance, are computed on each sub-band, resulting in a 20-valued descriptor. To compare two frame descriptors a difference measure is used to evaluate the color histograms, wavelet statistics and edge histograms. The difference between two color histograms (dH) is based on the intersection measure. The difference between two edge direction histograms (dD) is computed using the Euclidean distance as such in the case of two wavelet statistics (dW). The three resulting values are then combined to form the final frame difference measure (dHWD) as follow:

d HWD = (d H ⋅ d W ) + (d W ⋅ d D ) + (d D ⋅ d H )

(4)

Significant differences are obtained only if at least two of the single differences exhibit high values. By weighing each difference against the other, the measure detects significant changes, ignoring the smaller differences due to the moving camera, or acquisition and compression noise. If we were to use, for example, only the color histogram, a highly dynamic frame sequence (i.e. one containing fast moving or panning effects) but with the same color contents, would result in a sequence of small frame difference values. On the contrary, if the frame sequence contained a flash, or some color effects, the corresponding frame difference value would be greater than the real contents of the sequences call for. With the use of three descriptors, only if the changes in the sequence are significant in terms of color, texture and overall edges, will dhwd result in high values.

Unsupervised Color Film Restoration Using Adaptive Color Equalization

7

The algorithm proposed by Chetverikov et al. [16] is used to detect the high curvature points. The algorithm defines as a “corner” a location where a triangle of specified size and opening angle can be inscribed in a curve. Using each curve point P as a fixed vertex point, the algorithm tries to inscribe a triangle in the curve, and then determines the opening angle in correspondence of P. Different triangles are considered using points that fall within a window of a given size centered in P, and the sharpest angle, under a predefined threshold θmax, is retained as a possible high curvature point. Finally, those points in the set of candidates high curvature points that are sharper than their neighbors (within a certain distance) are classified as high curvature points. The algorithm does not require processing the whole video, unlike some earlier methods that extract, for example, key frames based on the length of the shots. Another advantage of our algorithm is that it can be easily adapted in order to extract the key frames on-the-fly: to detect a high curvature point we can limit our analysis to a fixed number of frame differences within a predefined window. Thus the curvature points can be determined while computing the frame differences, and the key frames can be extracted as soon as a second high curvature point has been detected. It must be noted that the first and last frames of the shot are implicitly assumed to correspond to high curvature points. If a shot does not present a dynamic behavior, i.e. the frames within the shot are highly correlated, the graph does not show evident curvature points, signify that the shot can be summarized by a single representative frame: The middle frame in the sequence is chosen as the key frame.

6 Experimental Results In this section results are presented with comments regarding the ACE parameter tuning. All the pictures are key frames selected with the previously described method. We also present results on different key frames of the movie “La ciudad en la playa” (directed by Ferruccio Musitelli) an Uruguayan short movie of the 60’s. Color Cast Removal: Fig. 3a shows a frame from “La ciudad en la playa” with a bluish color cast. Fig. 3b shows the ability of ACE filtering to remove the color cast without a priori information. ACE eliminates the bluish color cast restoring the color of the wall inside the window.

Fig. 3. ACE removal of the color cast. a) Original frame from La ciudad en la playa. b) Frame filtered with SLOPE=2.5.

8

A. Rizzi et al.

Controlling the Contrast Tuning the Slope: Fig. 4 shows the effect of ACE filtering with different values of SLOPE. Fig. 4a shows a frame without a strong color casts. Fig. from 4b to 4d show ACE filtering results of Fig. 4a, with increasing SLOPE value (1, 2, 2.5). As it can be notice the contrast increase as the SLOPE increase.

Fig. 4. Effect of ACE filtering. a) Original frame from La ciudad en la playa. b) Frame filtered with SLOPE=2.

c)

d)

Fig. 4. (cont.) Effect of ACE filtering. c) Frame filtered with SLOPE=2.5. d) Frame filtered with SLOPE=5.

Fig. 5. Behavior of the KOC function. a) Original frame from Nosferatu. b) Frame filtered with SLOPE=10 and KOC.

Unsupervised Color Film Restoration Using Adaptive Color Equalization

9

Keeping Selected Properties of the Input Image: Fig. 5 shows the behavior of the KOC function. The algorithm changes contrast and mean lightness keeping its color cast. Fig. 6 shows the behavior of the KOG and the KODR functions. Fig. 6b shows the filtering without any functions. While KOG (Fig. 6c) keeps the same mean lightness, KODR (Fig. 6d) keeps the dynamic range and thus the reddish color cast. Finally, Fig. 7 shows the application of ACE on a black and white film in order to restore its original dynamic range. The example is from TOM TIGHT ET DUM DUM by Georges Méliès (1903).

Fig. 6. Behavior of the KOG and the KODR functions. a) Original frame from La ciudad en la playa. b) Frame filtered with SLOPE=3.3. c) Frame filtered with SLOPE=3.3 and KOG. d) Frame filtered with SLOPE=3.3 and KODR.

Fig. 7. Application of ACE on a black and white film. a) Original frame from “Tom tight et dum dum”. b) : Frame filtered with SLOPE=20.

10

A. Rizzi et al.

More Examples

Fig. 8. Examples of key frames filtering. On the left (a, c, e) the original key frames from La ciudad en la playa. On the right (b, d, f) the filtered key frames.

7 Conclusions This paper has presented a technique for digital color and dynamic range restoration of faded movies combining a method that identifies different shots and automatically selects key frames from a movie. The color and dynamic range restoration uses an unsupervised color equalization algorithm, based on a perceptual approach. To meet

Unsupervised Color Film Restoration Using Adaptive Color Equalization

11

the requirements in the digital restoration field new functions have been added and results are satisfactory and suggest potential research. The objective for future research will consider the problem of ACE parameter fine tuning, by investigating improvements to the speed of film processing and optimizing and accelerating the algorithm.

Aknowledgments The authors want to thank Ferruccio Musitelli for the permission to use the film and his precious and encouraging friendship.

References [1] M. Chambah, B. Besserer, P. Courtellemont, “Recent Progress in Automatic Digital Restoration of Color Motion Pictures”, SPIE Electronic Imaging 2002, San Jose, CA, USA, January 2002, vol. 4663, pp. 98-109. [2] M. Chambah, B. Besserer, P. Courtellemont, “Latest Results in Digital Color Film Restoration”, Machine Graphics and Vision (MG&V) Journal, Vol. 11 no. 2/3, 2002. [3] A. Rizzi, C. Gatta, D. Marini, “A New Algorithm for Unsupervised Global and Local Color Correction”, Pattern Recognition Letters, Vol 24 (11), pp. 1663-1677, July 2003. [4] A. Rizzi, C. Gatta, D. Marini, “From Retinex to Automatic Color Equalization: issues in developing a new algorithm for unsupervised color equalization”, Journal of Electronic Imaging, Vol 13 (1), pp. 75-84, January 2004. [5] Y. Tonomura, A. Akutsu, K. Otsugi, and T. Sadakata, VideoMAP and VideoSpaceIcon: Tools for automatizing video content, Proc. ACM INTERCHI ’93 Conference, pp.131141, 1993. [6] H. Ueda, T. Miyatake, and S. Yoshizawa, An interactive natural-motion-picture dedicated multimedia authoring system, Proc. ACM CHI ’91 Conference, pp. 343-350, 1991. [7] Y. Rui, T. S. Huang and S. Mehrotra, Exploring Video Structure Beyond the Shots, appeared in Proc. of IEEE Int. Conf. on Multimedia Computing and Systems (ICMCS), Texas USA, 1998. [8] F. Arman, A. Hsu and M.Y. Chiu, Image Processing on Compressed Data for Large Video Databases, Proc. ACM Multimedia '93, Annaheim, CA, pp. 267-272, 1993. [9] A. Pentland, R. Picard, G. Davenport and K. Haase, Video and Image Semantics: Advanced Tools for Telecommunications, IEEE MultiMedia 1(2), pp. 73-75, 1994. [10] S. Han, K. Yoon, and I. Kweon, A New Technique for Shot Detection and Key Frames Selection in Histogram Space. Proc. 12th Workshop on Image Processing and Image Understanding, 2000. [11] A. Hanjalic, R. Lagendijk, and J. Biemond, A New Method for Key Frame based Video Content Representation. Proc. First International Workshop on Image Databases and Multimedia Search, 1996. [12] Zhuang Y., Rui Y., Huang T.S., Mehrotra S.: Key Frame Extraction Using Unsupervised Clustering, in Proc. of ICIP’98, Chicago, USA, 1998. [13] A. Girgensohn, J. Boreczky, Time-Constrained Keyframe Selection Technique, Multimedia Tools and Application, vol. 11, pp. 347-358, 2000.

12

A. Rizzi et al.

[14] A. C. Fernando, C. N. Canaharajah, D. R. Bull, Fade-In and Fade-Out Detection in Video Sequences Using Histograms, Proc. ISCAS 2000 – IEEE International Symposium on Circuits and System, IV 709-712, May 28-31, Geneva, Switzerland, 2000. [15] G. Ciocca, I. Gagliardi, R. Schettini, Quicklook2: An Integrated Multimedia System, International Journal of Visual Languages and Computing, Special issue on Querying Multiple Data Sources, Vol. 12, pp. 81-103, 2001. [16] D. Chetverikov and Zs. Szabo, A Simple and Efficient Algorithm for Detection of High Curvature Points in Planar Curves, Proc. 23rd Workshop of the Austrian Pattern Recognition Group, pp.175-184, 1999.

Grey-Scale Image Colorization by Local Correlation Based Optimization Algorithm Dongdong Nie, Lizhuang Ma, Shuangjiu Xiao, and XueZhong Xiao Computer Science & Engineering Department, Shanghai Jiao Tong University [email protected], [email protected], [email protected], [email protected]

Abstract. In this paper, we present a grey-scale image colorization technique by using local correlation based optimization algorithm. The core of our colorization method is to formalize the colorization problem as minimizing a quadratic cost function under some assumptions that are mainly based on the local image characters. It can be successfully applied in colorization a variety of grey-scale images. In our colorization method, users only need to freely scribble the desired color in the input grey-scale image, which is a great improvement upon the traditional manual colorization techniques. By introducing new local connectivity factor and distance factor, our approach can effectively alleviate the color diffuseness in different regions, which is one of main problem in previous colorization methods. Additionally, by exploiting subsampling in YUV space, we accelerate the colorization process with nearly the same good results. Experiments show that better colorization results can be obtained faster with our method. Keywords: Colorization, Local Correlation, Boundary Extraction, Connectivity Detection, Least Square Solver, Optimization, Subsample.

1 Introduction In our daily life, there are many valuable photos retaining the memory that we cannot write in words. Many of them are recorded in black-and-white, especially for those old photos. Traditionally, to colorize them is a very time-consuming handmade work. Fortunately, the digital image processing technology results in significantly progress on image colorization, and allows us to color the black-and-white world with little interaction. Colorization is a term introduced by Wilson Markle in 1970 to describe the computer-assisted process he invented for adding color to black and white movies or TV programs [1]. The term is now generically used to describe any technique for adding color to grey-scale image and videos. That is, colorization is to assign true color (RGB) pixel values to a grey-scale image in which only one dimension (luminance or intensity) value is determinate. The problem has no inherently “correct” solution, since different RGB color can be get from the same determinate intensity value if various hue S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 13 – 23, 2005. © Springer-Verlag Berlin Heidelberg 2005

14

D. Nie et al.

or saturation value are used. The colorization process usually needs lots of human interaction due to the ambiguity. In this paper, we describe an interactive colorization technique by local correlation based optimization algorithm. To overcome the ambiguity in colorization process, we use the colors user scribbled in certain regions as initial conditions, and propose some assumptions mainly based on the local characters suitably to various grey-scale images. Under these assumptions, we succeed in turning the under-constrained colorization problem to a minimizing problem of a quadratic cost function, which can be solved using traditional least square method easily. Moreover, by taking the advantage of the characters of psychology, we can effectively accelerate the colorization process by subsampling with nearly the same good results. Experimental results show that our colorization method is faster and more effective than the traditional colorization methods. The flow chart of our colorization system is shown in figure 1. It is summarized as the following steps: − − − − −

User scribbles the desired color in the input grey-scale image; Subsample the intensity values and chromatic values respectively; Extract the boundary of the gray-scale image and detect the local connectivity; Extract the pixels of user indicated colors; Compute the weighting functions and solve the colorization problem as a minimizing problem of a quadratic cost function.

2 Previous Works Colorization has been extensively studied in the movie industry since 1970’s. Various semiautomatic analogue techniques have been used to accomplish this challenging task [2]. In 1987, Gonzalez and Woods used luminance keying to transfer color into the grey-scale image [3]. Their approach exploits the user defined look-up table in which each level of grey-scale intensity is corresponding the specified hue, saturation and brightness. This method is extensively used in some commercial post-production systems to apply color on the grey-scale image. However, luminance keying cannot resolve the problem when one wants to apply different colors at the same intensity level. Although this limitation could be overcome by simultaneously using a few luminance keys for different regions segmented manually, the process becomes very tedious. Input grey-scale image

Subsampling Extracted pixels whose color is known Boundary extractions and connectivity detection

Output colored image Compute weighting functions in each local window

Solve the colorization problem as a minimization problem of a quadratic cost function

Fig. 1. The flow chart of our colorization process

Grey Scale Image Colorization by Local Correlation

15

Shi and Malik color the grey-scale image by segmentation and color filling method [4], where the image is first segmented into regions and then the desired colors are used to fill each region. However there is a main problem, namely the existing image segmentation algorithms are usually time consuming, and often can’t segment the image into meaningful regions. An advanced automatic technique exploits local luminance distribution as textural information and transfers the color between the similar texture regions in the source color image and the target grey-scale image [5]. It is inspired by the method of color transfer between images [6] and by the idea of image analogies [7]. In this method, texture matching, which has the same properties as luminance keying, replaces the single intensity level comparing. It is surprisingly successful when the method is applied to some specific natural scenarios. But the local textural information is roughly represented by only computing the weighted average of luminance (50%) and standard deviation (50%) within a pixel’s neighborhood, which induces great error in transferring color to the grey-scale image. Sýkora et al suggest to use unsupervised image segmentation in cartoons colorization [8]. Cartoon generally includes two layers: background and foreground. The foreground layer contains homogeneous regions surrounded by outlines. The background layer is a more complicated textural image that remains static during the animation. Taking the advantage of this property, the original grey-scale image can be divided into a few regions using robust outline detector. Then regions are classified roughly into foreground or background via size threshold. However, because of the difficulty of region segmentation, the method usually cannot get ideal result for other type of images. Our concept of colorization is inspired by the work of Levin et al. in which a new interactive colorization technique is introduced. Their method is based on the simple premise that nearby pixels in space that have similar gray levels should also have similar colors [9]. It’s a surprisingly effective colorization technique that reduces the amount of input required from the user. However, because no boundary or region information is considered in colorization process, unseemly color sometimes diffuses from one region to others, and the process is also time-consuming, for a 240×360 grey-scale image, the process time is about 30~60 seconds in INTEL Pentium4 2.80G CPU and 512M memory. We develop their colorization technique by proposing a local correlation based optimization algorithm, in which boundary information is exploited so that the color spread can be restricted in one region. Furthermore, by using the character of visual psychology we accelerate the colorization process and greatly reduce the process time.

3 Colorization Algorithm Colorization can be considered as a problem that has an input of an intensity value Y(x; y) and an output of a RGB color vector C(x; y) =[r; g; b]. It is known that the intensity value and the color vector satisfy the following relationship: Y(x; y) = [0.299, 0.587, 0.114] C(x; y) .

(1)

16

D. Nie et al.

Therefore, colorization is an ill-posed problem in which two and more solutions exist, since different colors may have the same intensity value Y(x; y). Then, some constraints are required to ensure an ideal colorization result to be achieved. We use the colors that is scribbled by users in some regions of the input grey-scale image as an initial condition and propose the following assumptions that can be easily satisfied in many kinds of grey-scale images: (1) In one local region, the closer two pixels are, the more similar color value they may have; (2) In one local region, the more similar intensity that two pixels have, the more similar color value they may have; (3) For a pixel if the intensity values in its neighborhood have little variance, the color values in its neighboring pixels also have little variance, and vice versa; (4) The color values in the given image change smoothly. That is, there is little difference between the color value of a given pixel and the weighted summary of its neighboring pixels. In assumption (1) and (2), there is a common premise: “in one region”, mainly because we distinguish the correlativity of pixels in our colorization process, and penalizes the pixels in different regions by setting low correlativity between them in colorization. Under above assumptions, we can formulate the colorization problem to minimize the following objective function, which can be solved using traditional least square method.

§ Wpqe Wpqd Wpqg U (q) · ¦ ¨ ¸ q∈N ( p ) J (U ) = ¦ ¨U ( p) − ¸ e d g Wpq Wpq Wpq ¸ p ¨ ¦ q∈N ( p ) © ¹

2

(2)

where N(p) is the neighborhood of a pixel p in the inputted grey-scale image.

Wpqe , Wpqd , Wpqg are three weight functions respectively represent local connectivity factor, distance factor and intensity similarity factor based on assumptions above.

Wpqe is set by the corresponding connectivity between q and p. ­ 1, q is connected with p; Wpqe = ® ¯we , q is disconnected with p.

(3)

where we is a constant value in experience. For most grey-scale image, we set we be a positive number near zero, for instance, we=0.01. And in our work, we determine whether pixel q is connected with pixel p by using connectivity detection that is given in detail in section 3.1.

Grey Scale Image Colorization by Local Correlation

17

Wpqd is a standard Gaussian function with parameter σd. Wpqd = e − (p − q )

2

/ 2σ d2

(4)

d where σd is an experiential value. By using Wpq , we set bigger weight to the pixels that

are close to the pixel p.

Wpqg is based on the square intensity difference between the two pixels: Wpqg = e

− (I (p) − I (q))2 / 2σ 2p

(5)

where I(q), I(p) is respectively the intensity value of the input image I at coordinate p and q, and σp is the local intensity variance of the input image I in the neighborhood window of p. It gives different error tolerance for different pixel p under assumption (3). If σp is small, the difference of color values between the central pixel and the weighting summary of its neighbor pixels should be small, too, and vice versa. 3.1 Boundary Extractions and Connectivity Detection In our work, we just need a simple boundary extraction algorithm to roughly segment the pixels in one neighborhood window into different connecting local regions. Therefore, the traditional boundary extraction method, Canny edge detector [10], is adopted to extract the boundary of the inputted grey image I(x; y), and the boundary image is denoted as E(x; y), in which “1” presented the boundary pixel. The Canny method finds edges by looking for local maxima of the gradient of I, which is calculated using the derivative of a Gaussian filter. It uses two thresholds to detect strong and weak edges separately, and only the weak edges, which are connected to strong edges, are output as real edges. It is therefore less likely affected by noise than other approaches, and more likely to detect true edges in the image. Although there are many more delicate boundary extraction algorithms, we found there are little improvements to the colorization result from our experiments. After getting the boundary image E by Canny edge detector, we determine whether the pixel q in a neighborhood window is connected with the central pixel p by using the following principles: − Two pixels have the same value in the boundary image; − There is a route connecting these two pixels and all the pixels in this route have the same value as these two pixels in the boundary image. It’s demonstrated in figure 2, in which (a) is a neighborhood window of pixel p in the boundary image E, where each grid means a pixel and the grids in grey mean the boundary pixels; (b) is the result of connectivity detection, where the grids in green mean the connected region of p and the grids in blue mean the disconnected region of p.

18

D. Nie et al.

(a)

(b)

Fig. 2. Demonstration of connectivity detection in a neighbourhood window of pixel p: (a) the window in the boundary image E, in which each grid mean a pixel and the grids in grey mean the boundary; (b) the result of connectivity detection, in which the grids in green mean the region connected with p and the grids in blue mean the region disconnected with p.

3.2 Acceleration by Subsampling From psychological studies, we know that the U, V color components of human vision are less sensitive than the intensity component. The discovery has been used effectively in image compression by chroma subsampling. Here, we introduce a similar method in grey-scale image colorization problem by subsampling pixels in YUV space, which is proven effective to accelerate colorization process without ruining the colorization result.

the pixels to be subsampled

the pixels to be interpolated

Fig. 3. Demonstration of the subsample structure using in our method

The intensity component and the chromatic components are subsampled in the structure, as shown in Figure 3. The corresponding U; V values of the subsampled pixels are computed with above optimization method, and the U; V value of other pixels can be calculated easily by linear interpolation. By subsampling, we can get a faster colorization process with nearly the same good results as those without using subsampling, as shown in Figure 4, the left two images are colorization results by using subsampling acceleration, and the right two images are corresponding results by non-subsampling approach. It’s clear that they are nearly the same good in colorization quality. However the average process time for these four images is 7s, 6s, 32s, and 24s respectively. The algorithm is implemented on a desktop PC with INTEL Pentium4 2.80G and 512M memory.

Grey Scale Image Colorization by Local Correlation

(a) results by subsampling

19

(b) results by non-subsampling

Fig. 4. Compare the colorization quality by subsampling and non-subsampling. (a) are results by subsampling; (b), are results by non-subsampling. They are nearly the same good in colorization quality; however, the average colorization time for the two images in (a) is only 7s, 6s, to be compared with the time 32s, and 24s in (b).

4 Colorization Results We have tested our method on various images; the experiment results show that our local correlation based optimization algorithm can produce good results for a wide range of grey-scale images. Some experiment results are shown in the following. Firstly, we compare the colorization quality between the method of Levin et al and ours, as shown in figure 5 and figure 6, which demonstrate that our method is highly effective in preserving the visual structures of original image, especially in preventing the color diffusion between different regions. An demonstration on a hand-drawing picture is shown in figure 5, in which we draw a color picture freely at first, then turn it into a grey-scale image, and then scribbled the corresponding colors in some areas, we recolor it at last using the two methods. It is obvious that the colorization method of Levin et al. fails in retaining the consistent color

(a)

(b)

(c)

(d)

(e)

Fig. 5. An experiment demonstration on a hand-drawing picture: (a) the original image; (b) the grey-scale image; (c) the input image after user scribbling the desired color as initial condition; (d) colorization result by Levin et al. (e) colorization result by our method. There is serious color diffusion in the colorization result by Levin et al.; and the colorization result by our method is nearly the same good as the original color image, except a little difference because of the system error introduced in the color space transform.

20

D. Nie et al.

(a)

(e)

(b)

(f)

(c)

(g)

(d)

(h)

Fig. 6. An demonstration on a real black-and-white photo (taken in 70’s of last century): (a) is the original black-and-white photo; (b) is the image after user scribbled the desired color in certain regions; (c) is the result by the method of Levin et al., (d) is the result by our method, and (e) (f) (g) (h) are enlarged details for comparison between the two methods, in which (e) (g) come from (c); (f) (h) come from (d).

result with the structure of the original grey-scale image, and the colorization result by our method is nearly the same good as the original color image, except a little difference because of the computation error introduced in the color space transformation. Another demonstration on a real black-and-white photo taken in 70’s of last century is shown in figure 6. The original black-and-white photo, the image after user indicating colors and the colorization results by the method of Levin et al. and ours are respectively shown in the top row from left to right. Pay attention to the region around the right ear and that around the flowers on the coat, which are enlarged shown on the bottom row. There are unseemly color diffusion between the different regions in the colorized result by Levin et al., which are effectively weaken in the result by our method. We also compare the colorization time between the method of Levin et al. and ours. Both are implemented in matlab 6.0. The results are shown in Figure7, in which X-axis refers to the test image index and Y-axis refers to the process time. For each image, the process time here used is the average of 10 times colorization process, and is timed on a desktop PC with INTEL Pentium4 2.80G and 512MB memory. Statistically, the colorization time by the method of Levin et al. is about 3~4 times of our method. For the last few images, the colorization time by the method of Levin et al. increase greatly, mainly because the memory storage is out and the disk storage is used in their colorization process.

Grey Scale Image Colorization by Local Correlation

Colorizaion Time Comparison 200

Method of Levin et al.

Time(s)

150

Our method

100 50 0 1

2

3

4

5 6 Image Index

7

Fig. 7. Colorization time comparison

8

9

10

21

22

D. Nie et al.

(a)

(b)

(c)

(d)

Fig. 8. Some colorization results by our method. (a) original color images; (b) corresponding grey-scale images; (c) images after user scribbled colors in certain regions; (d) the final colorization results by our method.

Finally, we show some other results by our method, which are illustrated in Figures 8. To demonstrate the effectivity of our method, we give the original color image in the first column. And the corresponding grey-scale images, the images after user scribbled colors and the colorization results by our method are also shown orderly in the following column. Even though the initial condition indicated by the user are limited corresponding to the abundant colors in the original color images, there are only little difference between the original color images and the results by our colorization images.

5 Conclusions In this paper, we present a grey-scale image colorization technique by using local correlation based optimization algorithm. By introducing the new local connectivity factor and distance factor, our approach can effectively alleviate the color diffuseness in different regions. It’s an interactive colorization method, whereas the only thing user need to do is to scribble the desired colors in some regions to indicate how these regions should be colorized. The basic idea of our method is some assumptions which can be easily satisfied for many kinds of grey-scale images, which mainly including that the colors have high correlativity between pixels in one local region and low correlativity between those in

Grey Scale Image Colorization by Local Correlation

23

different regions; two near pixels having similar intensities have similar colors; and that the color values in the input image change smoothly. Under these assumptions, we succeed in turning the ambiguous colorization problem to a minimization problem, which penalize those pixels in different local regions with far distance either in coordinate space or the intensity space. Moreover, by taking the advantage of the characters of visual psychology, we can effectively accelerate the colorization process by subsampling with nearly the same good results. Experimental results demonstrate that our colorization method can be successfully applied in colorization a variety of grey-scale images with excellent results.

Acknowledgement Our work is partly supported by National Science Foundation of China (No.60403044, No.60373070) and Microsoft Research Asia: Project-2004-Image-01. Great thanks are due to Ma Qinyong who provided us his old black-and-white photos to test our method and to ZhangQiu and Tan Wuzheng for their help to improve the manuscript of this paper. We also want to acknowledge A. Levin, D. Lischinski, and Y. Weiss who open their colorization approach that is used in this paper to compare with ours.

References 1. Burns, G.: Colorization. Museum of Broadcast Communications: Encyclopedia of Television, http://www.museum.tv/archives/etv/index.html. 2. Markle, W.: The Development and Application of Colorization, SMPTE Journal (1984) 632-635. 3. Gonzalez, R. C., Woods, R. E.: Digital Image Processing. 2nd edn. Addison-Wesley Publishing, Reading, Massachusetts (1987) 4. Shi, J., Malik, J.: Normalized Cuts and Image Segmentation. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (1997) 731-737 5. Welsh, T., Ashikhmin, M., Mueller, K.: Transferring Color to Greyscale Images. In SIGGRAPH 2002 Conference Proceedings (2002) 277-280 6. Reinhard, E., Ashikhmin, M., Gooch, B., Shirley, P.: Color Transfer between Images. IEEE Transactions on Computer Graphics and Applications 21, 5, (2001) 34-41 7. Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., Salesin, D. H.: Image Analogies. In SIGGRAPH 2001 Conference Proceedings (2001) 327-340 8. Sýkora, D., Buriánek, J., Žára, J.: Segmentation of Black and White Cartoons, In Proceedings of Spring Conference on Computer Graphics (2003) 245-254 9. Levin A., Lischinski D., Weiss Y.: Colorization using Optimization. In SIGGRAPH 2004 Conference Proceedings (2004) 689-694 10. Canny, J.: A Computational Approach to Edge Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence (1986) 679-698

Face Recognition Using Modular Bilinear Discriminant Analysis Muriel Visani1,2 , Christophe Garcia1 , and Jean-Michel Jolion2 1

France Telecom R&D TECH/IRIS, 4, rue du Clos Courtel, 35512 Cesson-Sevigne, France {muriel.visani, christophe.garcia}@rd.francetelecom.com 2 Laboratoire LIRIS, INSA Lyon, 20, Avenue Albert Einstein, Villeurbanne, 69621 cedex, France [email protected]

Abstract. In this paper, we present a new approach for face recognition, named Modular Bilinear Discriminant Analysis (MBDA). In a first step, a set of experts is created, each one being trained independently on specific face regions using a new supervised technique named Bilinear Discriminant Analysis (BDA). BDA relies on the maximization of a generalized Fisher criterion based on bilinear projections of face image matrices. In a second step, the experts are combined to assign an identity with a confidence measure to each of the query faces. A series of experiments is performed in order to evaluate and compare the effectiveness of MBDA with respect to BDA and to the Modular Eigenspaces method. The experimental results indicate that MBDA is more effective than both BDA and the Modular Eigenspaces approach for face recognition.

1

Introduction

In the eigenfaces [1] (resp. fisherfaces [2]) method, the 2D face images of size h × w are first transformed into 1D image vectors of size h · w, and then a Principal Component Analysis (PCA) (resp. Linear Discriminant Analysis (LDA)) is applied to this high-dimensional image vector space, where statistical analysis is costly and may be unstable. To overcome these drawbacks, Yang et al. [3] proposed the Two Dimensional PCA (2D PCA) method that aims at performing PCA using directly the face image matrices. It has been shown that 2D PCA is more effective [3] and robust [4] than the eigenfaces when dealing with face segmentation inaccuracies, low-quality images and partial occlusions. In [5], we proposed the Two-Dimensional-Oriented Linear Discriminant Analysis (2DoLDA) approach, that consists in applying LDA on image matrices. We have shown on various face image databases that 2DoLDA is more effective than both 2D PCA and the Fisherfaces method for face recognition, and that it is more robust to variations in lighting conditions, facial expressions and head poses. S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 24–34, 2005. c Springer-Verlag Berlin Heidelberg 2005 

Face Recognition Using Modular Bilinear Discriminant Analysis

25

The first contribution of this paper is a new supervised feature extraction method generalizing and outperforming 2DoLDA, namely Bilinear Discriminant Analysis (BDA). This method is based upon the optimization of a generalized Fisher criterion [6, 2] computed from image matrices directly, and we call it BDA because this criterion uses bilinear projections. The second contribution of this paper is a modular classification scheme combining BDA experts trained on different regions of the face, chosen as in [7, 8] and designed to be more robust to facial expression changes. The remainder of this paper is organized as follows. In section 2, we describe in details the principle and algorithm of the proposed BDA technique, pointing out its advantages over previous methods. In section 3, we present our multiple expert scheme named MBDA. Then, we provide in section 4 a series of two experiments performed on an international data set, demonstrating the effectiveness and robustness of MBDA and comparing its performances with respect to 2DoLDA and the Modular Eigenspaces method [7]. Finally, conclusions and closing remarks are drawn in section 5.

2

Bilinear Discriminant Analysis (BDA)

In this section, we describe the proposed BDA feature extraction technique. The model is constructed from a training set Ω containing n face images, with more than one view per each of the C registered persons. The set of images corresponding to one person is called a class ; class c is denoted by Ωc . Each face image is stored as a h × w matrix Xi labelled by its class. Let us consider two projection matrices Q ∈ Rh×k and P ∈ Rw×k , and the following bilinear projection: XiQ,P = QT Xi P

(1)

where the matrix XiQ,P , of size k × k, is considered as the signature of the face Xi . We are searching for the optimal pair of matrices (Q∗ , P ∗ ) maximizing the separation between signatures from different classes while minimizing the separation between signatures from the same class. As a consequence, we can consider the following generalized Fisher criterion: (Q∗ , P ∗ ) =

Argmax (Q,P )∈Rh×k ×Rw×k

=

Argmax

|SbQ,P |

(2)

Q,P |Sw |

(Q,P )∈Rh×k ×Rw×k |

|

C c=1 C c=1

nc (XcQ,P −X Q,P )T (XcQ,P −X Q,P )| Q,P −XcQ,P )T (XiQ,P −XcQ,P )| i∈Ωc (Xi

(3)

Q,P where Sw and SbQ,P are respectively the within-class and between-class covariance matrices of the set (XiQ,P )i∈{1,...,n} of the projected samples from Ω, X Q,P

and XcQ,P are the mean face matrices calculated respectively over Ω and Ωc . The objective function given in equation (3) is biquadratic and has no analytical solution. We therefore propose an iterative procedure that we call Bilinear

26

M. Visani, C. Garcia, and J.-M. Jolion

Discriminant Analysis. Let us expand the expression (3):   T T T | C c=1 nc (P (Xc −X) QQ (Xc −X)P )| (Q∗ , P ∗ ) = (4) Argmax | C (P T (X −X )T QQT (X −X )P )| c=1

(Q,P )∈Rh×k ×Rw×k

i

i∈Ωc

c

i

c

For any fixed Q ∈ Rh×k and using equation (4), the objective function (3) can be rewritten:   P ∗ = Argmax P ∈Rw×k

= Argmax P ∈Rw×k

|P T

|P T

C c=1

C c=1

nc (XcQ −X Q )T (XcQ −X Q ) P | Q Q T Q Q i∈Ωc (Xi −Xc ) (Xi −Xc )

(5)

P|

|P T SbQ P |

(6)

Q |P T Sw P|

Q and SbQ being respectively the generalized within-class covariance matrix with Sw and the generalized between-class covariance matrix of the set (XiQ )i∈{1,...,n} , where

∀i ∈ {1, . . . , n}, XiQ = QT · Xi

(7) −1

Q Therefore, the columns of matrix P ∗ are the k eigenvectors of Sw SbQ with largest eigenvalues. A stable way to compute the eigen-decomposition, by applying Singular Value Decomposition (SVD) on the covariance matrices, is given in [6]. If Q = Ih , the identity matrix of size h × h, P ∗ is the projection matrix of 2DoLDA [5]. Given that, for every square matrix A, |AT A| = |AAT | and considering the matrix P T (Xc − X)T Q of size k × k, the objective function (3) can be rewritten:   T T T | C c=1 nc (Q (Xc −X)P P (Xc −X) Q)| (Q∗ , P ∗ ) = Argmax (8) [ C (QT (X −X )P P T (X −X )T Q)| (Q,P )∈Rh×k ×Rw×k

c=1

i∈Ωc

i

c

i

c

For any fixed P ∈ Rw×k , using equation (8), the objective function (3) can be rewritten: Q∗ = Argmax Q∈Rh×k

|QT ΣbP Q| P Q| |QT Σw

(9)

P Σw and ΣbP being the generalized within-class and between-class covariance matrices of the set ((XiP )T )i∈{1,...,n} , where

∀i ∈ {1, . . . , n}, XiP = Xi · P

(10) −1

P ) ΣbP with Therefore, the columns of matrix Q∗ are the k eigenvectors of (Σw largest eigenvalues. We can note that BDA leads to a significant reduction in the dimensionality of the signatures compared to 2DPCA and 2DoLDA: the size of a signature using BDA is k 2 , versus h · k for 2DoLDA and 2D PCA.

Face Recognition Using Modular Bilinear Discriminant Analysis

2.1

27

Algorithm of the BDA Approach

Let us initialize P0 = Iw and α0 = 0. The number k of components is fixed. The choice of k will be discussed in section 4. The proposed algorithm for BDA is: While αt < τ 1. 2. 3. 4. 5. 6. 7.

For i ∈ {1, . . . , n}, compute XiPt = Xi · Pt . Pt Pt −1 Compute Σw , ΣbPt and (Σw ) · ΣbPt ; Pt −1 Compute Qt , whose columns are the first k eigenvectors of (Σw ) · ΣbPt ; Qt For i ∈ {1, . . . , n}, compute Xi = (Qt )T · Xi . Qt Qt −1 , SbQt , and (Sw ) · SbQt ; Compute Sw Qt −1 Compute Pt , whose columns are the first k eigenvectors of (Sw ) · SbQt ;  Compute αt = (Pt − Pt−1 22 + Qt − Qt−1 22 .

It should be noted that the roles of P and Q can be switched, by initializing Q0 = Ih , computing XiQt instead of XiPt at step 1., and so on. Experimental results show similar performances for the two versions of the algorithm. The stopping parameter τ can be determined empirically, from experiments. As Pt and Qt are normal matrices, no drastic variation of τ is observed from one test set to another, and therefore τ can be determined easily. However, experimental results have also shown that after one iteration the recognition results are satisfying. Therefore, in the following, we will use the preceding algorithm with only one iteration, which is less costly, ensures good recognition results, and frees us from determining τ .

3

Modular Bilinear Discriminant Analysis (MBDA)

We can consider that there are basically two expert combination scenarios. In the first scenario, all the experts use the same input pattern but different feature extraction techniques, chosen to be complementary. In the second scenario, each expert uses its own representation of the input signal, but all the experts use the same feature extractor. In this paper, we focus on the second scenario, and propose a face recognition system based on three experts built by using BDA, each one being trained on a specific face template. 3.1

Description of the Face Templates

Fig. 1 shows the different facial regions from which the experts are trained. Expert 1 is trained from a face region of size 75 × 65 pixels containing all the facial features, centered on the position of the nose. Expert 2 is trained from a template of size 40 × 65 pixels containing the nose, eyes and eyebrows, while expert 3 uses a template of size 30 × 65 pixels containing only the eyes and eyebrows. These regions are chosen to guarantee good recognition rates in most configurations, according to the results obtained in [7, 8].

28

M. Visani, C. Garcia, and J.-M. Jolion

1

2

3

Fig. 1. Templates from which are trained the three BDA experts

3.2

Expert Combination

We investigated two ways of using simultaneously the three experts: multistage expert combining and expert voting. Multistage Expert Combining. Multistage Expert Combining (see Fig. 2) requires a two-step training stage. In the first step, each expert j ∈ {1, . . . , 3} is trained separately, to build the corresponding pair of projection matrices (Qj , Pj ). In the second step, a combiner applies PCA on the concatenation of the signatures obtained in the first step. Each training sample (Xi ) ∈ Ω is projected Q ,P onto (Qj , Pj ), giving the matrix Xi j j , of size kj × kj (see equation (1)). Then, Q ,P each of the three matrices Xi j j is transformed into a vector. Next, these three vectors are concatenated to obtain a single vector Xˆi of large size k12 + k22 + k32 . A ˆi )i∈{1,...,n} , in order subspace F is built by applying PCA to the set of vectors (X to reduce the dimensionality of the signatures to size l 0, which is defined as [1] :

v(t ) = σ (−σ t )α / 2 eσ t / 2 u (−t ) .

(6)

where u is the heaviside function (u(t)=1 for t ≥ 0, u(t)=0 for t0, these filters have a non-symmetric bell-shaped envelope, i.e. a gamma-shaped window. It also implies that analyzed events correspond to those in a close past of t0. In this case, temporal information is more on the basis of past events than on current ones. Besides, for large α the window v(t) increasingly resembles a Gaussian window. These 1-D filters are then defined as: d n (t ) =

(

)

n !/ Γ(n + α + 1) σ (σ t )α e −σ t L(nα ) (σ t )u (t ) .

(7)

where Γ is the gamma function [1]. The generalized Laguerre polynomials Ln(α)(t), which are orthogonal with respect to the weighting function tα⋅e–t, are defined by Rodrigues’ formula [10] as: L(nα ) (t ) =

t −α e t d n n +α − t (t e ) . n ! dt n

(8)

From (8) one can see that generalized Laguerre filters are related to time derivatives of the localizing window v(t) defined in (6). Hence, filters of increasing order analyze successively higher frequencies or temporal variations in the signal. 3.2.2 Meixner Filters Meixner filters are the discrete equivalent of generalized Laguerre filters [3]. They are equal to Meixner polynomials multiplied by a square window v2(x) = c x (b) x / x ! , which is the discrete counterpart of a gamma window and it behaves similarly to a Poisson kernel [11]. (b)x is the Pochhammer symbol defined by (b)0 = 1 and (b)x = b(b+1)(b+2)…(b+x–1), x=1,2,… . Parameters b and c are equivalent to parameters of the generalized Laguerre filters α and σ, respectively. However, for the discrete case, b>0 and 0 g where g and s are two parameters, see Figure 4 for this function for various s values at fixed g = 0.2. The parameter g is a cut value to restrict the adjustment 1

0.3 0.5 1.0 2.0 3.0

0.5

0 0

0.2

0.4

0.6

0.8

1

Fig. 4. Mapping function f (x) for various s values at fixed g = 0.2

114

X. Jiang and S. Lewin

only to those segments whose local shape width is below g. The parameter s represents to what degree the segment is made shorter. A large value of s strongly reduces the segment length and thus its overall perception. A problem remains to consider. In general, the computed curve is not necessarily closed. Two solutions have been suggested in [5] to close a curve. In this work we use the simpler one. For notational simplicity we assume the starting point being the origin of the coordinate system. In this case only the amount (−xn , −yn ) is missing for the last point to reach the origin. If we introduce a yn xn , − n−1 ) to each directional vector, the i-th point p∗i will small correction (− n−1 yn xn   take the new position (xi , yi ) = (xi − (i − 1) · n−1 , yi − (i − 1) · n−1 ). Then the ∗   last point pn = (xn , yn ) corresponds to the origin. In contrast to the original shape, the computed closed shapes have a nonuniform sampling. As a final step, we perform the sampling process again to achieve the uniform sampling. This operation finishes one evolution step and the generated shapes are now ready to be handled by shape matching algorithms. 0.4 15

4

3

4

0.3

1

10

2

1

2

0.2

3

0.1

5 0

0.2

0.4

0.6

0.8

1

0

t

2

3

4

5

6

7

8

9

10

2

3

4

5

6

7

8

9

10

0.4 4

3

15

4 0.3

1

2

10

3

0.2 1 2

0.1

5 0

0.2

0.4

0.6 t

0.8

1

0

Fig. 5. Evolution of shape B (s = 0.5, top; s = 2.0, bottom; in both cases g = 0.2): CSS representation (middle) and Fourier descriptors (right)

Fig. 6. Additional examples of shape evolution. From left to right: (a) shape; (b) s = 0.5; (c) s = 1.0; (c) s = 2.0. In all cases g = 0.2.

An Approach to Perceptual Shape Matching

115

Figure 5 illustrates the evolution for shape B in Figure 1. Now let us see how the evolution makes the fundamental problems discussed in the introduction section less harmful. With increasing values of s, shape B continuously evolves to become geometrically similar to shape A. The perceptually less important tail is shortened without distorting the overall shape. In accordance with this, the shape representations evolve to become very similar to that of shape A. For shape B the two original features in CSS receive increasingly larger distance apart from each other. In addition, the two new significant features resulting from the tail continuously lose their magnitude and, finally, their influence becomes so small that we can reasonably assume that it will be tolerated by shape matching algorithms. Also in the case of Fourier descriptors the shape representation becomes much more similar to that of shape A, see Figure 1. Figure 6 shows the evolution of two more complex shapes. Again, it is able to reduce the parts which are rated as perceptually less important according to the criterion defined in this work. For a fixed g value the evolution process with increasing s values converges to a shape where all line segments with local shape width smaller than g vanish. This is the ground shape resulting from the input shape with respect to the particular scale g. 3.1

Integration in Shape Matching Algorithms

For matching two shapes X and Y , both can be preprocessed by shape evolution for N various parameter values of s and g to produce a series of deduced shapes X1 , X2 , . . . , XN and Y1 , Y2 , . . . , YN , respectively. Then, they are matched pairwise and the best match is chosen. If a shape matching algorithm computes the shape distance d(X, Y ), the adapted shape distance is given by d∗ (X, Y ) =

min d(Xi , Yj )

1≤i,j≤N

In our tests we have fixed the parameter g and varied the parameter s only. This integration scheme is applicable to any shape matching method. For conducting experiments we will consider CSS and Fourier descriptors only. Note that our goal is to show that given some shape matching method, the approach proposed in this paper helps improve the performance for retrieving perceptually similar shapes. For doing so, we do not intend to rate the existing shape matching algorithms and to experiment only with the “best” ones. For instance, wavelet shape representations have been shown to outperform Fourier descriptors [3]. 3.2

Experimental Results

We compiled a database of 100 shapes to test our approach; see Figure 7 for the whole database. It is divided into twelve groups in such a way that the within-class similarity is reasonably high. The shapes are mainly from the SQUID database [8]. In addition we included some shapes which are perceptually similar to the SQUID shapes, but have less details in their contour. For instance, the last shape of group 9 in Figure 7 is originally from SQUID database, while the

116

X. Jiang and S. Lewin

Fig. 7. Groups of the test database. They are numbered in top-to-bottom and left-toright manner.

5th shape of the same group is artificially generated. The goal of doing this is to increase the complexity of perceptual similarity retrieval and we expect that traditional shape analysis methods will encounter difficulties in dealing with this sort of shape similarity and the use of our approach helps overcome the difficulties. The shapes are processed based on three parameter settings: s = 0.5/1.0/2.0 with constant g = 0.2. Together with the original shape they build the extended database. We use the same procedure as in [8] to measure the retrieval performance. For each shape a search is done to retrieve the n most similar shapes from the database. As in [8] n = 15 is chosen for the test, Then, we count the number of retrieved shapes which are in the same group as the input and divide this number by the size of the group. The result is the performance measure for a particular shape. Repeating this operation for all members of a group and all groups gives us the performance measure for the particular group and the whole database, respectively. We determine this retrieval performance measure for a reference shape matching method (CSS or Fourier descriptors) with and without the application of our approach and compute the difference. Table 1 lists the improvement for each group of the database due to the shape evolution. The average improvement over the whole database amounts to 9.3% and 15.0%, respectively.

An Approach to Perceptual Shape Matching

117

Table 1. Improvement due to preprocessing for each group of the database CSS shape representation 1 2 3 4 5 6 7 8 9 10 11 12 12.4% -2.5% 37.0% 13.9% -7.4% 14.0% 7.4% 28.6% 18.5% -1.2% 0.0% -9.4% Elliptic Fourier descriptors 1 2 3 4 5 6 7 8 9 10 11 12 37.0% -5.0% 28.4% 8.3% 12.4% 19.8% -7.4% 20.4% 11.1% 33.3% 18.4% 3.1%

Fig. 8. Two test results in combination with CSS matching

Overall, a significant improvement has been achieved. In both cases the most remarkable enhancement is observed for groups 1/3/4/6/8/9. For the two shapes in Figure 6 from group 3 and 9, respectively, the retrieval results using the CSS shape representation are given in Figure 8. As one can expect, the shape evolution synthesizes a series of shapes, in which perceptually less important parts are shortened and are accordingly under-weighted in the shape representation. For large s values they even tend to vanish. This fact provides the basis for a potential match. Without the shape evolution two shapes may be so different with respect to their shape representation (but not to their perception) that they cannot be matched well. There are also small performance degradation, in particular in the case of CSS shape matching. The reason is that CSS matching is itself of some qualitative nature. Evolving a couple of shapes makes them qualitatively similar to shapes from other groups. This confuses the CSS matcher in certain cases.

4

Non-uniform Sampling

In this section we discuss another way of utilizing the local shape width. Recall the two shapes in Figure 1. The remarkable difference in their representations, say CSS, is due to the fact that the tail absorbs a considerable portion of the normalized arclength, while the major shape part shrinks accordingly. One solution could be to perform a non-uniform sampling of the shape. Parts with low local shape width should be sampled more coarsely. Given a constant number n of sampled points this strategy implies that perceptually meaningful parts (with

118

X. Jiang and S. Lewin

larger local shape width) will receive a larger number of sampled points compared to a uniform sampling. For shape B, for instance, a sufficient suppression of the tail will produce a list of sampled points which contains very few points from the tail and thus becomes similar to that of shape A. If this list is directly fed to CSS computation, then we have a good chance to obtain similar CSS representations for the two shapes. One possible realization of the idea is described as follows. The interval of the local shape width [0, 1] is first quantized into k levels. Then, the original shape is grouped into m segments, each containing points of the same shape width. For each segment i with a total length li , we determine how many points it should be adaptively sampled into. This number is given by ni = α · li · f (lswi ), where lswi represents the local shape width of segment i. Since we want to coarsely sample the segments with low lswi , f () should be some monotonically ascending function. In our experiments we use the same function as in the last section, see Figure 4. We cannot fix ni locally because there is a constraint to keep the total number n of points constant. This is managed by the factor α. Adding all ni ’s together, we obtain m 

m 

ni = α

i=1

li · f (lswi ) = n + m

i=1

 which immediately leads to α = (n + m)/ m i=1 li · f (lswi ). Figure 9 shows the CSS of shape B for two non-uniform samplings. With increasing s, the shape representation evolves to become very similar to shape A.

20

20 1 1

2

15

2

15 3

10

4

10

5

5 0

0.2

0.4

0.6

0.8

1

0

0.2

0.4

t

15

3

15 3 1

10

5

5

1

0.2

0.8

1

4

4

2

10

0

0.6 t

0.4

0.6 t

0.8

1

0

2

0.2

0.4

0.6 t

Fig. 9. Non-uniform sampling applied to CSS

0.8

1

An Approach to Perceptual Shape Matching

119

This behavior is comparable to that illustrated in Figure 5. Finally, the difference with CSS of shape A becomes small enough to be tolerated by shape matching algorithms.

5

Conclusions

In this paper we have proposed the concept of local shape width and its application to perceptual shape matching. A shape preprocessing approach naturally emerges from this concept. It delivers a series of new shapes from an input shape. They have the property that perceptually less significant parts smoothly vanish while other parts remain unchanged. This tool can be used in combination with any shape matching algorithm. Together with the non-uniform shape sampling the concept of local shape width gives us a powerful tool to handle the sort of perceptual shape similarity under consideration. Local shape width may serve as a shape matching feature directly. For shape representation a curve function r(t) = (lsw(t)) can be built. In the discrete case it is simply a list of local shape width values at the points of the curve. Two curve functions can then be compared by means of some distance measure. One interesting property of this approach is that it enables a sort of articulated shape matching. As long as the local shape width remains (almost) unchanged, it does not make any difference how the parts are geometrically related to each other. In the case of a shape with tails, for instance, the tails can be arbitrarily bended without (substantially) effecting the shape representation. This property is obviously not given in many other shape representations. In contrast to geometric similarity, perceptual similarity is manifold and much more difficult to deal with in general. So far relatively few work is known towards capturing the perceptual similarity and there remains considerable room for research. Our work is another attempt in this direction. We hope that it will find further applications in addition to those discussed in this paper.

Acknowledgments The authors are grateful to the maintainers of SQUID database for public availability.

References 1. Y. Avrithis, Y. Xirouhakis and S. Kollias. Affine-invariant curve normalization for object shape representation, classification, and retrieval. Machine Vision and Applications, 13: 80–94, 2001. 2. S. Berretti, A. Del Bimbo and P. Pala. Retrieval by shape similarity with perceptual distance and effective indexing. IEEE Trans. on Multimedia, 2(4): 225–239, 2000. 3. G. Chuang and C.-C. Kuo. Wavelet descriptor of planar curves: Theory and applications. IEEE Trans. on Image Processing, 5: 56–70, 1996.

120

X. Jiang and S. Lewin

4. Y. Gdalyahu and D. Weinshall. Flexible syntactic matching of curves and its application to automatic hierarchical classification of silhouettes. IEEE Trans. on PAMI, 21(12): 1312–1328, 1999. 5. X. Jiang, H. Bunke, K. Abegglen, and A. Kandel. Curve morphing by weighted mean of strings. Proc. of 16th ICPR, Vol. IV, 192–195, 2002. 6. V. Kindratenko. On using functions to describe the shape. Journal of Mathematical Imaging and Vision, 18: 225–245, 2003. 7. L.J. Latecki, R. Lakmper and D. Wolter. Optimal partial shape similarity. Image and Vision Computing, 23: 227-236, 2005. 8. F. Mokhtarian and M. Bober. Curvature Scale Space Representation: Theory, Applications, and MPEG-7 Standardization. Kluwer Academic Publishers, 2003. 9. M Nixon and A. Aguado. Feature Extraction & Image Processing. Newnes, 2002. 10. D. Sinclair and A. Blake. Isoperimetric normalization of planar curves. IEEE Trans. on PAMI, 16(8): 769–777, 1994. 11. R.C. Veltkamp and M. Hagedoorn. State of the art in shape matching. In: Principles of Visual Information Retrieval (M.S. Lew, Ed.), Springer-Verlag, 2001. 12. D. Zhang and G. Lu. Review of shape representation and description techniques. Pattern Recognition, 37: 1–19, 2004.

Interactive Volume Visualization Techniques for Subsurface Data Timo Ropinski and Klaus Hinrichs Institut für Informatik, Westfälische Wilhelms-Universität Münster, 48149 Münster, Germany {ropinski, khh}@math.uni-muenster.de

Abstract. In this paper we describe concepts, which support the interactive exploration of subsurface information extracted from seismic datasets. Since in general subsurface information is of volumetric nature, appropriate visualization techniques are needed to provide an insight view to special regions of interest. Usually clipping planes or surface extraction techniques are used for this purpose. We will present an approach, which allows the user to interactively change the visual representation for distinct regions of seismic datasets. Using this technique highlighting of regions of interest as well as clipping against volumetric regions can be realized. Volumetric clipping regions have the potential to assist the user when visually intruding into a 3D dataset by permitting an occlusion free view to inner regions of the dataset. During this process it is desirable to know where the current position is located relative to the whole dataset. We will introduce a 3D widget, which displays information concerning the location and orientation of the virtual camera during the exploration process.

1 Introduction Geographic information systems (GIS) have been proven essential for the tasks of exploring and communicating spatiotemporal data and its associated information. Particularly the development of 2D GIS has advanced in the last decades, because there has been a great demand for these systems to support map-based planning tasks. Since geographic data is inherently three-dimensional, a 2D representation is insufficient for many tasks, e.g. the visualization of terrain data. Therefore the idea of 3D GIS has been proposed [7]. Most 3D GIS adapt the concepts of polygonal representations for storing geographic information. One disadvantage of this form of representation is that geo-objects can only be represented by a polygonal approximation of their surface. This is sufficient as long as no information associated with volumetric subsurface data has to be represented. Since the visualization of subsurface information is essential in many application fields, e.g. locating oil or gas reservoirs as well as earthquake prediction, appropriate forms of representation are needed. For representing information associated with subsurface data volumetric techniques are adequate ([8], [9], [18]), which have been already applied in medical visualization. Hence these techniques are highly developed. Especially for the use with large datasets, as found in geo-applications, many algorithms have been proposed, which allow interactive visualization on standard desktop computer systems. S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 121 – 131, 2005. © Springer-Verlag Berlin Heidelberg 2005

122

T. Ropinski and K. Hinrichs

In this paper we will propose techniques for enhancing interactive visualization and information retrieval of volumetric subsurface datasets. One problem of visualizing information associated with volumetric data is that usually no insight view can be achieved. Especially when navigating through a visual representation the view of the camera will always be occluded. To avoid this problem we propose the application of 3D magic lenses ([13], [16]) to enhance visualization of subsurface information. By using 3D magic lenses it is possible to assign different visual representations to special regions of interest. Thus an occlusion-free view can be obtained by removing distracting information when intruding into the visual representation. While navigating through the dataset it is desirable to know where the current view is located and how it is oriented in relation to the whole volume dataset. Therefore, we propose a 3D widget, called orientation widget, which shows the current position and the orientation of the virtual camera in relation to the 3D dataset. This paper is divided into five sections. The next section will describe some related work concerning interactive visualization and exploration of subsurface information. In Section 3 we will describe some visualization techniques and give some examples, explaining how interactive visualization of subsurface information can benefit from the use of these techniques. Section 4 introduces the orientation widget, which assists the user while navigating through the visual representation. The paper concludes in Section 5 by giving an overview of the proposed concepts and future work.

2 Related Work Visualization techniques for representing volumetric phenomena have advanced in the past years. Most of the proposed techniques have been developed for medical applications. A good overview of visualization techniques used to visualize seismic datasets can be found in [9]. There are two kinds of geological data from which subsurface information can be retrieved: well data and seismic data [8]. Well data is directly extracted from a well, which contains precise information of the geological structure of the subsurface. Since it is very expensive to acquire well datasets, numerous techniques have been developed, which focus on information retrieval from seismic datasets [5]. Seismic data is acquired indirectly. Explosives are used to emit sound waves, which are propagated through the subsurface. The reflections of those sound waves are measured with special reflectors located at the surface. Thus geological layers, which reflect the sound waves, can be identified based on the time elapsed before a signal is received. The resulting information is encoded in a 3D volume composed of discrete samples, each representing the amplitude reflected beneath the surface. This 3D volume is usually stored as a SEG-Y dataset [14]. The acquired amplitudes can be visualized using different transfer functions to display the geological structure [15]. Since volumetric datasets, either of medical or geological type, are usually very large, many techniques have been proposed, which allow an interactive visualization of such datasets ([6], [12]). We do no refer directly to these kinds of techniques, but are focusing on visualization techniques, which reveal information usually hidden in the interior of seismic datasets. The described techniques are an extension to direct seismic visualization, i.e. visualizing without first performing an edge or surface detection.

Interactive Volume Visualization Techniques for Subsurface Data

123

Two different visualization techniques are widely accepted for displaying information encoded in seismic volume datasets. The slicing approach displays a sequence of successive slices of the 3D dataset [3]. By displaying these slices rapidly the user can construct a mental image containing the desired information. This concept can be implemented easily by applying clipping planes, which are supported by current graphics hardware. A disadvantage of this method is that it only leads to satisfying results when an image sequence is displayed. Hence showing a single image representing a single slice is less meaningful. Wolfe and Liu [18] have proposed another approach, which allows to reveal information hidden inside the dataset. Their technique uses a threshold and visualizes only certain data values present in the volume by assigning transparency. Thus it is possible to reduce the information being visualized and therefore to give the user a better overview of the whole dataset. Both visualization concepts have been widely adapted in interactive visualization tools specialized for the interpretation of seismic volume datasets ([10], [11]). Fröhlich et al. [4] have described the application of volumetric lenses to seismic datasets. They claim that the exploration of seismic datasets can benefit from the use of volumetric lenses, which can be freely positioned in the scene. However, they support only cuboid lenses. Since the surfaces of the cuboid represent orthogonal planes, this results in a visualization similar to the slicing approach when multiple slices are used. Therefore their technique may be considered as a generalization of the slicing approach, which uses more than one clipping plane.

3 Visualization Techniques Due to the volumetric nature of seismic datasets it is difficult to provide mechanisms for enabling a sophisticated interactive visualization, since several problems may arise. First of all, because most commonly used graphics systems just provide a 2D projection of the data to be visualized, information hidden deep inside the dataset is usually occluded by the regions, which lie at the border. Currently most visualization techniques applied to seismic datasets use the slicing approach to deal with the occlusion problem, i.e. one or multiple clipping planes are used to reveal occluded regions of the dataset, and the user has to construct a mental image representing the information of interest. 3D magic lenses have the potential to deal with this shortcoming. By using the magic lens metaphor occlusions can be avoided without loosing information, which is needed to visualize the 3D structure of a seismic dataset. Interactive visualization of 3D magic lenses gives the user the ability to explore seismic datasets by providing full contextual information. The mechanism is based on the 2D magic lens metaphor, which has been introduced by Bier et al. [1]. In their work they describe Toolglass¥ widgets as new interface tools for information visualization that can appear, as though on a transparent sheet of glass, between an application and a traditional cursor. Toolglass widgets can be positioned with one hand while the other positions the cursor. They may incorporate visual filters, known as magic lenses, which modify the visual representation of application objects, enhance data of interest or suppress distracting information in the region of interest determined by the

124

T. Ropinski and K. Hinrichs

shape of the lens. 3D Magic lenses [16] are represented by glass volumes associated with a visual representation, which is applied to the geometry intersecting the lens volume. Image-based rendering of convex magic lenses [13] can be performed with threepass rendering. The first pass renders the parts of the virtual environment which lie behind and next to the lens volume (a), the second pass renders those parts which intersect the lens volume (b), and the third pass renders everything in front of the lens volume (c) (see Figure 1).

Fig. 1. Rendering of the three regions (a-c)

During each pass a second depth test is used to check if the current sample of the volume dataset lies behind, inside or in front of the lens volume. Therefore the depth component of the current sample is compared with the corresponding depth value stored in a depth texture, which is part of the data structure representing the depth structure of the lens volume. Because the lens volume has to be convex, two different depth textures are used: dt0 stores the depth values representing the front boundary of the lens volume and dt1 stores the depth values representing the back boundary. Each depth texture is obtained during an extra rendering pass where just the lens geometry is rendered, before rendering the volume dataset. After rendering the lens geometry the resulting depth buffer information of each pass is stored in the corresponding depth texture. Thus the viewport-sized depth textures dt0 and dt1 obtained in this way store the depth structure of the lens volume. The depth textures are used in three subsequent rendering passes to determine whether a sample intersects one of the three described regions (a-c): 1. 2. 3.

use depth texture dt1 with its additional depth test set to greater (render region a) use depth texture dt1 with its additional depth test set to less and use depth texture dt0 with its additional depth test set to greater (render region b) use depth texture dt0 with its additional depth test set to less (render region c)

We have adapted the described concepts to volume rendering techniques using texture mapping hardware [2] which we use to visualize seismic datasets. These volume rendering techniques visualize a volumetric dataset which is represented as either several 2D textures or one 3D texture by rendering a stack of screen aligned polygons onto which the volume dataset is projected using appropriate texture coordinates.

Interactive Volume Visualization Techniques for Subsurface Data

125

When using texture based volume rendering techniques it is important to ensure that the slices are rendered in a predefined order, either back to front or front to back, since the slices are blended based on the transparency values stored in the volume dataset to achieve a correct visualization. To assign a special visual representation when rendering the parts of the volume dataset intersecting the lens volume, this representation must be applied during rendering pass 2 described above. The visual representation can be altered by using the methods provided by the magic lens interface: interface MagicLens { void addRepresentationAttr(Attribute

attr);

void removeRepresentationAttr(Attribute attr); AttributeComposite getRepresentation(); void addDSRepresentationAttr(string name, Attribute attr); void removeDSRepresentationAttr(string name, Attribute attr); AttributeComposite getDSRepresentation(string name); void addInvalidDS(string name); void removeInvalidDS(string name); }

We have designed this interface as an extension to a scene graph based 3D graphics system to allow easy use of the described concepts. There are three different groups of methods usable for altering the visual representation inside the lens volume: methods for accessing general representations, methods for accessing representations for particular shapes, and methods needed to exclude shapes from rendering. With the methods addRepresentationAttr() and removeRepresentationAttr() the visual representation used inside the lens volume can be specified. All those attributes are bundled in an object of the class AttributeComposite, which is returned by the method getRepresentation(). If several overlapping datasets have to be visualized, similar methods can be used to assign visual representations to these datasets. The datasets have to be identifiable by an object name. The object name together with the attribute representing the representation is passed to the methods addDSRepresentationAttr() and removeDSRepresentationAttr(). The representation used for a particular dataset can be queried by calling the method getDSRepresentation() with the corresponding object name. Furthermore a lens can be configured to exclude particular datasets from rendering by calling the methods addInvalidDS() and removeInvalidDS(). By giving the user the ability to change the region of interest interactively, magic lenses have the potential to support users in the task of interactive information visualization by allowing 3D exploration. We have classified two types of 3D magic lenses: camera lenses and scene lenses. Camera lenses are positioned relative to the virtual camera, whereas scene lenses can be positioned anywhere in the virtual environment.

126

T. Ropinski and K. Hinrichs

Camera lenses can assist the user while intruding in dense datasets like seismic datasets by removing or translucently rendering parts, which occlude the view of the camera. Thus it is possible to navigate through virtual environments containing dense data without removing context information. To assist the user in exploring seismic datasets we have implemented three visual representations bound to camera lenses: (1) semi-transparent rendering, (2) removing of partial or complete data intersecting the lens volume, and (3) highlighting data inside the lens volume by applying extra light sources. These representations can be combined in a single lens. In contrast to camera lenses, scene lenses are particularly suitable for accentuating special regions of interest interactively. By using the magic lens metaphor it is assured that the context information does not change, while it is possible to alter the visual representation inside the lens volume. In order to assist the user in exploring the data different visual representations can be applied inside lenses. One example is changing the transfer function used when rendering the dataset. Although in most cases the same transfer function is used for the whole dataset, it may be desirable to use different transfer functions in different parts of the dataset, e.g. to identify special regions of interest [15]. Thus it is possible to reveal structures within the data without changing the context, since in the parts not intersecting the lens volume the original transfer function is retained (see Figure 2 (left)). Magic lenses can also be used as clipping volumes as shown in Figure 2 (middle), where a spherical clipping lens is used to reveal information hidden inside the dataset. For an easier identification of the clipping region its silhouette edges are enhanced. Figure 2 (right) shows the usage of the interactive drawer metaphor. With this metaphor the user can interactively cut differently shaped subvolumes out of the whole dataset. These volumes are visualized using the representation assigned to the magic lens. To better display the information hidden inside these subvolumes, they are duplicated and pulled out in a drawer like manner to be positioned outside the dataset. Thus the user can explore even the sides otherwise occluded. The green arrow shown in Figure 2 (right) indicates in which direction the subvolume has been pulled out.

Fig. 2. Magic lenses with different visual representations applied to a seismic volume dataset. Cuboid lens using different transfer function in combination with edge enhancement (left), spherical clipping lens with enhanced silhouette edges (middle), and interactive drawer metaphor with a transfer function highlighting extremely dense regions (right).

Interactive Volume Visualization Techniques for Subsurface Data

127

Fig. 3. Clipping lens combined with slice

Furthermore it is possible to combine the well known techniques described in Section 2 with the magic lens metaphor. For example when only displaying certain data values present in the volume [18], this can be done exclusively inside the lens volume by applying threshold during rendering (see Figure 6 (top right)). In Figure 3 a magic lens is used to combine volumetric clipping with the slicing approach. The seismic dataset is volumetrically clipped by a sphere shaped lens, but to give a better overview, two slices representing the right and the bottom outer face of the dataset are visualized. Alternatively the interactions needed for the slicing approach can be adopted by the magic lens technique. Because the size and the shape of the lens volume can be altered interactively, a resizable volumetric clipping region can be realized [17]. When resizing a volumetric clipping region, the information shown on the border changes with the size of the volume. Similar to moving a single clipping plane, the resulting information can be used to reveal information hidden within the dataset.

4 Navigation Aspects The use of the 3D magic lens metaphor as described in Section 3 gives the user the ability to intrude into the dataset. Thus a fly-through metaphor can be used for navigation. In aboveground navigation the user usually has a set of landmarks he can use for position sensing. In contrast the user does not have any such static reference points when navigating beneath the surface. Therefore a tool is needed for determining the position and the orientation while navigating through dense datasets. We propose to use an orientation widget to display the current orientation and position relative to the dataset. The widget is composed of a bounding volume, representing the boundaries of the seismic volume dataset, and an arrow displaying the current orientation and position of the virtual camera (see Figure 4 (left)).

128

T. Ropinski and K. Hinrichs

Camera Tf Tf SS

Orientation Widget

Fig. 4. The orientation widget showing the current position of the camera (left). Scene graph with orientation widget (right).

The position as well as the orientation of the arrow is derived from the current orientation matrix determined by the virtual camera. Although the concepts described in this paper can be easily realized with different kinds of graphics systems, we are going to explain our implementation, which is based on a high-level rendering system using a scene graph to define the scene content. The structure of the orientation widget as well as its integration in the scene graph is shown in Figure 4 (right). The orientation widget consists of two primitives, the bounding box and the arrow. The orientation of the arrow depends on the virtual camera. Both elements are transformed by the scaling transformation S. This scaling transformation defines the size of the widget relative to the size of the seismic volume. It directly influences the size of the orientation widget in screen space, since the front face of the bounding box is always parallel to the image plane and has a fixed distance to the camera. The matrix for this transformation can be either assigned automatically based on screen coordinates and the dimensions of the seismic dataset, or it can be set by the user. Tf denotes the camera dependent transformation used to orient the arrow. It is updated during each redisplay, when the scene graph is evaluated. To perform this update, the transform needs to have a reference to the virtual camera to read its current orientation matrix. Using the orientation widget the user can visually intrude into the dataset without loosing orientation (see Figure 5). Hence most navigation techniques, which have been established in 3D virtual environments can be used subsurface, e.g. fly-through metaphors which may be combined with exponential acceleration algorithms. The primitives used for the orientation widget, i.e. the bounding box and the arrow, can be altered for different purposes. For example to enable a more accurate prediction of the position inside the dataset, auxiliary lines can be visualized, each as long as the corresponding dimension of the bounding box. By using these kind of auxiliary lines the starting point of each line at a side of the bounding box gives the user an indication of the camera’s orientation and location inside the lens volume. Another extension of the orientation widget could allow interactive positioning of the virtual camera by providing handles integrated into the widget.

Interactive Volume Visualization Techniques for Subsurface Data

129

Furthermore the orientation widget can be positioned freely by applying appropriate transformations. Thus it is possible to provide a zoomed view of the widget by scaling and centering it on the screen. This visualization of the orientation widget may be helpful for a more accurate positioning.

Fig. 5. Orientation widget during intrusion

Fig. 6. Different types of magic lenses. Cuboid lens with enhanced clipping plane edges (top left), a spherical lens showing different transfer function and threshold (top right), visualization of the drawer metaphor using a highlight transfer function (bottom left), and the drawer metaphor combined with threshold (bottom right).

130

T. Ropinski and K. Hinrichs

5 Conclusion and Future Work In this paper we have presented techniques for enhancing interactive visualization of subsurface information extracted from seismic datasets. All presented visualization techniques perform on off-the-shelf graphics hardware at interactive frame rates. We have proposed the use of 3D magic lenses to allow intruding into seismic datasets without occluding the view of the camera. Using the 3D magic lens metaphor different visual representations can be applied inside the lens volume. Those representations can be used to highlight regions of interest or filter the kind of information visualized for the desired region. Furthermore different transfer functions as well as threshold based transparency techniques can be applied inside the lens volume to improve information visualization. For orientation purposes a 3D orientation widget has been introduced, which displays the current position and the orientation of the virtual camera. This widget gives the user the ability to plan navigation tasks by providing an immediate feedback. The proposed techniques can be combined with existing information visualization techniques used to display information associated with volumetric seismic and well datasets. Since volumetric data is used in many geo-related fields as well as in medical visualization, there is a bright spectrum of applications, which can benefit from the described techniques. For example the oil and petroleum industry can apply the concepts for well planning. Furthermore archeological applications may benefit from the described concepts when analyzing strata. Currently we are working on an easy to use library, which implements the described concepts. Since the proposed visualization techniques can be extended easily, i.e. new visual representations can be defined to be applied inside the lens volume, geo-scientists can develop their own lenses for their preferred graphics system using the library.

References 1. E.A. Bier, M.C. Stone, K. Pier, W. Buxton, and T. DeRose, “Toolglass and Magic Lenses: The See-Through Interface”, In Proceedings of SIGGRAPH’93, ACM Press, pp. 73–80, 1993. 2. B. Cabral, N. Cam, and J. Foran, “Accelerated Volume Rendering and Tomographic Reconstruction using Texture Mapping Hardware”, In Proceedings of Symposium on Volume Visualization, ACM Press, pp. 91–98, 1994. 3. M.P. Curtis, A.C. Gerhardstein, and R.E. Howard, “Interpretation of Large 3-D Data Volumes”, Paper 510.5, Expanded Abstracts of the Society of Exploration Geophysicists 56th Annual Meeting, pp. 497-499, 1986. 4. B. Fröhlich, S. Barrass, B. Zehner, J. Plate, and M. Göbel, “Exploring Geo-Scientific Data in Virtual Environments”, Proceedings of IEEE Visualization, pp. 169-174, 1999. 5. D. Gao, “Volume Texture Extraction for 3D Seismic Visualization and Interpretation”, In Geophysics, Vol. 68, No. 4, pp. 1294–1302, 2003. 6. S. Guthe, M. Wand, J. Gonser, and W. Straßer, “Interactive rendering of large volume data sets”, In Proceedings of IEEE Visualization, pp. 53–59, 2002.

Interactive Volume Visualization Techniques for Subsurface Data

131

7. D. Koller, P. Lindstrom, W. Ribarsky, L.F. Hodges, N. Faust, and G. Turner, “Virtual GIS: A Real-Time 3D Geographic Information System”, In Proceedings of IEEE Visualization, pp. 94-100, 2003. 8. L.A. Lima, and R. Bastos, “Seismic Data Volume Rendering”, Technical Report TR 98004, Department of Computer Science, University of North Carolina at Chapel Hill, 1998. 9. C. Ma, and J. Rokne, “3D Seismic Volume Visualization”, In Integrated Image and Graphics Technologies, Kluwer Academic Publishers, pp. 241-262, 2004. 10. Magic Earth: GeoProbe. (http://www.magic-earth.com/geo.asp). 11. Paradigm: Voxelgeo. (http://www.paradigmgeo.com/products/voxelgeo.php). 12. S. Roettger, S. Guthe, D. Weiskopf, and T. Ertl, “Smart Hardware-Accelerated Volume Rendering”, In Proceedings of Symposium on Visualization, IEEE, pp. 231-238, 2003. 13. T. Ropinski, and K. Hinrichs, “Real-Time Rendering of 3D Magic Lenses having arbitrary convex Shapes”, International Winter School of Computer Graphics, Journal of WSCG, 12(2), pp. 379-386, 2004. 14. SEG Technical Standards Committee, “SEG Y rev 1 Data Exchange Format”. Society of Exploration Geophysicist, Editors Michael W. Norris and Alan K. Faichney, 2002. 15. T.M. Sheffield, D. Meyer, J. Lees, G. Kahle, B. Payne, and M. J. Zeitlin, “Geovolume visualization interpretation: Color in 3-D volumes”, The Leading Edge, pp. 668-674, 1999. 16. J. Viega, M. Conway, G. Williams, and R. Pausch, “3D Magic Lenses”. In Proceedings of UIST, pp. 51-58, 1996. 17. D. Weiskopf, K. Engel, and T. Ertl, “Interactive Clipping Techniques for Texture-Based Volume Visualization and Volume Shading”, In Transactions on Visualization and Computer Graphics, Vol. 9, No. 3, pp. 298-312, 2003. 18. R.H. Wolfe, and C.N. Liu, “Interactive Visualization of 3D Seismic Data: A Volumetric Method”, In Computer Graphics and Applications, Volume 8, Issue 4, pp. 24-30, 1988.

Compressed Domain Image Retrieval Using JPEG2000 and Gaussian Mixture Models Alexandra Teynor1 , Wolfgang M¨ uller2 , and Wolfgang Kowarschick3 1

Albert-Ludwigs-University of Freiburg, Institute for Pattern Recognition and Image Processing, 79110 Freiburg, Germany [email protected] 2 Bamberg University, LS Medieninformatik, 96045 Bamberg, Germany [email protected] 3 Augsburg University of Applied Sciences, Department of Computer Science, 86169 Augsburg, Germany [email protected]

Abstract. We describe and compare three probabilistic ways to perform Content Based Image Retrieval (CBIR) in compressed domain using images in JPEG2000 format. Our main focus are arbitrary non-uniformly textured color images, as can be found, e.g., in home user image collections. JPEG2000 offers data that can be easily transferred into features for image retrieval. Thus, when converting images to JPEG2000, feature extraction comes at a low cost. For feature creation, wavelet subband data is used. Color and texture features are modelled independently and can be weighted by the user in the retrieval process. For texture features in common databases, we show in which cases modelling wavelet coefficient distributions with Gaussian Mixture Models (GMM) is superior in to approaches with Generalized Gaussian Densities (GGD). Empirical tests with data collected by non-expert users evaluate the usefulness of the ideas presented.

1

Introduction

Content Based Image Retrieval (CBIR) has been motivated many times. While there are currently great advances in semantic features and image annotations, the proliferation of digital cameras operated by annotation-lazy novices motivates the automatic, feature based CBIR today. JPEG2000 [2] compressed domain CBIR draws its interest from the fact that probabilistic texture modelling techniques have been developed based on wavelets. In this paper, we show how to compute and compare color and texture features based on wavelet subband data provided by JPEG2000. The main focus of this work is to show how subband data can be modelled accurately in order to capture the properties of non-uniformly textured images better. Work already done in this area mainly focuses on uniform texture images [9][16], or homogeneous areas in general images [4]. However, especially personal image collections do not consist of pure texture images, nor are users always interested in only S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 132–142, 2005. c Springer-Verlag Berlin Heidelberg 2005 

Compressed Domain Image Retrieval Using JPEG2000 and GMM

133

partial matches. An other way to use JPEG2000 data for image retrieval was described by Mandal et al. [12]. They propose two JPEG2000 indexing techniques based on significant bitmaps and the properties of bitplanes used to encode individual code blocks, however using subband data directly offers more information. The outline of this paper is as follows: Section two deals with color and texture features derived from subband data of JPEG2000 images. In section three, evaluation methods are described and results are shown. Section four concludes this work.

2

Features from JPEG2000 Data

Minka and Picard state in [13] that searching for the ultimate feature that represents an image is usually not successful, instead, it is better to rely on a number of primitive features and to combine them. We follow this approach. For arbitrary image databases of natural scenes, color and texture features are considered most important. We compute and compare them independently, getting two individual rankings for the best matches. To get the final result, the scores of each image in both rankings are combined. The default is to consider color and texture equally important, but the user has the possibility to emphasize either part by specifying a weight factor. 2.1

Data for Computing Features

The JPEG2000 image compression standard is rather complex and consists of different steps, details can be found, e.g., in [2]. During en-/decoding, there exists a state where the data is especially suited for computing features for CBIR. We extract features after the three color layers Y Cb Cr have been wavelet transformed (each layer independently). Each color layer is represented by a number of subbands, depending on the number of wavelet decomposition steps. Layer Y carries the luminance information of the image. It can be interpreted as a black and white representation of the original image. Layers Cb and Cr account for chrominance. A graphical representation of the wavelet coefficients from a sample layer Y with seven subbands in different orientations is shown in figure 1. For computing features, a JPEG2000 image has to be partially decompressed until it is in the format described above. A more efficient way would be to perform feature extraction right at compression time. 2.2

Color Features

On the lowest resolution subbands (LL0 ) of layers Y , Cb and Cr , only low pass filtering and down sampling was performed during wavelet transform. Hence we can interpret these three subbands as a down scaled version of the original image in Y Cb Cr space. This data can be used to create a color histogram [15] by assigning the color values to 5x8x8 bins. The Y -component has a coarser binning, since we want to be robust against small illumination changes. A similar method

134

A. Teynor, W. M¨ uller, and W. Kowarschick

Fig. 1. Wavelet coefficients and their distribution for different subbands

for creating color features for JPEG2000 images was proposed by [4], however they used more and equal sized bins for all color layers. In order to find out the color similarity of two images I1 and I2 , the histograms are compared using the well-known histogram intersection (HI). 2.3

Texture Features with Generalized Gaussian Densities

The higher order subbands represent horizontally and/or vertically high pass filtered data and therefore the “details” of the image. As shown in [9], the distribution of wavelet coefficients belonging to different subbands can represent the texture of images. In figure 1 histograms of subband data of the example image can be seen. For texture analysis, it is sufficient to consider only subbands of layer Y , being a black and white version of the image. Do and Vetterli proposed in [9] to model the distribution of wavelet coefficients by a Generalized Gaussian Density (GGD): −(|x|/α) β p(x; α, β) = 1 e 2αΓ ( β )

β

(1)

∞ where Γ (.) is the Gamma function: Γ (z) = 0 e−t tz−1 dt, z > 0 Here only two parameters α and β are necessary to describe the distribution. This is a very compact representation compared to, e.g., the histogram, where a large number of bins would be necessary to model the distribution satisfyingly. As described in [9], a way to estimate the parameters α and β in a statistical framework is by a maximum-likelihood (ML) estimator, which can be computed efficiently. The similarity of two GGD can be determined using the Kullback-Leibler divergence or relative entropy [9]. In general terms, the KLD between two probability density functions (PDFs) p1 (X) and p2 (X) is defined as:

KLD[p1 (X)||p2 (X)] =

p1 (x)log

p1 (x) dx p2 (x)

(2)

For the KLD on GGDs there exists a closed form expression, where only the model parameters α and β are involved [9].

Compressed Domain Image Retrieval Using JPEG2000 and GMM



135



β1 α2 Γ (1/β2 ) β2 α1 Γ (1/β1 ) β2 α1 Γ ((β2 + 1)/β1 ) + α2 Γ (1/β1 ) 1 − β1

KLD[p(X; α1, β1 )||p(X; α2 , β2 )] =log

(3)

Assuming that the wavelet coefficients in different subbands are independent, we can compute the overall texture similarity by applying the chain rule [9]. This means, in order to compute the overall KLD between images I1 and I2 , we can simply sum the KLD between corresponding subbands: KLD[I1 ||I2 ] =

N 

KL[pj1 (X)||pj2 (X)]

(4)

j=1

with pji denoting the PDF of the j-th subband of image i and N the number of subbands. In this work we assume that the number of decompostion levels for all images in the entire image database is the same, since the different subbands are matched one to another. We examined the fitting quality of GGDs to real wavelet coefficient distributions in different cases. We performed tests with data from very coarse images, very smooth images and images that contain either part. For uniformly textured images, the fidelity is usually well, however it is not sufficient for images that have distinct regions. Especially affected are images that contain hard partitioned smooth and coarse regions, e.g., a picture of a landscape with a sky, or an object in front of a uniform background. However, these types of images are common in standard user image databases. Figure 2 shows sample images for three test cases, each with a wavelet coefficient distribution of a characteristic subband next to it. The dashed blue curves show how well the GGDs model the subband data distribution, respectively. The problem becomes evident when looking at the image with distinct regions. The wavelet coefficients belonging to the sky are close to zero and are forming a peak, the ones representing the crowd are very wide spread and form a kind of base. The GGD can not model this kind of distribution accurately.

2.4

Wavelet Coefficient Distribution with Gaussian Mixture Models

Because of this fitting inaccuracy, we propose to model the wavelet coefficient distribution as Gaussian Mixture Model (GMM). A similar proposition was also made by Crouse et al. [6] who use a two state zero mean Gaussian mixture model to build a hidden Markov model for characterizing wavelet transformed signals. Do et al. [8] also used these hidden Markov models for texture description. In our case, we do not use hidden Markov models, but a single GMM for each subband directly.

136

A. Teynor, W. M¨ uller, and W. Kowarschick Fitting quality 0.16 GGD GMM 0.14

0.12

0.1

0.08

0.06

0.04

0.02

0 −150

−100

−50

0

50

100

150

Fitting quality 0.25 GGD GMM

0.2

0.15

0.1

0.05

0 −150

−100

−50

0

50

100

150

Fitting quality 0.16 GGD GMM 0.14

0.12

0.1

0.08

0.06

0.04

0.02

0 −150

−100

−50

0

50

100

150

Fig. 2. Wavelet coefficient distribution of sample subbands, fitted with GGD and GMM with 2 components. Images are from the Benchathlon collection [1].

A mixture density has the form p(x; Θ) =

N 

wi pi (x; θi )

(5)

i=1

N where the parameters are Θ = (w1 , . . . , wN ; θ1 , . . . , θN ), such that i=1 wi = 1, and each pi is a density function parameterized by θi . The functions used can be any valid probability density function, i.e. any set of non-negative functions integrating to one. In the case of Gaussian mixture models, Gaussian probability density functions are used.

Compressed Domain Image Retrieval Using JPEG2000 and GMM

137

The parameters wi and θi (here μi and σi ) can be estimated using the Expectation-Maximization (EM) algorithm [5]. It is an iterative procedure that needs good starting values and the number of mixture components necessary to model the distribution accurately. In a general setting this is a problem. But we already have knowledge how the mixture model to be estimated looks like. The wavelet coefficient distribution of a subband of a natural image is centered about zero and approximately symmetric (see figure 2). This means a small number of Gaussian mixtures with mean μ ≈ 0 will be sufficient. Experiments showed that already GMMs with two components model problematic wavelet coefficient distributions better than GGDs. In figure 2, the fitting quality of GMMs for different subband data can be viewed (solid red curves). In this experiment, we used GMMs with two components. In the first two cases, where the GGD already performed well, the fidelity of the GMM is comparable. In the third case, modelling the subband data with GMMs is much more accurate compared to using GGDs. In contrast to GGDs, there does not exist a simple way to compute the KLD on GMMs. A general approach is to use a Monte-Carlo (MC) estimation to approximate the integral in equation 2 [7]. Then, the KLD between the mixture densities p1 (X) and p2 (X) is estimated by: KLD[p1 (X)||p2 (X)] ≈

1 Nrnd

N rnd  i=1

log

p1 (xi ) p2 (xi )

(6)

where the sample values {x1 , . . . , xNrnd } are drawn randomly and independently from the model density p1 (X). The number of Nrnd has usually to be large which spoils retrieval time, however, we get improved results. Experiments showed that 60 to 100 samples are sufficient. Several scientists [7,11,10] have proposed ideas to compute the similarity between GMMs more efficiently. We used the approach described by Goldberger et al. [10]. They state different approximations of the KLD between mixture densities. As our GMMs have the same number of components, we used the method for calculating an upper bound for the KLD on mixture models. Details on this can be found in [10]. In the following, we refer to this method as “GMM-UB”.

3

Experimental Evaluation

In order to compare the efficiency of the methods described above, we implemented a test framework. As image database we used a subset of the Benchathlon collection [1] (4501 images) which consists of consumer photos. Since there is no official ground truth available, we collected them by ourselves. We designed a tool for acquiring ground truth by non expert users1 . The test persons were shown a query image and had to look through the whole database in order to look for similar images. The subjects had no previous experience with image 1

This tool is available under http : //muscle.prip.tuwien.ac.at/sof tware here.php

138

A. Teynor, W. M¨ uller, and W. Kowarschick

retrieval systems. They were told to select images that they wanted to have retrieved by a system when presented the query image. So this ground truth can be considered quite hard, since not only visually similar images were selected, but also semantically similar ones. We acquired ground truth for a set of 11 randomly selected images by three different test persons, since similarity tends to be judged differently by different persons. For evaluation we generated precision/recall plots, averaged over all three test persons and query images. Precision and recall are defined as [14]: precision =

recall =

|R ∩ T| |T|

|R ∩ T| |R|

(7)

(8)

where R is the set of images that are relevant to the query, T is the set of returned images and |A| is the cardinality of set A. 3.1

Evaluation of Features and Measures

The procedure for evaluating the algorithms was as follows: As explained in section 2, we calculated two different feature sets, one for color and one for texture. In order to access the wavelet coefficients in the JPEG2000 images, we used a modified version of the JasPer JPEG2000 codec [3]. Separate queries for color and texture similarity were performed implicitly and the individual rankings combined on the basis of the score of each image. As score of an image, a weighted sum of the normalized texture and color similarity measure is used. For color, this is the normalized HI value, for texture it is 1/(1 + KLD). These values are always between 0 and 1. The more similar an image is regarded by the system, the closer its value is to 1, with the query image itself having the maximum score of 1. As we evaluate arbitrary color pictures, equal weighting of the different feature types was performed. The texture features and the way their similarity is determined change in every experiment, color features however stay the same and thus will not be mentioned explicitly. When calculating the graphs for the MC method, the retrieval process was performed 3 times and the results averaged, due to the random nature of MC integration and the therefore slightly variable results. As could be seen in figure 2, the modelling of the data only improved for images with hard partitioned textured and smooth areas. So a significant improvement is only expected here. For this reason, we evaluate the results for the “crowd and sky” image independently from the other images in the ground truth. Figure 3 shows a screen shot of the 10 best matches for images with texture modelled with GGDs and GMMs respectively. In the GMM-MC case, the matching was performed with Monte Carlo integration with 60 samples (GMM-MC60). As we can see, the results improved noticeable. A more precise evaluation can be done by looking at the precision/recall plots. In figure 4 on the left, the retrieval results for the different experiments

Compressed Domain Image Retrieval Using JPEG2000 and GMM

139

GGD

GMM-MC-60

GMM-UB

Fig. 3. Example query results. The query image is in the upper left corner.

are displayed. In this example, the GMM-based queries (GMM-UB and GMMMC60) performed better than the GGD based one, since the precision is higher for the same level of recall. The best result was achieved by the GMM-MC method, because it has the highest recall values at the beginning of the precision/recall plot. This means that after few images retrieved, more relevant images are found compared to the other methods (the user wants to browse as few images as possible). The average result for all images in the ground truth can be viewed in figure 4 on the right. Here also the advantage of the GMM method can be seen, however not that clearly, since many of the randomly selected query images do not have a problematic distribution. The resources needed by the different algorithms vary. In the GGD case, two parameters (α, β) describe the wavelet coefficient distribution per subband, in the GMM case we need 3 × N with N being the number of mixture components

140

A. Teynor, W. M¨ uller, and W. Kowarschick benchathlon 4501, result for image "crowd and sky"

benchathlon 4501 database, all ground truth GGD GMM−MC60 GMM−UB

1

0.8

precision

precision

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0

GGD GMM−MC60 GMM−UB

1

0

0.1

0.2

0.3

0.4

0.5 recall

0.6

0.7

0.8

0.9

1

0

0

0.1

0.2

0.3

0.4

0.5 recall

0.6

0.7

0.8

0.9

1

Fig. 4. Results for the problematic image and the entire ground truth

(2 in our case). Since the JPEG2000 images used had 5 decomposition levels, the number of coefficients to characterize texture information in the image is 12 × 2 = 24 for GGDs and 12 × 6 = 72 for GMMs (we have 4 × 3 high pass subbands). More severe are the differences in matching time for an image pair: for the calculation of the KLD on GGDs a single expression needs to be evaluated (see equation 3), so this is rather quick. When using the MC approximation, the retrieval time depends linearly on Nrnd , the number of random samples drawn. The calculation speed of the KLD for GMMs with an upper bound depends on the number of mixture components N . As described in [10], in order to obtain a tight upper bound for mixture models with equal N , minimizing over N ! permutations of the mixing components is necessary. However, since we use only N = 2 Gaussians, this is much less than the MC integration. The absolute retrieval times for a texture query in our java framework on a P4 2.8GHz machine for 4501 images were about 600 ms for the GGD model, 14000 ms for GMM model with MC computiaton with 60 samples and 900 ms for the GMM-UB method. The evaluation time for the color features is the same in all cases. There the query time depends on the number of non-empty bins to be compared for histogram intersection, on average this was about 700 ms. We see, the computation of the KLD on GMMs with an upper bound is only 50% slower as computation of GGD similarity, while the method with MC integration with 60 samples takes over 23 times as long. As the performance of the GMM-UB method is almost as good as the GMM-MC method, it should be preferred.

4

Conclusions

In this paper, we presented and evaluated methods for image retrieval with features derived from JPEG2000 wavelet subband data. In particular, color and texture features were computed, which represent characteristics of the image well. We illustrated that for certain images, the modelling of texture features by

Compressed Domain Image Retrieval Using JPEG2000 and GMM

141

GGDs has weaknesses and proposed to model them by GMMs. As shown, the probabilisitc modelling of the data in terms of distributions makes it easy and efficient to compare collections. Approximate evaluation of KLD between GMMs showed good results for problematic images, combined with fast evaluation. To further improve speed and retrieval quality, methods should be researched how to choose the most suitable method for modelling subband data automatically according to the data distribution.

Acknowledgment This work was partially funded by the German Ministry for Education and Research (BMBF) through grant FKZ 01IRB02B. We would also like to thank Neuland Multimedia GmbH for providing their interns for the collection of the relevance data.

References 1. The Benchathlon collection. http://www.benchathlon.net/img/done/. 2. ISO/IEC 15444-1:2000, JPEG 2000 image coding system – part 1: Core coding system, edition 1. 3. M. D. Adams. Jasper: A software-based JPEG2000 codec implementation. http://www.ece.uvic.ca/ mdadams/jasper/. 4. J. Bhalod, G. F. Fahmy, and S. Panchanathan. Region based indexing in the JPEG2000 framework. In Internet Multimedia Management Systems II, SPIE, 2001. 5. J. A. Blimes. A gentle tutorial on the EM algorithm and its application to parameter estimation for gaussian mixture and hidden markov models. Technical Report ICSI-TR-97-021, University of Berkeley, 1997. 6. M. S. Crouse, R. D. Nowak, and R. G. Baraniuk. Wavelet-based statistical signal processing using hidden markov models. IEEE Transactions on Signal Processing, 46, 1998. 7. M. N. Do. Fast Approximation of Kullback Leibler Distance for Dependence Trees and Hidden Markov Models. IEEE Signal Processing Letters, 10:115–118, April 2003. 8. M. N. Do and M. Vetterli. Rotation Invariant Texture Characterization and Retrieval Using Steerable Wavelet Domain Hidden Markov Models. In IEEE Transactions on Multimedia, volume 4, pages 517–527, 2002. 9. M. N. Do and M. Vetterli. Wavelet-Based Texture Retrieval Using Generalized Gaussian Density and Kullback-Leibler Distance. In IEEE Transactions on Image Processing, volume 11, pages 146–157, February 2002. 10. J. Goldberger, H. Greenspan, and S. Gordon. An efficient similarity measure based on approximations of KL-divergence between two Gaussian mixtures. International Conference on Computer Vision (ICCV), 2003. 11. Z. Liu and Q. Huang. A new distance measure for probability distribution function of mixture type. ICASSP-2000 Istanbul, Turkey. 12. M. K. Mandal and C. Liu. Efficient image indexing techniques in the JPEG2000 domain. Journal of Electronic Imaging, 13, 2004.

142

A. Teynor, W. M¨ uller, and W. Kowarschick

13. T. Minka and R. Picard. Interactive learning using a ‘society of models’. In Proceeding of IEEE Conference on Computer Vision and Pattern Recognition (CVPR1996), pages 447–452, 1996. 14. H. M¨ uller, W. M¨ uller, D. M. Squire, S. Marchand-Maillet, and T. Pun. Performance evaluation in content-based image retrieval: Overview and proposals. Pattern Recognition Letters, 22(5), April 2001. 15. M. J. Swain and D. H. Ballard. Color indexing. Int. J. Comput. Vision, 7(1):11–32, 1991. 16. Z. Xiong and T. S. Huang. Subband-based, memory-efficient JPEG2000 images indexing in compressed-domain. Fifth IEEE Symposium on Image Analysis and Interpretation (SSIAI’02), 2002.

Indexing and Retrieving Oil Paintings Using Style Information Yan Yan and Jesse S. Jin School of Design, Communication and I.T., The University of Newcastle, Callaghan 2308, Australia {yan.yan, jesse.jin}@newcastle.edu.au

Abstract. In this paper we discuss the principle of color perception in oil paintings, and analyze the existing color spaces used in digital image processing. Because of the complexity of color perception, there is no single color space which can represent various properties of color perception and provide a consistent retrieval capability in indexing oil paintings. We propose a new color space for indexing and retrieving oil paintings, which is based on pigment color mixing and the psychology of seeing. The new space includes seven color elements: pure color, white, black, tint, tone, shade and grey. Finally, six semantic query categories are defined in accordance with six color arrangements in oil paintings. Keywords: Color circle, color space, additive color mixing, subtractive color mixing, partitive color mixing.

1 Introduction Color is one of the important low-level features frequently used in content-based image retrieval [14, 13, 24, 18, 10, 8]. Using color in retrieval involves two issues, the color space and the color features. Color space is a one to three dimensional coordinate system for representing color in terms of intensity values. Widely used color spaces include RGB (red, green, blue), CMY (cyan, magenta, yellow), HSV (hue, saturation, value), CIE Luv, CIE Lab, Munsell, etc. It has been shown that hue is invariant under highlights, shadowing, and geometry changes of viewing and illumination angles [8]. Usually less importance is given to value dimension, as human vision is less sensitive to it [25]. In 1976, the CIE defined two new color spaces, CIE Lvu and CIE Lab, to enable us to get more uniform and accurate models. L component defines the luminancy, and u and v define chrominancy. CIE Luv is used in calculation of small colors or color differences, especially with additive colors. As CIE Luv, CIE Lab is a color space incorporated color space in TIFF specs. L is the luminancy, a and b are respectively red/blue and yellow/blue chrominancies. CIE Luv and CIE Lab color spaces have also been used in some image retrieval systems [19, 14, 9]. These color spaces are constructed to be perceptually uniform and device independent. The color features are extracted from a particular color space. Many color features have been proposed. The simplest form of color representation is histogram [27]. The similarity of two color histograms can be compared by simple Euclidean S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 143 – 152, 2005. © Springer-Verlag Berlin Heidelberg 2005

144

Y. Yan and J.S. Jin

distance. Histogram intersection is also proposed as the measure of distance by Swain and Ballard [27]. Other distance measures suggested for color histograms include quadratic-form distance [11] and the earth mover’s distance [16]. Another color feature is color moments, which are the statistical distribution of colors [26]. Only the first moment (mean), the second (variance) and the third (skewness) moments are extracted for color indexing. Color moments are more robust in CIE Luv and Lab spaces than in HSV space. Smith and Chang [21] proposed color sets as an approximation to color histogram in order to facilitate color-based retrieval in large-scale image databases. Pass et al. [15] proposed a color representation method using the color coherence vectors (CCV). Color correlogram [12] incorporates the spatial information into color features. Funt and Finlayson [7] developed a color constant indexing which is insensitive to illumination. Drew et al. [2] provided a liner transform among RGB channels that can make color representation illumination invariant. In Gevers and Smeulders [10], they proposed a color representation method which is viewpoint invariant. A more recently study on color invariance can be found in Geusebroek et al. [8]. It is also well known that there is a gap between low-level features and high level semantics. Many approaches have been proposed to bridging the gap, such as using domain knowledge [18, 29] or using interactive relevance feedback of users [17, 20, 28]. In some limited domains, the high-level concepts can be easily mapped to the low-level features, e.g. mapping an apple to red color and round shape [18]. In most cases, there is no such direct mapping from the high-level concepts to the low-level visual features. State-of-the-art surveys of CBIR techniques and systems can be found in Rui et al. [17], Smeulders et al. [20], and Veltkamp and Tanase [28]. Color perception is a psychophysical process, which involves complicated information cognition and processing. From the antiquity to modern time, there was a long history of color perception and representation. In the ancient Greek, Aristotle was probably the first person to study the order of colors and devised the linear sequence of colors arising from darkness at night to brightness in the day. In 1704, Newton created the first spectral color circle. From then, the study of color systems became more systematic and professional. Oil paintings have a long history which is closely associated with human understanding of color perception. In this paper, we present a new scheme for indexing and retrieval oil paintings using color. We propose a new color space based on the combination of HSV and CIE Luv. The new color space incorporates and displays the visual and psychological relationships of colors in accordance with the practicalities of artists. Furthermore, the color space provides more effective ways for indexing and retrieval oil paintings by their semantic color features. The paper is organized as follows. Section 2 describes the principle of color perception in oil paintings. Section 3 discusses our color feature. Several semantic retrieval categories are given in Section 4. Finally, there is a conclusion.

2 The Principle of Color Perception in Oil Paintings Color perception is a very complex topic. Under the daylight, paints selectively absorb part of the light energy and remit part in all directions. The reflected light enters

Indexing and Retrieving Oil Paintings Using Style Information

145

Fig. 1. Seeing process and color mixing

the eye of the viewer, and then the viewer’s eye interprets this reflected light as color. Figure 1 indicates the seeing process and relative color mixing theories. “Paints”, “Light” and “Vision” colors are essentially different. 2.1 Color in Oil Paintings Several aspects of color have been shown in oil paintings: • Paint mixtures are subtractive because of the light absorbing behavior of paints. Magenta, yellow and cyan are three paint primaries that will form other hues under normal conditions. Such mixture darkens the reflected color because each substance subtracts light reflectance from the others. Therefore, a combination of all three paint primaries will form black or a deep brown. • Light mixtures are additive and based on unequally stimulating the three types of color receptor cones by different wavelengths. Three light primaries are red, green, and blue. Mixing the three will add their luminosity to produce the perception of white. • In vision, there are four elementary color sensations or psychological primaries, red, yellow, green and blue. Vision color mixtures are formed by the proportional average of optical mixing [6]. Equal mixing all four elemental colors will form grey. 2.2 Pros and Cons of Various Color Spaces Color spaces are geometrical representations of the relationships among possible colors of light, vision or paint. There are many color spaces currently in use by scien tists and artists, for example, RGB, HSV, Munsell, NCS, etc. For indexing and retrieving oil paintings, a color space should meet some specific requirements: • The color circle according to subtractive color mixing theory: Color harmonies are based on the reasonable arrangement of colors. The foundation of oil painting’s color circle is the subtractive “primary” triad that is composed of magenta, light yellow, and cyan [3]. • Equal distribution of warm and cool colors in the color circle: From the biological structure of human eye, there are unequal proportions of R (red), G (green) and

146

Y. Yan and J.S. Jin

B (blue) cones in the retina. 64% color receptors are R cones, 32% are G cones, and 4% are B cones [1]. Because the “warm” color bias in our color experience, most artists will give more importance to warm colors than to cool ones. Therefore, oil painting color space should have a color circle with equal warm and cool regions. • Color definition: Primary, secondary and tertiary colors should be as saturated as possible. Specifying color names will avoid the confusion of color conceptions. The definition should be obtained by clustering • Representation of the visual and psychological relationships of colors: Oiling painting color space should provide simple way to survey and descript the vast world of color. Bearing in mind, beauty is the result of good ordering of colors. Three color spaces are compared and analyzed in Table 1. Table 1. Comparison of color spaces

Color Space CIELab

Munsell

NCS

Features red, green, blue and yellow are basic colors; Cube modified according to psychometric color diagram Red, yellow, green, blue and purple are basic colors; Color-tree form

Red, yellow, blue and green are basic colors; Double cone form

Advantages

Disadvantages

Partitive vision mixture; Treat all colors as a combination of surface color and illuminating color; Color space perceptually uniform; Represent the y/b and r/g opponent dimensions.

partitive vision mixing is different from the subtractive paint mixing; Goo determination of color difference, but lack of perceptual and psychological relationship of colors.

Based on perceived equidistance and psychological color arrangement; Color circle traces to the primaries of physics; Color space is based on the laws of brightness and color perception.

Enlarge the cool color region in the color circle; Difficult to understand Value and saturation attributes; Hard to trace oil painting color expressions and harmonies. NCS color circle corresponds to visual color theory (partitive), not paint color theory (subtractive); Color proportional expression results in color saturation costs [3]

Enable users with normal color vision to determine colors; Four psychological basiccolors form a partitive color circle; All tints, shades and tones of the hue contain in the boundaries of the triangle.

Indexing and Retrieving Oil Paintings Using Style Information

147

3 A Color Space for Indexing and Retrieving Oil Paintings Color spaces are geometrical representations of the relationships among possible colors. In this paper, we propose a new color space based on the combination of HSV and CIE Luv. The HSV model describes colors in an intuitive way. Three color variants, hue, saturation and value give a close relation to human visual perception. The calculation of chrominancies in CIE Luv provides an accurate division of various colors. The color features we extracted from the new space are pure color, white, black, tint, tone, shade and grey. These components have been widely used in oil paintings. The color theory was created in 1978 by Alvy Ray Smith based on tint, shade and tone color theory in art. For indexing and retrieving oil paintings, we propose to cluster HSV space using the Birren Color Triangle [5], which can be used to trace six paths to color expression by six great masters of oil paintings. • Color triangle: Faber Birren introduced the triangular arrangements, which incorporate and display the visual and psychological relationships of colors. When the basic elements (pure color, white and black) are combined, four secondary forms are created to form tints, shades, tones and grays. All color variations and modifications will be classified within the boundaries of color triangle. This color triangle is indicated in Figure 2a. • The color triangle formulas: They are given for the development of color scales that follow straight paths on the triangle. The first numeral in each formula refers to pure color content, the second one to white content and third one to black content. Hue content is the sum left over to equal 100 and the resultant visual mixture is matched with paints. Thus, 100-0-0 refer to a pure hue; 75-25-0 a clean tint with no black in it; 25-0-75 a rich shade with no white in it; and 40-10-50 a tone with 40% pure color, 10% white and 50% black, as shown in Figure 2b.

(a)

(b)

Fig. 2. Color triangle (a) and formation formula (b)

4 Semantic Color Retrieval Based on the Oil Painting Styles The straight-line sequences of the color triangle in the oil painting color space are all natural and concordant. Following any path, harmony results are obtained because related visual elements are involved. Tints harmonize with pure color and white because they contain both. Shades harmonize with pure color and black for a similar

148

Y. Yan and J.S. Jin

reason. Tone is the coordinating or integrating form [4]. Based on the harmony of oil paintings, six styles have been defined in oil paintings. We propose six semantic retrieval categories from these styles. 4.1 Chiaroscuro Style Color harmony exists in tint, tone, and shade. The color form identifies the famous chiaroscuro style of painting invented by Leonardo da Vinci and adopted by eminent artists during the Renaissance and after- up to the time of Impressionism. In chiaroscuro style, highlight becomes purer, but not chalky; shadow becomes richer, but not flat. Transferring from highlight to shadow is achieved by adjusting the amount of white and black in the tone, as shown in Figure 3a. Figure 3b illustrates the distribution of color components in our visual retrieval, ie, black, white, gray, color, tint, shade and tone. The distribution shows a significant amount of tint, shade and tone.

25000

20000

15000

10000

5000

t

ne to

r

ad e sh

lo

tin

co

k

hi te w

bl ac

(a)

gr ay

0

(b)

Fig. 3. (a) Mona Lisa (Leonardo da Vinci 1452-1519); (b) distribution of color components

4.2 Greco Style There is a combination of shade, tone, and white. In this arrangement the deeper value is pure and the lighter value grayish. It is undoubted that this combination tends to be “dry” and “dusty” in impression. However, El Greco, who was a Spanish painter, perfectly used this unnatural color expression to create his religious subjects, as shown in Figure 4a. Figure 4b illustrates the distribution of color components in Figure 4a. The distribution shows a significant amount of gray, shade and tone. 4.3 Turner Impressionism Style There is color expression that runs from pure light tints or pastels to grayish tones. The new principle of color arrangement was firstly used by J.M.W. Turner and influenced Impressionism and much of modern art expression. He discovered colors could be rendered to appear amazing luminous, not by contrasting them with black and deep shade, but with grays and softly hue tones, as shown in Figure 5a. By keeping his lights pure in hue and featuring them against muted tones of medium brightness, the fugitive radiances of dawn of sunset were gained [5]. Figure 5b illustrated the distribution of color components of Figure 5a.

Indexing and Retrieving Oil Paintings Using Style Information

149

80000 70000 60000 50000 40000 30000 20000 10000 0 black white gr ay color

tint

shade tone

(a)

(b)

Fig. 4. (a) Christ driving the traders from the temple (El Greco ca. 1548-1614); (b) distribution of color components

300000

250000

200000

150000

100000

50000

0 bl ack whi te gr ay col or

tint shade tone

(a)

(b)

Fig. 5. (a) The fighting temeraire (J. M. Turner 1775-1851); (b) distribution of color components

250000

200000

150000

100000

50000

0 bl ack whi te gr ay col or

(a)

tint shade tone

(b)

Fig. 6. (a) A Wheatfield, with Cypresses (Vincent van Gogh 1853 - 1890); (b) distribution of color components

4.4 Neo-impressionism and Post-impressionism Style The Impressionists and Neo-Impressionists entirely devoted to the combination of pure color, tint and white. They glorified the phenomena of light and used spots of color in an

150

Y. Yan and J.S. Jin

attempt to achieve luminous visual mixtures. Many of them refined their color palettes in spectral hues and avoided black and brown [5]. Vincent van Gogh is one of those remarkable impressionism masters who painted in that color style, as shown in Figure 6a. Figure 6b illustrated the distribution of color components of Figure 6a. 4.5 Modern Style The modern painters use this color expression, which involves seven color forms in the color triangle. Along with recent development in the manufacture of pigments, artists have unlimited color choices and color expressions. Where painting is abstract or non-objective, natural phenomena can be disregarded for more personal and creative achievements, as shown in Figure 7a. Kandinsky was one of the first and greatest of the abstract painters, one of the first to use colors and forms to convey emotions and moods without any representational elements [5]. Figure 7b illustrated the

50000 45000 40000 35000 30000 25000 20000 15000 10000 5000 0 black whi te gr ay

col or

tint

shade tone

(a)

(b)

Fig. 7. (a) Improvisation No.30 (Wassily Kandinsky 1866-1944); (b) distribution of color components

(a) 25000

45000

25000

25000

20000

20000

15000

15000

10000

10000

5000

5000

0

0

40000 20000

35000 30000

15000 25000 20000 10000 15000 10000

5000

to ne

tin t sh ad e

gr ay co lo r

bla ck wh ite

to ne

tin t sh ad e

gr ay co lo r

to ne

tin t sh ad e

gr ay co lo r

bla ck wh ite

to ne

tin t sh ad e

gr ay co lo r

bla ck wh ite

0

bla ck wh ite

5000 0

(b) Fig. 8. Correlation among Romanticism style

Indexing and Retrieving Oil Paintings Using Style Information

151

distribution of color components of Figure 7a. It contains the combination of rich color components, certainly more than style in early age of oil paintings. Note that the distribution of rich gray, tint, shade and tone indicates the color mixing capability of artists. They have not relied on pure color in their paintings. However, you can still perceive a colorful picture. 4.6 Intra-style Correlation The color feature we developed has shown a strong intra-style correlation. The 23 classes we tested contain 1503 paintings. Figure 8a shows four paintings from the same class. Their style histograms have shown highly correlated patterns.

5 Conclusion In this paper we review different color retrieval schemes. Due to the semantic-gap between color features and high-level concepts, retrieval using color is very much domain and task dependent. The semantics should be extracted directly from a specific task or an application. For indexing and retrieving oil paintings, a new color space should be proposed because there are differences among the light, paint, visual perception theories. Oil paintings have specific color arrangements and expressions. We propose a new color scheme which consists with painting scheme among artists. The preliminary results show the feasibility of the direction. Future work includes the correlation within the style and clustering boundaries of seven visual features.

References 1. Color Vision (2005). Light and the eye, http://www.handprint.com/HP/WCL/color1.html 2. Drew, M. S., Wei, J. and Li, Z. N. (1998). On illumination invariance in color object recognition, Pattern Recognition, 31(8):1077-1087. 3. Faber Birren (1961). Creative Color, Van Nostrand Reinhold Company, pp.17-22. 4. Faber Birren (1969). Principles of Color, Van Nostrand Reinhold Company, pp.31-65. 5. Faber Birren (1965). History of Color in Painting, Reinhold Publishing Corporation, pp.111-118. 6. Frans Gerritsen (1971). Theory and Practice of Color, Studio Vista Publishers, London. 7. Funt, B. V. and Finlayson, G. D. (1995). Color constant color indexing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(5):522-529. 8. Geusebroek, J. M., van den Boomgaard, R., Smeulders, A. W. M. & Geerts, H. (2001). Color invariance, IEEE Trans. on PAMI, 23(12):1338-1350. 9. Gevers, T. & Smeulders, A. W. M. (1996). Evaluation color and shape invariant image indexing of consumer photography. First Int. Conf. on Visual Information Systems, no. 1306 in Lecture Notes in Computer Science, Melbourne, Springer-Verlay. 10. Gevers, T. & Smeulders, A. W. M. (1999). Content-based image retrieval by viewpointinvariant image indexing, Image and Vision Computing, 17(7):475-488. 11. Hafner, J., Sawhney, H. S., Equitz, W., Flickner, M. and Niblack, W. (1995). Efficient color histogram indexing for quadratic form distance functions, IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(7):729-736.

152

Y. Yan and J.S. Jin

12. Huang, J., Kumar, S. R., Mitra, M., Zhu, W. and Zabih, R. (1997a). Image indexing using color correlogram, Proceedings of International Conference on Computer Vision and Pattern Recognition, pp.762-768. 13. Nagasaka A & Tanaka Y (1992). Automatic video indexing and full-video search for object appearances. Visual Database System, II, IFIP, Elsevier Science Publishers, pp.113127. 14. Niblack W, Barber R, Equitz W, Flickner M, Glasman E, Petkovic D, Yanker P, Faloutsos C & Taubin G (1993). The QBIC project: Querying images by content using color, texture and shape. Proc. SPIE, 1908:173-187. 15. Pass, G., Zabih, R. and Miller, J. (1996). Comparing images using color coherence vectors. Proc. of ACM International Conference on Multimedia, Boston, USA, pp.65-73. 16. Rubner, Y., Tomasi, C. and Guibas, L. J. (1998). The earth mover’s distance as a metric for image retrieval, Technical Report CS-TN-98-86, Stanford University. 17. Rui, Y., Huang, T. S. & Chang, S. F. (1999). Image retrieval: current techniques, promising directions and open issues, Journal of Visual Communication and Image Representation, 10:39-62. 18. Rui, Y., Huang, T. S., Ortega, M. & Mehrotra, S. (1998). Relevance feedback: a power tool in interactive content-based image retrieval, IEEE Transaction on Circuits and Systems for Video Technology, 8(5):644-655. 19. Sclaroff, S., Taycher, L. & La Cascia, M. (1997). ImageRover: A content-based browser for the world wide web. IEEE Workshop on Content-Based Access of Image and Video libraries, San Juan, Puerto Rico, pp. 2-9. 20. Smeulders, A. W. M., Worring, M., Santini, S., Gupta, A. and Jain, R. (2000). Contentbased image retrieval at the end of the early years, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12):1349-1380. 21. Smith, J. R. and Chang, S. F. (1995). Single color extraction and image query, Proceedings of IEEE international Conference on Image Processing, vol. 3, pp.528-531. 22. -Smith, J. R. and Chang, S. F. (1996). VisualSEEK: A fully automated content-based image query system, Proceedings of ACM Multimedia, Boston, MA, USA, pp. 87-93. 23. Smith, J. R. and Chang, S. F. (1997). Visual searching the web for content, IEEE Multimedia, 4(3):12-20. 24. Smoliar, S W & Zhang, H J (1994). Content-Based Video Indexing and Retrieval. IEEE Trans Multimedia 1:62-72. 25. Squire, D. M., Muller, W., Muller, H. & Raki, J. (1999). Content-based query of image databases, inspirations from text retrieval: inverted files, frequency-based weighs and relevance feedback. Proceedings of the 11th Scandinavian Conference on Image Analysis, Kangerlussuaq, Greenland, pp.143-149. 26. Striker, M. and Orengo, M. (1995). Similarity of color images, SPIE Proc. of Storage and Retrieval for Image and Video Databases, San Jose, CA, USA, vol. 2420, pp.381-392. 27. Swain, M. J. and Ballard, D. H. (1991). Color indexing, International Journal of Computer Vision, 7(11):11-32. 28. Veltkamp, R. and Tanase, M. (2000). Content-based image retrieval systems: A survey, Technical Report UU-CS-2000-34, Utrecht University. 29. Corridoni, J. M. Del Bimbo, A. & Pala, P. (1999). Image retrieval by color semantics, Multimedia Systems, 7:175-183.

Semi-automatic Feature-Adaptive Relevance Feedback (SA-FR-RF) for Content-Based Image Retrieval Anelia Grigorova and Francesco G.B. De Natale Dept. of Information and Communication Technology, University of Trento, Italy [email protected], [email protected]

Abstract. The paper proposes an adaptive retrieval approach based on the concept of relevance-feedback, which establishes a link between high-level concepts and low-level features. The user's feedback is used not only to assign proper weights to the features, but also to dynamically select them and to identify the set of relevant features according to a user query, maintaining at the same time a small sized feature vector to attain better matching and lower complexity. Results achieved on a large image dataset show that the proposed algorithm outperforms previously proposed methods. Further, it is experimentally demonstrated that it approaches the results obtained by optimum feature selection techniques having complete knowledge of the data set.

1 Introduction Content-based image retrieval (CBIR) systems are usually based on the description of images by low- and middle-level features [1], [2]. The retrieval algorithm simply matches these descriptions with a user query according to some similarity metric. The effectiveness of a CBIR system depends on the choice of the set of visual features and similarity metric that models user perception of similarity. A powerful mechanism for retrieval performance improvement by user interaction is relevance-feedback (RF) [3], [4]. Query shifting and feature relevance weighting are two main RF strategies proposed in the literature of CBIR. Image description is probably the most important aspect for the retrieval performance. Increasing the dimension of the feature vector strongly affects computation and storage requirements. Research has been done to deal with the problem of feature set selection and high dimensionality of the feature space. The feature set selection algorithms are based mainly on statistical pattern recognition classification techniques or artificial neural networks, some examples are [5] and [6]. For a thorough review on the subject, please consider [7]. By analyzing the behavior of feature re-weighting algorithms, it is possible to observe that after a few iterations several parameters become almost unused, and without additional information inclusion the algorithm converges to a suboptimal solution which cannot be further refined. In this work, the proposed objective is not S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 153 – 162, 2005. © Springer-Verlag Berlin Heidelberg 2005

154

A. Grigorova and F.G.B. De Natale

only to perform effective feature dimension reduction according to user’s feedbacks, but also to improve the image description during the retrieval process by introduction of new significant features for image description in the feature vector. Experimental results demonstrate that the proposed scheme achieves better results as compared to non-adaptive techniques. Section 2 describes RF scheme. In section 3 the featurespace transformation algorithm is described. In sections 4 and 5 the experimental setup is defined and the achieved results are analyzed and compared with state-of-art techniques. To further assess the performance of SA-FA-RF, comparisons were also done with the results achieved by applying some classical feature selection algorithms, on the hypothesis that the all pre-classified image database and a huge set of features are available a-priori.

2 The Relevance Feedback Scheme SA-FA-RF is a RF based approach [2], [3]. The Relevance Feedback is the process by which a retrieval system gathers information from its users about the relevance of previously retrieved objects. The goal is to adjust iteratively system’s parameters in order to achieve a better approximation of the user's information needs for the following search in the image database and to refine the subsequent retrieval results. There are many ways of using the information input by the user. Many approaches concentrate on query refinement and feature relative importance weights calculation [3], [4], [10] others propose to use the RF information either to learn a linear transformation in the feature space or the correlations among feature components, or for density estimation or classification [2]. In our work we apply query and feature weights iterative refinement in addition to the image description adaptation. The process is based on the information inputs by the user, the relevant and irrelevant image descriptions as well as on their number and proportions during RF iterations. The specification of the query is very critical in CBIR. If it is an image located near the boundary of the relevant region in the feature space the first retrieval could contain few relevant images. In this case, query refinement mechanism will be very useful for moving the query towards the region of relevant images. On the other hand when the query is able to find a significant number of relevant images, then it should be more effective to re-weight the features. In this case we set the query feature vector to the average of all relevant feature vectors (its optimality was proven by Ishikawa [8]). For query refinement we use the sets of relevant and non-relevant images (DR‘ and DN‘) obtained by the user’s input and we calculate the new query by Rocchio’s [9] formula: ' ' · § · § ¸ ¨ 1 NR ¸ ¨ 1 N N ¦ Fi ¸ - γ ¨ ¦ Fj ¸ ¨ N 'R i = 1 ¸ ¨ N 'N j = 1 ¸ © ¹ © ¹

Q′ = Į Q + ȕ ¨

(1)

where N’R and N’N are the numbers of documents in D’R’ and D’N and Fi and Fj the feature vectors of the relevant and irrelevant image sets respectively. α, β and γ are parameters (α + β + γ = 1 ) controlling the relative importance of the previous query point, the average

Semi-automatic Feature-Adaptive Relevance Feedback (SA-FR-RF)

155

of relevant images and the average of non-relevant images, respectively. As the relevance feedback iteration moves on Q’ approaches the optimal query. The feature re-weighting we use is based on a set of statistical characteristics [10]. It uses relevant images as well as irrelevant ones to associate and dynamically update variable weight to each feature, which define its relative importance. For a given query image, after the k-th relevance feedback iteration we calculate for the m-th feature a set of statistical characteristics - dominant range, confusion set and discriminant ratio įmk, which indicates the ability of this feature in separating irrelevant images from relevant ones. Denote the standard deviation of the feature in the relevant set as ımk,R we update the weights wmk by:

w

k +1 m

=

δmk σmk ,R

(2)

After the computation of the new query and the weighting vector, the matching is performed again. Every image and the query are represented by own feature vector vi and vq respectively. Having the weight vector w we use the weighted Euclidean distance as a measure of image similarity: WMSE =

1 2 ¦ (v − v ) w j N j qj ij

(3)

3 Feature Space Modification and the Feature Selection Problem In a CBIR system, the feature set must be able to discriminate two sets of images: relevant and irrelevant. It is difficult to state a-priori which feature is significant or less significant from the viewpoint of user perception. To overcome this problem we propose feature-adaptive RF strategy (FA-RF). During the retrieval process we adapt not only the query parameters and feature’s weights but also the set of image descriptors (number and type) in order to better satisfy user’s information needs. It integrates two different techniques in the iteration process: feature sorting on the basis of their discrimination capabilities and transformation of the feature space with dynamic updating of the feature set. In a pre-processing we extract from every image an extended set of parameters S. We start the retrieval with initial reduced set of color, luminance and edge bins - the composite histogram H1 ⊂ S [11]. After the first relevance feedback iteration we replace the features with null or almost null weights with new ones for better description of the features with sufficiently high weights. This leads to the definition of an updated histogram H2 ⊂ S. Since the old features are substituted to the new ones, the size of the vector is kept small, thus achieving a fast computation. Then the new histogram is weighted using the same information provided by the user at the previous iterations, without requiring a further feedback. To evaluate our method and to compare it with well-known feature selection algorithms we performed experiments with WEKA data mining tool, which is a comprehensive suite of Java class libraries that implement many state-of-the-art machine learning and data mining algorithms [12].

156

A. Grigorova and F.G.B. De Natale

Notwithstanding that the relevant images are similar from the user’s point of view, there is a difference in their visual description and some of them can be more informative and representative regarding the image class and therefore able to find more relevant images. Therefore in the new feature space we apply internal retrieval iterations using as a query every image from the relevant image set, collecting most similar images after each iteration. Thus, we obtain as many of the relevant images as possible before allowing next user’s interaction, in order to reduce the number of RF iterations. Experimental results demonstrated that the retrieval process is improved by the achieved semi-automatic feature adaptive RF based retrieval (SA-FA-RF). A formal description of the feature space modification algorithm follows:

Input: the initial histogram H1, the weight vector w, the features order Ord (the feature vector is sorted in decreasing weight order), the extended set of parameters S for i=1 to number_of_ features { if w(Ord(i)) >0 if the feature with number Ord(i) is a color feature then add the corresponding extended color description to H2 if the feature Ord(i) is a luminance feature then add the corresponding extended luminance description to H2 } add the edge bins from H1 to H2 for i=1 to N2 {w(i) =1} - N2 is the size of the new histogram H2

Output: the new histogram H2, weight vector w with no weights

4 Experimental Setup For image description we use histograms based features, which are the most common instruments to represent global visual image content for indexing and retrieval purposes. To characterize the color information in the images we are using a description compatible with MPEG-7 multimedia description standard [13] - small number of dominant colors in YCrCb color space and color distribution descriptors. Performing image retrieval by color indexing is computationally simple and fast but the retrieval with color descriptors only, can retrieve false positives, i.e. images with completely different color content which just happen to have a similar color composition as the query image. Thus, in practice to improve the retrieval performance it is better to combine color features and texture or shape features. We have decided to describe the composition of edge and luminance features as well as colors in images. We are using a block-based extraction scheme considering the presence of edges and the dominance of luminance or color in every block. After the generation of the histograms, the values associated to each feature are mapped into the interval [-1, 1] by a Gaussian feature normalization (4), which ensures an equal emphasis to all feature components (histogram bins). f

ij

=

f −m ij j 3s j

(4)

Semi-automatic Feature-Adaptive Relevance Feedback (SA-FR-RF)

157

In our first experiments, the set S is made up of 12 color bins, each one associated to color distribution descriptors, 11 luminance levels, and 3 edge bins. As initial feature set H1, we adopted a modification of the composite histogram proposed in [11], which consists of 14 bins (6 color bins, 5 levels of luminance, and 3 edge bins). It combines different important features of the image in a very low sized vector, thus limiting the needs in terms of memory requirements and computation. In accordance with above described FA-RF scheme if a specific color is important we describe it better, introducing a spatial information about its distribution - color mass center and color variance which are computed for every dominant color in the pre-processing phase. The images usually are with different dimension. To make the spatial information descriptors compatible we are normalizing the values in respect to x and y image dimensions.

5 Experimental Results 5.1 Performance and Retrieval Accuracy Evaluation For evaluation of the performance we are using the precision (the number of the retrieved relevant images divided by the total number of retrieved images) and the recall (the number of the retrieved relevant images divided by the total number of relevant images in the database). The only assumption that we make is that the user is consistent during the relevance feedback. The proposed SA-FA-RF strategy has been implemented and tested on mixed database with synthetic and natural images and on a database with 800 natural images in 19 categories. The query-by-similarity is performed by selecting a random image from every category for a query image and considering as relevant only the images from the same category. Fig.1 shows the precision and recall curves for the classical RF method, the proposed FA-RF method with feature space transformation and the SA-FA-RF method with feature space transformation and automatic internal iterations. As it can be seen, while RF converges after a few iterations to a local minimum, FA-RF continues to improve, thus reaching a much better convergence. Additional improvement is introduced by automatic internal iterations in the new feature space, presented by the SA-FA-RF curve.

recall / precision

80

precision SA-FA-RF

60

precision FA-RF precision RF

40

recall SA-FA-RF recall FA-RF

20

recall RF

0 0

1

2

3

4

Iteration Number

Fig. 1. Average Precision and Recall curves

158

A. Grigorova and F.G.B. De Natale

An example of the feedback process for a natural image query is illustrated in Fig. 2. It shows the retrieved images after the third RF iteration. In Fig. 2 (a) the RF method in [10] is applied and the results are obtained based on the similarity to the query computed from the initial image representation. Fig. 2 (b) shows the result obtained by SA-FA-RF strategy for the same query image and after the same number of iterations. Here the similarity is computed from the new features in the updated by the FA-RF strategy histogram H2. Since it is a better description of the user’s information need the retrieval result is improved.

(a)

(b)

Fig. 2. Retrieval results comparison between RF (a) and SA-FA-RF (b) 5.2 Feature Selection and Relative Importance Evaluation Experiments Since the re-weighting techniques are intuitive in learning we decided to compare the performance of FA-RF method with state-of-the-art feature selection algorithms for data classification, performing some experiments with WEKA data mining tool [12]. The image retrieval problem can be viewed as a problem of adaptively selecting the most important features to discriminate one image class in a database. The difficulty is that the classes cannot be defined a-priori, but depend on the user perception of similarity. In our experiments we consider the best-case, where the complete knowledge of the problem is available i.e. the feature selector knows a-priori which images are relevant and irrelevant and all the possible features are given in input to the selector. To this end we prepare new set of databases. For each image category in previously described image collection there is a corresponding database, where every image description contains all features from the extended set S (Sect. 3) and the images are divided in two groups (classes), the first one contains the images from selected category and the second one - all other images i.e. we are dividing the images in relevant and irrelevant to a category images. For comparison of the attributes selected by our algorithm and by Weka attribute evaluation schemes we execute for each image category first 10 retrievals by FA-RF algorithm with different queries from this category and then we perform 10 experiments with different Weka attribute evaluation schemes on the corresponding to this category of images new database. Analysing selected features we observe that FA-RF algorithm selects after feature adaptation 60%-70% of the features selected by Weka schemes for classification of the corresponding database. For each feature we

Semi-automatic Feature-Adaptive Relevance Feedback (SA-FR-RF)

159

calculate the percentage of the FA-RF executions, or WEKA schemes which have chosen this feature. Fig.3 shows graphically a comparison of the features selected by more than 60% of FA-RF algorithm retrievals and by more than 60% of Weka schemes applied on the corresponding new database. 0 0 0 0 0 0 0 0 0 0 0

F A - F R W

e k a

o

om

ri

p

z

t

g

e

C

H ge d E

Ed

y

er e dg

E

C

o

l6

V

v-

-x

-y ol C

C

ol

5

6

v

v

-x

y l5 C

o

l2 C

o

2 ol C

v

v-

-x

-y v l1 o C

o C

v

x v-

3

l1

m u

u

m

L

L

C

o

l1

2

2

l

1 0 9 8 7 6 5 4 3 2 1

Fig. 3. Features selected by FA-RF and Weka schemes We can see that there are features selected only by Weka schemes. It is important to remember that Weka schemes use databases with all semantic information - all available image descriptors and all available relevant images, whereas our FA-RF algorithm starts the retrieval only with the initial feature set and uses for the attribute selection only the information about the relevant images selected by the user after the first matching and the first RF iterations. Our goal is not to evaluate all available features (a work already done in many different ways), but in very simple way to describe better only the features important for the concrete user and concrete retrieval (specific query and relevance perception), maintaining a small sized feature vector. We can see that nevertheless of the worse conditions our FA-RF algorithm selects in 60%-70% the features selected by Weka schemes for the classification of the same image database. For comparison of the average evaluation of the attribute importance for the classification process we perform the next experiment. We assign weights to the attributes selected by different Weka schemes and different retrievals by FA-RF algorithm. The average values show that the FA-RF algorithm’s evaluation of the features importance is very similar to the Weka schemes attribute evaluation - for more than 70% of the features, the difference in both schemes importance evaluation is less than 30%. This experiment makes evident that not only the two schemes select similar feature vectors, but also the biggest part of selected features shows similar average attribute importance. Fig.4 shows more clear presentation of the two schemes results agreement. Many feature selection algorithms after the features evaluation consider as important the features with weights bigger than a fixed threshold – typically a percentage of the maximum weight. Therefore we fix a threshold of 30% of the maximum weight value and observe the decision of both schemes “if this feature is important or not”. The diagram (Fig.4.) shows that both schemes decision is the same for 70% of the features. The conclusion of this experiment is that both schemes would choose as important or not almost the same sets of features.

160

A. Grigorova and F.G.B. De Natale

Weka

Lu m 2 C ol or 5vy

Lu m 9 C ol or 1E dg vy e_ C om pl ex

Lu m 7

C ol or 12

Lu m 8

Lu m 6

Lu m 11

C ol or 5

C ol or 10

C ol or 6

C ol or 3

C ol or 7

FA-RF

C ol or 8

C ol or 2vx C ol or 3vx C ol or 4vx

100 90 80 70 60 50 40 30 20 10 0

Fig. 4. Average feature importance calculated by WEKA schemes and by FA-RF algorithm

5.3 Relative Accuracy Comparison Experiments We perform another experiment for evaluation of the performance and retrieval accuracy of our algorithm. We select a target group of images and built a database, nearly optimal for retrieval of images from the selected group, where feature vectors contain only the features selected by more than 60% of Weka evaluation schemes. On this database we executed retrieval experiments using a classical RF method [10] with query and weights refinement, but without feature space adaptation. For each query, 3 feedback iterations are run and the results are reported in Table 1, in comparison with the same RF method with the initial image description, and our SA-FA-RF method. Table 1. Comparison of precision and recall results

RF SA-FA-RF RF-Weka DB

It 0

precision It 1 It 2

It 3

It 0

It 1

recall It 2

It 3

46,3 46,3 70,6

55 53,8 66,9

65,6 71,3 92,5

14,8 14,8 22,6

17,6 17,2 21,4

20,2 21,6 28,2

21 22,8 29,6

63,1 67,5 88,1

100

recall / precision

90 80 70

precision RF WEKA DB precision SA-FA-RF

60

precision RF

50

recall RF WEKA DB

40

recall SA-FA-RF

30 20

recall RF

10 0 0

1

2

3

4

Iteration Number

Fig. 5. Average Precision and Recall curves

Fig. 5 shows the precision (a) and recall (b) curves for the three methods, averaged over a substantial number of trials. It can be seen that executed on the initial composite histogram, RF method converges after a few iterations to a local minimum. SA-FA-RF method curve continues to improve over the RF curve, approximating the

Semi-automatic Feature-Adaptive Relevance Feedback (SA-FR-RF)

161

values achieved by RF method, executed on the nearly optimal image description, which contains the parameters selected by Weka schemes for retrieval of this group of images and allow the achievement of higher values of both parameters.

6 Conclusions and Further Work A new image retrieval approach based on RF was presented. During the retrieval process we dynamically adapt not only the query parameters and feature weights but also the set of image descriptors (number and type) in order to better satisfy user’s information needs. The adaptation process is based on the relevant and irrelevant image descriptions as well as on their number and proportions during consecutive RF iterations. The experimental results concerning the performance and the retrieval accuracy evaluation, reported in Sect.5.1. show an improvement of the retrieval accuracy in comparison with other RF-based approaches. Feature selection and relative importance evaluation experimental results demonstrate that during the retrieval process FA-RF algorithm is effective in selecting the features important for the classification of the images in the database, thus improving the retrieval accuracy (Sect. 5.2). Further work is being devoted at improving image description and replacement rules related to features correlation and compatibility, and experiments with a synthetic query applied to a natural-image database. Other further experiments are planed with different feature descriptors. Our algorithm is not fixed to the concrete feature set and it could perform with different hierarchical feature sets after the definition of the initial and extended feature sets (with rough and more detailed level features respectively) and the association rules corresponding to the feature meaning.

References 1. A.W.M. Smeulders, M. Worring, S. Santini, A.Gupta, R.Jain, "Content-based image retrieval at the end of the early years", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol 22, no.12, pp 1349-1380, 2000 2. X. Zhou and T. S. Huang. Relevance Feedback in Image Retrieval: A Comprehensive Review. Multimedia Systems Journal, 8(6), pp. 536--544, 2003 3. Y. Rui, T.S. Huang, “Relevance feedback: A power tool for interactive content-based image retrieval”. IEEE Transactions on Circuits and Video Technology, 8(5):644--655, Sept. 1998 4. D.R. Hesterkamp, J. Peng, H.K. Dai, "Feature relevance learning with query shifting for content-based image retrieval", Proceedings of the 15th IEEE International Conference on Pattern Recognition, ICPR 5. Ng, Raymond and Sedighian, Andishe. “Evaluating multi-dimensional indexing structures for images transformed by principal component analysis”, Proceedings of SPIE/IS&T Conference on Storage and Retrieval for Image and Video Databases IV, Volume 2670, pages 50-61, 1996 6. Philippe Leray and Patrick Gallinari. Feature selection with neural networks. Technical report LIP6 1998/012, LIP6, 1998

162

A. Grigorova and F.G.B. De Natale

7. D.W.Aha, R.L.Bankert, “A comparative evaluation of sequential feature selection algorithms”, In D. Fisher and J.H.Lenx (Eds.), Artificial Intelligence and statistics V. New York: Springer-Verlag, 1999 8. Y. Ishikawa, R. Subramanya, and C. Faloutsos, “Mindreader: Query databases through multiple examples” in Proceedings of the 24th International Conference on Very Large Databases VLDB, pages 218-227, 1998 9. Ricardo Baeza-Yates, Berthier Ribeiro-Neto, “Modern Information Retrieval”, Addison Wesley Publishing Company, Essex, England,1999 10. Y.Wu, A.Zhang, "A feature re-weighting approach for relevance feedback in image retrieval", IEEE Int. Conf. on Image Processing ICIP2002, Vol II, pp 581-584, 2002 11. D.K.Park, Y.S.Jeon, C.S.Won, S.J.Park, S.J.Yoo, "A Composite Histogram for Image Retrieval", 2000 IEEE International Conference on Multimedia and Expo, ICME2000,Vol I, pp 355-358, 2000 12. I.H.Witten, E.Frank, L.Trigg, M.Hall, G.Holmes, M.Apperley "Weka: Practical Machine Learning Tools and Techniques with Java Implementations" Proc ICONIP/ANZIIS/ANNES'99 Int. Workshop: Emerging Knowledge Engineering and Connectionist-Based Information Systems, New Zealand, pp192-196, 1999 13. MPEG-7 (2002) Overview of the MPEG-7 standard (version 8.0), ISO/ IECJTC1/ SC29/WG11 N4980

A Visual Query Language for Uncertain Spatial and Temporal Data Karin Silvervarg and Erland Jungert FOI, (Swedish Defence Research Agency), Box 1165, S-581 11 Linköping, Sweden {karin, jungert}@foi.se

Abstract. Query languages for sensor data will have similarities with

traditional query languages but will also have diverging properties that cause a higher complexity than the traditional ones. Both types require data independence. However, as different sensors create data of heterogeneous type the commonly used methods for data selection cannot be used. Furthermore, sensor data will always be associated with uncertainties and since also sensor data fusion must be possible to carry out this cause further problem in development of the query languages. Here a visual query language for sensor data input is discussed from these perspectives to allow a complete set of spatial temporal queries by means of its visual user interface.

1 Introduction Query languages intended for multiple sensor data sources and other types of external data sources such as text messages differ in many ways from conventional query languages. A characteristic of this type of systems, which include multiple sensor data sources, is that they generally are more complex than traditional query systems. In particular, they will in most applications, be concerned with queries of spatial/temporal type. It will come from several different sources, but one important type of source is sensor data. Such data sources must be able to handle information that in various ways is uncertain. There are several sources of uncertainties in sensor data; one is due to limitations in the sensors and the navigation systems of the platforms carrying the sensors; another depends on the resolution of the sensor data; and a third reason is that some sensor types are sensitive to weather conditions such as snow or rain. All types of uncertainties will consequently have effects on the way the queries are executed. Another consequence when multiple data sources are involved is that in those cases fusion [16] of data must be possible to handle. Sensor data fusion becomes specifically complicated since the sensor data mostly are of heterogeneous type, i.e. sensor data are of different types. Other problems that appear concern the selection of the data sources. Sensor technologies are constantly developing and for this reason it becomes almost impossible for a user to have sufficient knowledge about the capabilities of all occurring sensors. Thus the selection of data sources should be carried out by the query system independently of the users, for instance by using ontology [21]. This S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 163 – 176, 2005. © Springer-Verlag Berlin Heidelberg 2005

A Visual Query Language for Uncertain Spatial and Temporal Data

164

will result in a system that from a user’s perspective is sensor and sensor data independent [22], [8]. Another motivation for this approach is to delimit the workload of the users and let them concentrate on their primary activities. In this way they do not need to have any particular technical knowledge on sensors and sensor data. It should be enough for a user to have a general understanding of the problems associated with the sensor data e.g. due to the uncertainties the sensors cannot correctly measure all possible attributes of the sensor data. The work discussed here, which is an extension of [26], is focusing on a visual user interface of a query language for multiple sensor data sources. Clearly, many of the characteristics mentioned above will have an impact on such a user interface; among them sensor data independence. To achieve sensor data independence the system cannot and should not communicate concepts related to the sensors to the users. Instead, the system should work with concepts relevant and familiar to the users. This is along with tradition of traditional query systems where data independence plays an important role. A large number of applications for query languages for sensor data fusion can be foreseen. Among these are applications where the query language is integrated to a command and control system for military applications and for crisis/emergency management systems but other less complex applications exist as well. In this paper we only look at the problems concerned with specifying the query. How to present the result of the query is also an important problem, but that is a different problem, and we will not discuss that further in this paper. Among the related works that should be pointed out are the work by Abdelmoty and El-Geresy [1], who have designed a system for graphical queries that is a filter based approach. It is primarily designed for spatial queries with similarities to ΣQL. Malan et al [24] works with data that in uncertain in the temporal aspect, in their case searching old African art where the dating is imprecise. Other approaches that focus on temporal queries of varying complexity are [17], [12], [11], [15] and [14]. Chang [6] has made an approach to use c-gestures for making spatial/temporal queries in what he calls a sentient map. Bonhomme [5] have proposed a visual representation for spatial/temporal queries based on a query-by example approach and related variation of this, called query-by-sketch, is presented in [4]. Hirzalla and Karmouch [19] have a more direct query approach to spatial/temporal queries, but they treat the spatial and the temporal parts separately and not together. This paper is structured as follows. In section 2 the problem definition is presented and discussed. After this follows a presentation of the ΣQL query language in section 3, which forms the bases of the work presented here. In section 4 some of the basic elements that are present in the query language are presented. In sections 5 and 6 are the aspects of spatial and temporal queries discussed. After this follows in section 7 a discussion of the aspects of uncertainty. The section on uncertainty is followed by a section including a set of examples illustrating Visual ΣQL. Furthermore, there is also a discussion of the problem of completeness of the queries in section 9. Finally, conclusions from the work are given in section 10.

165

K. Silvervarg and E. Jungert

2 Problem Definition The main focus of the query language described in this work is concerned with moving ground based objects that correspond to various types of vehicles. The sensor may either be airborne, e.g. UAVs, aircrafts, or satellites. As these objects may be moving from time to time the sensor system must be able to detect and classify these objects. The consequence of this is that the query system primarily must be designed to respond to spatial/temporal queries where uncertainties in the data must be taken into consideration. Given the above background the main problem in this work has been to develop a visual user interface for a query language where spatial/temporal queries are in focus. Generally, a set of elements can be associated with the queries. These elements relates to where?, when? and what?, that is where? corresponds the area of interest (AOI), when? to the time-interval of interest (IOI) and finally what? to the object type(s) that is asked for. These three elements are basically part of most, trivial as well as complex, queries since it is obvious that if we ask for a particular object (vehicle) then we must also consider a particular area where the object can be found but also a certain time interval during which it can be found in that area. A further problem is associated with the visualization of the query results. To be considered in connection to this are again the aspects of uncertainty and the context in which the query, as well as the result, are directed. The first one of these aspects cannot be avoided because of the limitations of the sensors and the sensor platforms. The context is concerned with the geographical background of the requested objects. Thus the context can be demonstrated by means of geographical map information, corresponding to traditional map objects, such as roads, forests, lakes etc. Clearly, the map information is not only required for visualization of the query result but also for representation of the area of interest in the queries as will be demonstrated subsequently.

3 The Query Language, ΣQL The query language discussed in this work is called ΣQL [1], [10], [7]. Originally, ΣQL was developed as a query language for querying of a single sensor image at a time. However, later it has evolved into a query language for multiple sensor data sources with capabilities for sensor data fusion and sensor data independence. The various sensors may be of different types and generates heterogeneous sensor data images. For this reason a large number of algorithms [20] for sensor data analysis must be available and administered by the system. It is required that selection of sensors and algorithms must be carried out autonomously and for this reason means for such selection must be available. In ΣQL this is controlled by the ontological knowledge based system [21], [22]. The reason for the autonomous selection is to allow sensor data independence. Sensor data independence is motivated for a large number of reasons. One reason, which already has been mentioned, is to allow the user to concentrate on the work at hand without any knowledge of the sensors and their data types. Another motivation is to make repetitive queries possible without interference from the users. In this way light and weather conditions may, for instance, change during the period when the query is repeated across a specified area of interest resulting in different selections of sensors.

A Visual Query Language for Uncertain Spatial and Temporal Data

166

This query language allows classification of objects not only from the sensor data sources; it also allows cuing (detection) of possible candidates as a first step towards classification. Sensors used for cuing can, for example, be a ground senor networks or a synthetic aperture radar (SAR), while sensors for classification can be IR/camera, laser/radar and CCD/camera. The distinction between these two classes of sensors is that the former covers a much larger area than the latter. Thus the former can be used to quickly search through the AOI for object candidates much faster than the sensors used for classification. They can also determine a much smaller search area for the classification sensors. Consequently, this depends on differences in coverage and resolution, i.e. the classification sensors have a low coverage and a high resolution opposite to the cuing sensors. The basic functionality of the query language can be described as follows. A query is inserted by the user and then the input is fed in to the dependency tree generator which in a dialogue with the ontological knowledge structure generates a set of nodes in the dependency tree. As long as there are nodes in this tree, new sub-queries can be built (one for each of the selected sensors), refined and executed by the query processor. Once the sub-queries have been executed instances are created in the multilevel view database to support query refinement in a subsequent step. As new sets of dependency tree nodes are generated new sub queries can be executed and their result refined. This goes on until the dependency tree becomes empty, i.e. when there is no more sensor data available to process. In the final step data fusion is, if applicable, carried out using the result of the sub queries as input and then the process terminates. This process is further discussed in [8]. Sensor data fusion [22] is another property of ΣQL. It is quite unique and does not occur in traditional query systems. The motivation for sensor data fusion is to allow information from different sensors to support identification, but also to complement the information since different sensors can register different things; for instance a CCD camera might see that a car is blue, while an IR camera might see that the engine is running. A serious question is how to interpret the fused result of a query. The approach taken here has been to associate a belief value to each result from the various sensors that are used in a query. Belief values are determined by the involved sensor data analysis algorithms and a common value is determined by the fusion process, which also is forwarded to the user as a part of the query result. All belief values are normalized and a high value means that a user may have a higher confidence in the result. It is not only to facilitate the fusion, but also to give the user a sense of how strong belief or confidence he can have in the result. This is an important aspect of the system that has been introduced to give the users a higher degree of trust in the query result as well as in the query system.

4 Basic Query Elements From a logical representation of ΣQL we have developed a visual language, called Visual ΣQL [25][26]. The basic questions to be considered are where?, when? and what?. We have chosen to let the user mark the AOI in a map and IOI, in its simplest form, is set by the start and end points in time, see figure 1 in the upper right corner. Queries can be repeated over time, i.e. the same query can be applied several times

167

K. Silvervarg and E. Jungert

but with different IOI. Object types can in their simplest form be chosen from a list, but the user often has other requirements on the result. He may not be looking for all vehicles, but only for all moving vehicles, or all vehicles moving along a certain road. Consequently, a way to select the object types and put them in relation to each other is required. The solution is to use a structure with both object types and relations. There are many different approaches to this, one is query-by-example [5], and another quite similar is query-by-sketch [4]. We have chosen the approach of dataflow [18]. The object types are simply selected from a list, and eventually the user can apply relations to these objects, thus putting restrictions on the query result. The user interface is built up around a work area and palettes. The work area is the space where the object types and the relations are placed and set in relation to each other by using a visual approach. The palettes contain the object types and the relations, organized according to attribute types, to be easily navigated.

Fig. 1. Simple selection of AOI, IOI and object type

Objects correspond to all kinds of object types that can be found by the sensors although in our application ground vehicles are of primary interest. The objects have both properties that are static, e.g. type of vehicle and color, but they also have properties that may vary over time e.g. velocity and orientation. Objects can also be of geographical type e.g. roads, towns. An ontology containing a hierarchy of the objects is available as well making it possible to say that a car is a vehicle etc. [21]. The ontology describes the properties of the different objects. A car may, for instance, have attributes of type orientation, color, size, position, velocity, while a road may have speed restrictions, pavement and position. When the user selects an object type and places it in the workspace it is visualized as a rectangular box with the object type written inside, see e.g. figure 2.

A Visual Query Language for Uncertain Spatial and Temporal Data

168

As all objects are assumed to be linked to various relations there has to be ways to specify the details of the query. Once the type of objects is set together with the relations, the relations delimit the answer to include just those objects for which the relations are true. The relations can be either unary or binary. Unary relations are for example “color equals blue” or “velocity is greater than 50 km/h”. The binary relations can be undirected or directed. Directed means that the order of the involved objects matters, for instance the relation “before” gives a different result depending on the order of the involved objects, whereas the result of “equals” does not. The user can select the relations from a palette. The relations are also visualized as boxes, where the possible relation is illustrated with an icon rather than by text, since it is often simpler for a human to grasp an icon [5]. To distinguish the relations further from the objects they also have a different color at the edge of the box, see figure 2.

Vehicle

Road

Fig. 2. The user has selected the object-types vehicle and road, and the relation inside

Objects and relations are connected to each other with arrows. Everything passed between a pair of such “boxes” is represented as a set of tuples. Output from an object box corresponds to a tuple of size one that simply contains an item from the set described in the object; e.g. a vehicle. In a binary relation two different tuples are related to each other and the resulting output tuples may contain more than one element. If an input tuple contains more than one element the user has to define which of the elements in the tuple that is part of the relation, see figure 3b. In the resulting tuple the user can choose to include any parts from the input tuples. For example, in figure 3a we have the relation inside. One of the participating sets contains cities and the other could be a result of a relation relating the cars to the nearby rivers. In this case we could relate the cars to the cities to find out which of the cars that are in the city. In this case probably only car1. The resulting tuple from the relation can contain car, river, and city, or any subset thereof i.e. car and river; car and city; river and city; car; river; city. In figure 3a the resulting set is shown where the tuple consists of river and city. The query language also includes some set operations, i.e. union, intersection and set difference. They are treated similarly to relations, but function a bit differently. The union operator simply produces a set containing all tuples from both incoming sets. The only restriction is that all the resulting tuples must have the same number of elements. To make it meaningful the user pairs elements from the incoming tuples. If some elements that are paired do not contain the same type of objects, the ontology

169

K. Silvervarg and E. Jungert

{, }

City {}

{}

a.

b.

Fig. 3. a. A relation where one of the tuples has more than one element and where the result contains only a part of all possible elements. b. Settings of that relation.

is used to find the object that is closest above both objects in the object hierarchy to get the resulting type. This situation may occur in queries where objects of similar types are asked for. For instance, the incoming tuples may hold elements of the tuples and ; then the result may contain , which is more meaningful than other possible combinations. Intersection is a bit different from union, because in intersection only a single element in each of the input tuples is chosen, just like in our normal relations. These objects are used to compute the intersection just like in normal set operations. The resulting tuple can include elements from all incoming tuples. If, for instance, is intersected with and car in both tuples is chosen for the intersection then the result could be . Set difference is similar to intersection in the aspect that only one element in the participating tuples are selected. Contrary to intersection set difference is a directed relation so all elements that are in the first, but not in the second tuple is kept as the result. Similarly to intersection the resulting tuple may contain elements from both participating tuples. On all relations the not operator can be applied. Usually this means that all results for which the relation would be true is now false and vice versa, but it will work a bit

A Visual Query Language for Uncertain Spatial and Temporal Data

170

different when dealing with uncertainties, see chapter 7 for further explanations. Not is visually denoted by drawing a line diagonally across the icon, see figure 4. All types of relations can not be applied to all types of objects, for instance, it is normally meaningless to apply a color comparison to a city. To satisfy the need to know, which relations that can be applied to which type of object the ontology has been expanded. It also contains information about, the properties of each object type. This can be matched with the requirements the relations have on the attribute values the objects being related must have. No relation is in any way selected to be the result of the query. Instead all “boxes” can be seen as the results or partial results. This will give the user a better opportunity to analyze the consequences of different relations. Another effect is that the workspace may contain several unrelated queries on the same objects, AOI and IOI.

Fig. 4. Not applied to the inside relation

5 Spatial Queries In [13], Egenhofer identifies eight atomic topological relations disjunct, meet, equal, inside, coveredBy, contains, covers, and overlap, which have different spatial relational properties. All these have been included in the query language as a basis for spatial queries. We have also found needs for directional spatial relations i.e. northOf, southOf, north-eastOf etc. Similar to this we have also directional relations that are determined with respect to a local object or position, i.e. inFrontOf, behind, leftOf etc. All relations require different properties with respect to the objects to which they relate. For instance, inFrontOf requires that the position and the orientation of the object is known, while inside requires an extension in space. To make sure that the relations only operate on objects with the required relational properties the ontology also includes information about the type of properties that can be expected from each type of object. Then each relation has a set of requirement on the object properties. The spatial properties defined here are hasExtension, hasPosition and hasOrientation. All of the mentioned spatial relations are binary. Sometimes there is a need to relate an object to a fixed point or area and for this purpose we have included the spatial attribute entity (SAE) that only has a position or area and no other properties. A SAE can be used together with all of the relations mentioned above to determine objects that relate to fixed points or areas. The only difference when relating an object to a SAE compared to when relating a pair of objects to each other is to determine the output tuple. Since a SAE is not an object it is not allowed to be included in the result.

171

K. Silvervarg and E. Jungert

6 Temporal Queries The classical work on temporal relations has been done by Allen [3]. He has defined 13 binary relations that concerns relating two time interval to each other. These relations are: before, after, meets, metBy, overlaps, overlappedBy, finishes, finishedBy, starts, startedBy, contains, during and equals. We have chosen to use the same icons for these relations as those used by Hibino et al [17]. The only difference is that we have decided to use squares instead of circles, see figure 5; the reason for this is described in the section on uncertainties.

Fig. 5. Visualization of Allen’s 13 temporal relations

In analogy to spatial queries there is also a need for relating objects to fixed points in time or time intervals. Thus the functionality for creation of a temporal attribute entity (TAE) has been included. If only a point in time is needed the start and end of the time interval become equal. Consequently, Allen’s relations [Allen, J. F., ] can be used to relate objects to fixed times as well. Determination of tracks is a quite common task in most sensor data systems. To produce tracks by means of a query language from sensor data requires special attention. Thus, when general attribute relations are combined with either spatial or temporal relations then it is quite simple to determine tracks. However, when combining general attribute relations with both spatial and temporal relations then the system must keep track of which observations that fulfill both the time and spatial relations since otherwise simultaneousness may be lost. The objects that are related by the temporal relations, just like the spatial relations, require certain properties. While the spatial relations have three different properties, the temporal relations have just a single one, i.e. hasTime. The reason for this is that time is one-dimensional while space, so far, in our system is 2D.

7 Uncertainties As have been pointed, out sensor data are always associated with uncertainties that depends not only on the limitations in the sensors but also to limitations in the

A Visual Query Language for Uncertain Spatial and Temporal Data

172

navigation systems, unless the sensors are at fixed positions. Obviously, these uncertainties will influence the result of the queries in more than one way; especially when dealing with extremely large data volumes and when the data from multiple data sources are part of the sensor data fusion process. It also turns out that from a user’s perspective there are two aspects that need to be handled. That is, uncertainties in space and in time. All spatial relations should be possible to apply in a mode where uncertainties will be considered. This is visually distinguished by representing the areas inside the icons by more fuzzy looking symbols. The effect of this is that a relation may be true with respect to the uncertainties but in reality with completely accurate data the relation may not be true. The advantage of this is that no crisp results will be missed. Take for instance the inside relation; here all areas that could be inside will be returned even though they also might be slightly outside, but with respect to uncertainties we can not tell for sure, which is the case. Uncertainties in time are treated similarly. All relations have an uncertain mode, which is visualized by replacing the squares with circles, see figure 6. There are two reasons for using squares in the accurate case and circles in the uncertain. One reason is because it is easier to associate the sharp corners of the squares with the actual data and the round shape of the circle with the more uncertain data. Another reason is that we believe that the users most often will use the uncertain mode of the relations and thus we will be using the same symbols as Hibino et al [17].

Fig. 6. The relations before and start, where consideration will be taken to the uncertainties in the data when evaluating the relation

The result of applying not to a relation that account for uncertainties is a bit different from when it does not. Usually, the complete inverse of the result would be kept, but when the uncertainties are considered the result will be a bit different. For example the inside relation for uncertainties will include all results that might just be inside. The not inside relation will include all results that might be outside, including some results that actually are inside, although it cannot be proven. Thus some results are part of both in the inside and in the not inside relation. So far, no complete implementation of this part of the system exists. However, the actual implementation of the evaluation of relations concerned with uncertainties can be made in several ways. Several theories exist, one is rough sets [2] another is the egg yolk theory [23]. There is also some research being done on how to evaluate information with different levels of uncertainty [24]. The aspects of uncertainty of data must be studied further to be able to logically respond to the queries in a correct way and to give the users a better support. The starting point for such a study must be to regard uncertainty manipulation in the queries as the default case.

173

K. Silvervarg and E. Jungert

8 Completeness The visual query language should be at least as powerful as the relational algebra. The operators of the relational algebra are union, set difference, projection, selection and cartesian product [27]. Union, is the set of tuples that are in R or S or both. Union in the relation algebra can only be applied to relations of the same arity. Union is directly implemented in our language, with the additional convenience to allow selection of subparts of the tuples to make them equal in size. The set difference of R and S contains those tuples in R which are not also in S. Set difference just like union is implemented directly in Visual ΣQL. The subset of R that contains all tuples that satisfies a given logical formula is a selection of R; this is the equivalent of selecting some of the rows in a table. In this visual query language this is carried out by the relations. The idea behind projection is to remove some elements/components from a relation. If a relation is seen as a table then projection is the act of selecting a set of columns in that table. In this work this corresponds to the selection of which elements that should be included in the resulting tuple. The cartesian product of R and S is the set of all tuples whose components forms a tuple in R and a tuple in S. Our binary relations are cartesian products with a selection and a projection. If the relation is set up to always be true then there is no selection, and if all elements are selected for the resulting tuple then there is no projection. The result is the pure cartesian product.

9 An Example To illustrate the use of Visual ΣQL some examples will be given. The examples can be seen as part of a simple scenario where the fundamental problem is to find different vehicles along or on a riverside express way passing an electrical power plant. There is also a second express way in the vicinity of the area of interest that is not interconnected to the other one. In this scenario there has been some incident directed towards the power plant. The user of the query language is trying to find, identify and track vehicles on the riverside express way where the vehicles may be engaged in further hostile activities directed towards the power plant. To monitor the area a set of ground based sensors have been deployed along the express way, that e.g. can read the license plates of the vehicles or from a top view can determine a set of other attributes such as speed and direction. Query 1: Find all vehicles close to the power plant near in time of the incident. The AOI corresponds to a relatively small area around the power plant and the IOI is the day of the incident. The visual representation of this query can be found in figure 7. The output of this query is saved by the user for further usage. The user also labels this information suspect vehicles. The rational of this query is to try to identify all vehicles that have been in the area at the time of the incident and, which for this reason may be considered suspicious

A Visual Query Language for Uncertain Spatial and Temporal Data

Vehicle

174

power-plant

Distance < 2 km

12-18

Fig. 7. Finding suspect vehicles around the power plant (query #1)

Road

Vehicle

River

Road.type = ‘freeway’

Suspect vehicles

Similar Fig. 8. Find similar vehicles on the riverside freeway (query #2)

Query 2: Find similar vehicles on the riverside freeway. (figure 8) Here the AOI is covering a much larger area and the IOI could be the same as before. This query is visualized in figure 8. Here the term similar is determined by means of the object ontology. This and other analogous concepts are also discussed further in [9]. This query is motivated by the needs to find also other vehicles that may be connected to the vehicles that originally were found suspicious.

10 Conclusions In this work the visual user interface of the query language ΣQL has been discussed together with some of its basic characteristics. The query language allows queries of spatial/temporal type where the input data are generated by sensors of various types. The applications here will be focusing on ground based targets (objects) that basically

175

K. Silvervarg and E. Jungert

may be vehicles in a geographical context. The sensors may be both airborne and ground based. Data from these sensors are generally of heterogeneous type. The query system can handle uncertainties due to the sensor system and since multiple sensors may cover the area of interest a method for sensor data fusion has been developed and integrated. The set of possible spatial/temporal queries is complete from a traditional theoretical viewpoint. Particularly, the queries may be concerned with information that in time and space may be both absolute and relative. Relations from such queries are normally determined by means of a set of predefined operators. A simple demonstrator of Visual ΣQL has been implemented and is gradually extended. Currently, five different sensor types have been integrated to the system among these are laser-radar, IR-camera and a CCD-camera but also an unattended ground based sensor network. In order to be able to handle more complicated and dynamic situations where quite large amounts of data must be handled the query processor will be hooked up to a simulation system. Future research will focus on user tests of the visual user interface and on the adaptation of the query system to Internet applications. The approach taken for the selection of the sensor data in cases where the sensors are distributed on different platforms attached to the Internet will be based on sets of intelligent agents. The focus of our current work is on how to specify queries. Future research also needs to find a satisfying solution on how to present the result of those queries. That solution has to solve the problem of displaying results that differ in time and location, and where uncertainties are present in all aspects of the result.

References 1. Abdelmoty, A. and El-Geresy, B., Qualitative Tools to Support Visual Querying in Large Spatial Databases, Proceedings of the workshop of Visual Language and Computing, Miami, Florida, September 24-26, 2003, pp 300-305. 2. Ahlqvist, O., Keukelaar, J. and Oukbir, K., Rough and fuzzy geographical data integration. International Journal of Geographical Information Science, 14(5):475-496, 2000. 3. Allen, J. F., Maintaining knowledge about temporal intervals, Communications of the ACM, vol. 26, no. 11, pp 832-843. 4. Blaser, A. D. and Egenhofer, M. J., Visual tool for querying geographic databases, Proceedings of the Workshop on Advanced Visual Interfaces, 2000, p 211-216 5. Bonhomme, C., Aufaure, M.-A. and Trépied, C., Metaphors for Visual Querying of Spatio-Temporal Databases, Advances in Visual Information Systems. 4th International Conference, VISUAL 2000. Proceedings (Lecture Notes in Computer Science Vol.1929), 2000, p 140-53 6. Chang, S.-K., The sentient map, Journal of Visual Languages and Computing, Vol 11, No. 4, August 2000, pp 455-474. 7. Chang, S.-K. and Jungert, E., Query Languages for Multimedia Search, In Principals of Visual Information Retrieval, M.S. Lew (Ed.), Springer Verlag, Berlin, 2001, pp 199-217. 8. Chang, S.-K., Costagliola, G., Jungert, E., Multi-sensor Information Fusion by query Refinement, Recent Advances in Visual information Systems, Lecture Notes in Computer Science, 2314, Springer Verlag, 2002, pp 1-11. 9. Chang, S.K., Jungert, E. Iterative Information Fusion using a Reasoner for Objects with Uninformative Belief Values, Proceedings of the seventh International Conference on Information Fusion, Stockholm, Sweden, June 30 - July 3, 2004.

A Visual Query Language for Uncertain Spatial and Temporal Data

176

10. Chang, S.-K., Costagliola, G., Jungert, E. and Orciuoli, F., Querying Distributed Multimedia Databases and Data Sources for Sensor Data Fusion, accepted for publication in the journal of IEEE transaction on Multimedia, 2004. 11. Chittaro, L. and Combi, C., Visualizing queries on databases of temporal histories: new metaphors and their evaluation, Data & Knowledge Engineering, v 44, n 2, Feb. 2003, p 239-64 12. Dionisio, J.D.N. and Cardenas, A.F., MQuery: a visual query language for multimedia, timeline and simulation data, Journal of Visual Languages and Computing, v 7, n 4, Dec. 1996, p 377-401 13. Egenhofer, M., Deriving the combination of binary topological relations, Journal of Visual languages and Computing, Vol 5, pp 133-49. 14. Erwig, M. and Schneider, M., Spatio-temporal predicates, IEEE Transactions on Knowledge and Data Engineering, v 14, n 4, July/August, 2002, p 881-901 15. Fernandes, S., Schiel, U. and Catarci, T., Visual query operators for temporal databases, Proceedings of the International Workshop on Temporal Representation and Reasoning, 1997, p 46-53 16. Handbook of Multisensor Data Fusion, D. L. Hall & J. Llinas (Eds.), CRC Press, New York, 2001. 17. Hibino, S. and Rundsteiner, E. A., User Interface Evaluation of a Direct Manipulation Temporal Visual Query Language, Proceedings ACM Multimedia 97, Seattle, WA, USA, 9-13 Nov. 1997, p 99-107. 18. Hils, D., "Visual Languages and Computing Survey: Data Flow Visual Programming Languages", Journal of Visual Languages and Computing, vol.3, 1992, pp.69-101. 19. Hirzalla, N. and Karmouch, A., Multimedia query user interface, Canadian Conference on Electrical and Computer Engineering, v 1, 1995, p 590-593. 20. Horney, T., Ahlberg, J., Jungert, E., Folkesson, M., Silvervarg, K., Lantz, F., Franssson, J., Grönwall, C., Klasén, L., Ulvklo, M., An Information Sustem for target recognition, Proceedings of the SPIE conference on defense and security, Orlando, Florida, April 12-16, 2004. 21. Horney, T., Design of an ontological knowledge structure for a query language for multiple data sources, FOI, scientific report, May 2002, FOI-R--0498--SE. 22. Horney, T., Jungert, E., Folkesson, M., An Ontology Controlled Data Fusion Process for Query Language, Proceedings of the International Conference on Information Fusion 2003 (Fusion’03), Cairns, Australia, July 8-11. 23. Lehmann, F. and Cohn, A. G., The eggyolk reliability hierarchy: Semantic data integration using sorts with prototypes. Proceedings of the third international conference on Information and knowledge management, ACM Press, 1995, pp 272-279. 24. Malan, K., Marsden, G. and Blake, E., Visual query tools for uncertain spatio-temporal data, Proceedings of the ACM International Multimedia Conference and Exhibition, n IV, 2001, p 522-524. 25. Silvervarg, K. and Jungert, E. Aspects of a visual user interface for spatial/temporal queries, Proceedings of the nineth International Conference on Distributed Multimedia Systems, Miami, Florida, September 24--26, 2003, pp 287-293. 26. Silvervarg, K. and Jungert, E., Visual specification of spatial/temporal queries in a sensor data independent information system, Proceedings of the tenth International Conference on Distributed Multimedia Systems, San Francisco, California, September 8-10, 2004, pp 263-268. 27. Ullman, J., Principles of Database and Knowledge - Base Systems, Volume 1. Computer science press, Rockville, 1988.

Surveying the Reality of Semantic Image Retrieval Peter G.B. Enser1, Christine J. Sandom1, and Paul H. Lewis2 1

School of Computing, Mathematical and Information Sciences, University of Brighton {p.g.b.enser, c.sandom}@bton.ac.uk 2 Department of Electronics and Computer Science, University of Southampton [email protected]

Abstract. An ongoing project is described which seeks to add to our understanding about the real challenge of semantic image retrieval. Consideration is given to the plurality of types of still image, a taxonomy for which is presented as a framework within which to show examples of real ‘semantic’ requests and the textual metadata by which such requests might be addressed. The specificity of subject indexing and underpinning domain knowledge which is necessary in order to assist in the realization of semantic content is noted. The potential for that semantic content to be represented and recovered using CBIR techniques is discussed.

1 Introduction Within the broad church of visual image users and practitioners there has been only a minimal engagement with the endeavours of the research community in visual image retrieval. Correspondingly, those endeavours have been little informed by the needs of real users or the logistics of the management of large scale image collections. With the developing maturity of these research endeavours has come a realisation of the limitations of CBIR processes in practice, however. These limitations reflect the fact that the retrieval utility of visual images is generally realised in terms of their inferred semantic content. The context for this inferential reasoning process is to be found in the distinction drawn in semiotics between the denotation, or presented form, of the image and the connotation(s) to which it gives rise. It is clear that personal knowledge and experience, cultural conditioning and collective memory contribute towards that reasoning process. The CBIR community has attached the label ‘semantic image retrieval’ to the formulation and resolution of information needs which engage that intellectual process. Within the three-level hierarchy of perception postulated by Eakins & Graham [1], semantic image retrieval subsumes retrieval by ‘derived features’ and ‘abstract attributes’. The sharply drawn distinction between those retrieval processes and the automatic extraction of low level features from denotative pixel structures is characterised as the ‘semantic gap’ [2]. A useful listing of research endeavours which have addressed the three levels of perception may be found in [3]. We are engaged upon a project, the aims of which are to provide both research and practitioner communities with a better-informed view of the incidence and S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 177 – 188, 2005. © Springer-Verlag Berlin Heidelberg 2005

178

P.G.B. Enser, C.J. Sandom, and P.H. Lewis

significance of the semantic gap in the context of still images; to seek enhanced functionality in image retrieval by bridging that gap; and to design and evaluate an experimental visual image retrieval system which incorporates the construction of such a bridge.

2 A Taxonomy of Images This paper is concerned with the first of the aims described above. To this end a broad spectrum of operational image retrieval activity has been surveyed in order to take account of the full plurality of types of image and user. This survey has been informed by, and has in turn informed, an image taxonomy, which is shown in diagrammatic form in Figure 1.

Image

Simple

Picture

Direct

Complex

Hybrid Picture

Indirect

Drawing

Visual surrogate

Diagram

Map/ chart/plan

Device

Fig. 1. A taxonomy of still images

The following definitions have been used in the construction of this taxonomy: Image Simple Image Complex Image Picture Hybrid Picture Visual surrogate Direct Picture Indirect Picture

Drawing

a two-dimensional visual artefact. an undifferentiated image. an image which comprises a set of simple images. a scenic or otherwise integrated assembly of visual features. a picture with integral text. non-scenic, definitional visual artefact. a picture, the features of which can be captured and/or viewed within the human visible spectrum. a picture, the features of which must be captured and/or viewed by means of equipment which extends viewability beyond the human visible spectrum. an accurate representation (possibly to scale) of an object, typical applications being in architecture and engineering.

Surveying the Reality of Semantic Image Retrieval

Diagram Map/chart/plan Device

179

a representation of the form, function or workings of an object or process. a representation (possibly to scale) of spatial data, typical applications being in geography, astronomy, and meteorology. a symbol or set of symbols which uniquely identifies an entity, e.g., trademark, logo, emblem, fingerprint.

3 Representing and Retrieving the Semantic Content of Different Types of Image We have been fortunate in securing the cooperation of a number of significant picture archives and libraries in the provision of sample requests addressed to their holdings, together with the images selected manually or automatically in response to those requests. We have also collected the metadata associated with each such image. For each of the leaf nodes pertaining to a Simple Image in Figure 1 a sample image drawn from our test collection is shown below. For each image the actual request in response to which the image was selected is shown. The intention is to provide an indication of the way in which real semantic retrieval requirements are expressed, across the spectrum of image types. Also shown is the textual subject annotation associated with each image, by means of which text-matching with the request was achieved. The specificity of subject indexing (and underpinning domain knowledge) which is necessary in order to assist in the realisation of semantic content is noteworthy. Finally, consideration is given to the potential functionality of CBIR techniques with respect to the type of image, and type of query, shown. 3.1 Direct Picture Whether captured by photographic process or created by human endeavour, the Direct Picture is that form of image with which the majority of literature concerned with the Request: A photo of a 1950s fridge Subject Metadata: [8] Title Date Description

Subject Keywords

Roomy Fridge

© Getty Images

Roomy Fridge circa 1952 An English Electric 76A Refrigerator with an internal storage capacity of 7.6 cubic feet, a substantial increase on the standard model. Domestic Life black & white, format landscape, Europe, Britain, England, appliance, kitchen appliance, food, drink, single, female, bending

180

P.G.B. Enser, C.J. Sandom, and P.H. Lewis

indexing and retrieval of still images has been concerned. A comprehensive overview of this literature may be found in [4], and further evidence of the predominance of this form of image may be found in Trant’s survey of image databases available wholly or in part on the Web [5]. The Corel image data set has been a heavily-used resource of this type in CBIR research projects, [e.g., 6] whilst the Brodatz album [7] has figured prominently in the particular context of textural analysis. Recovering the Desired Semantic Content In the image perceptual hierarchy described in section 1 above, the lowest level is thought to involve the engendering of a visual impression by the sensory stimuli, which is first cognitively matched to some form of syntactic equivalence [3]. The two higher levels “… require both interpretation of perceptual cues and application of a general level of knowledge or inference from that knowledge to name the attribute” [9]. The process by which perceptual and interpretive matter in an image is recognised is, as yet, an incompletely understood cognitive phenomenon [3,10]. Following Marr [11], it would seem reasonable to suggest that low-level features within the image might have a role to play, however. Shape may be especially significant, complemented by colour and texture, bringing to bear a previously learned linguistic identifier to generate meaning. Whatever the perceptual processes involved it would seem to be the case that identification is dependent upon the prior existence – and knowledge by the user – of a defining linguistic label. Studies of users of archival and documentary images and footage, in particular, which have revealed the emphasis placed on the recovery of images which depict features (persons, objects, events) uniquely identified by proper name [e.g., 12-15]. Note the example above makes reference to a specific manufacturer and model of the depicted object, whilst enabling requests at the more generic levels of refrigerator or fridge and kitchen appliance to be satisfied. We note also the prevalence of qualification (or ‘refinement’) [12] in requests, which can only be satisfied by textual annotation; e.g. the request for a 1950s fridge. Furthermore, the process of identification often involves context, recognition of which would seem to invoke high-level cognitive analysis supported by domain and tacit knowledge (see the Subject identifier in the above example). In general, contextual anchorage is an important role played by textual annotation within the image metadata. A yet more pressing need for supporting textual metadata occurs when the significance of some visual feature is at issue. Significance is an attribute which is unlikely to have any visual presence in the image. Often reflecting the reason for the image having been created in the first place, and recording the first, last or only instantiation of some object or event, it is frequently encountered in both indexing and querying of image collections [12-17], and involves interpretive processing completely removed from CBIR functionality. When the focus of interest lies with the abstract or affective content of the image, the client wanting images of suffering or happiness, for example, CBIR techniques might

Surveying the Reality of Semantic Image Retrieval

181

offer limited potential - colour can be an effective communicator of mood, for example – but the appropriate cognitive response may be dependent on the presence within metadata of an appropriate textual cue which conditions our interpretation of the image. Overall, a realistic assessment of the potential of CBIR techniques with Direct Pictures, in operational as opposed to artificial, laboratory-based environments, lends support to the image research community’s developing interest in the integration of text-based and CBIR indexing and retrieval strategies, as exemplified in techniques which are designed to uncover the latent correlation between low-level visual features and high-level semantics. 3.2 Indirect Picture Reported usage of the Indirect Picture in the image retrieval literature is most frequently encountered in the field of medicine, where variants of this form of image include x-rays and ultrasound/MRI/CRT scans. Other domains in which Indirect Pictures may be sought include molecular biology, optical astronomy, archaeology and picture conservation. Our research to date has shown that demand for the retrieval of Indirect Pictures held within organized collections is most frequently encountered in publishing and educational contexts. In medical practice, for example, such images are most frequently stored as adjuncts to a specific patient’s record, and it is the latter – as opposed to an image - which is the subject of a retrieval request Request: We are trying to source images for a new pregnancy timeline that we are developing and are finding it very difficult to get any decent ultrasound images. Subject Metadata: [18] Title Short Description. Description of image content

Fetal development Fetal development Fetal development Ultrasound scan, showing fetus

Wellcome Photo Library

Recovering the Desired Semantic Content The above example demonstrates the need for mediated domain knowledge in forging a relationship between ‘fetal [sic] development’ and ‘pregnancy’. In terms of the CBIR potential, however, such images do offer significant opportunities for the semi-automatic detection of certain conditions or objects based upon colour, texture or spatial properties; a number of applications in the medical domain have been reported [e. g. 19].

182

P.G.B. Enser, C.J. Sandom, and P.H. Lewis

3.3 Hybrid Picture Pictures with integral text are frequently encountered in the form of posters and other advertisements, and in the form of cartoons. Importantly, the integral text identifies a significant component of the semantic content of the image and may be the focus of a request for which the image is deemed relevant. Request: A British Rail poster circa 1955. Advertising New Brighton, Wallasey and Cheshire Coast. Woman in foreground - Mersey in background

NRM – Pictorial Collection

Subject Metadata: [20] Title Subject Caption

Keywords

'New Brighton - Wallasey, Cheshire Coast', BR (LMR) poster, 1949. PLACES > Europe: UK > Merseyside Poster produced for British Railways (BR), London Midland Region (LMR), promoting rail travel to the beaches of New Brighton - a district of Wallesey - on the River Mersey estuary. A woman and child are shown sunbathing, with the beach, sea and Mersey seen in the background. Artwork by George S Dixon. Printed by London Lithographic Co, London, SE5. Dimensions: 1010mm x 1270mm 1940s, 20th Century, Ads, Advertisements, Advertising, Art, Bathing, Bathing costumes, Beaches, Brighton, British, British Railway, Children, Coast, Costume, Design, Dixon, Geo S, Fashion, Fashion, 1940s, Girls, Graphic, Graphic design, Happiness, Holidaymakers, Holidays, Leisure, Merseyside, New, New Brighton, Poster, Poster art, Railway, Railway poster, Recreation, Resort, Sea, Seaside, Social, Summer, SUNBATHING, Swimming, Swimming costumes, Swimsuits, The 1940s (1945-1949, Tourism, Tourist, United Kingdom, Wallasey, Woman, Women

Recovering the Desired Semantic Content For image retrieval purposes, the integral text in such an image will normally be transcribed into the metadata, as shown in the above example. The need for this indexing effort could be mitigated by the use of automatic detection of embedded text [e.g., 21], but in general the same limitations of CBIR processing hold for this type of image as for the Direct Picture. 3.4 Drawing At this point in our research we have only encountered this type of image in the context of museum or archival collections; thisis clearly reflected in the textual annotation associated with the example image.

Surveying the Reality of Semantic Image Retrieval

183

Recovering the Desired Semantic Content Indications are that requests for images of this type reflect the need for visual information about highly specific objects, often identified by creator name or created object; the identification has to be resolved by means of textual metadata. However, given the absence of the foreground/background disambiguation problem in this type of image, the potential for recovering (the generic content of) such images by means of shape detection techniques has been recognized; [e.g., 22]. Request: John Llewellyn’s drawing of a Trevithick locomotive

National Railway Museum

Subject Metadata [20] Title Subject Caption

Keywords

Trevithick's tram engine, December 1803. TRANSPORT > Locomotives, Rolling Stock & Vehicles > Locomotives, Steam, Pre-1829 Drawing believed to have been made by John Llewellyn of Pen-y-darran. Found by FP Smith in 1862 and given by him to William Menelaus. Richard Trevithick (1771-1833) was the first to use high pressured steam to drive an engine. Until 1800, the weakness of existing boilers had restricted all engines to being atmospheric ones. Trevithick set about making a cylindrical boiler which could withstand steam at higher pressures. This new engine was well suited to driving vehicles. In 1804, Trevithick was responsible for the first successful railway locomotive. 19th Century, Drawing, Engine, F, Industrial Revolution (1780-18, John, Llewellyn, Llewellyn, John, Locomotive, Locomotives, Steam, Pre-1829, Menelaus, William, Menelaus, P, Pen-Y-Darran, Pre-1829, Richard, Smith, Smith, F P, Steam, Tram, Tram engines, Trevithick, Trevithick, Richard, United Kingdom, Wale, William

3.5 Diagram Diagrams may be encountered in a number of formats and applications, and may incorporate textual or other symbolic data. The semantic content will often be reflected in the title and will frequently be the form in which the request is cast. Recovering the Desired Semantic Content The example above illustrates the need for mediation in order to establish the pertinence of a specific entitled diagram to some condition or focus of interest. It would seem clear that CBIR techniques incorporating textual support, possibly in the form of ontologies or embedded text detection, could have some potential to address retrieval of this type of image.

184

P.G.B. Enser, C.J. Sandom, and P.H. Lewis

Request: the adverse health effects of space travel, specifically long periods of zero gravity … weakening of the heart Subject Metadata: [18] Title Description

ICD Code

Heart block Heart block Colour artwork of cut-away heart, showing right and left ventricles with diagrammatic representation of a right bundle block, usually caused by strain on the right ventricle as in pulmonary hypertension 426.9

Wellcome Photo Library

3.6 Map/Chart/Plan To date, collections of this type of image to which we have gained access focus on historic spatial data, and this is reflected in the types of request addressed to such collections. The requests are necessarily cast in the form of linguistic search statements, to which textual matching with metadata and, as the example below shows, intellectual mediation must be applied. Request: …. a map of central London before 1940. I wish to discover where was Redcross Street, Barbican E.C.1. The current London A-Z does not list any similar address in the E C 1 area. Subject Metadata [23] Title

Physical description Notes

Guildhall Library

Stanfords Library Map of London and its suburbs/ Edward Stanford, 6 Charing Cross Road 1; map; line engraving; 1842 x 1604 mm Extent: Crouch End – Canning Town – Mitcham – Hammersmith. Title in t. border. Imprint and scale in b. border. Hungerford and Lambeth bridges shown as intended. Exhibition buildings shown in Kensington.

Surveying the Reality of Semantic Image Retrieval

185

Recovering the Desired Semantic Content Consideration of CBIR potential in this case would again indicate the possible value of embedded text detection, possibly coupled with shape recognition of cartographic symbols; also colour and textural analysis of image features. Some CBIR applications were reported in [1]. 3.7 Device Such images may integrate a number of visual features, but are most likely to be requested by the identifying title of the device or by picture example. Recovering the Desired Semantic Content As in the case of drawings, such images tend to be context free and, as such, may lend themselves to shape recognition techniques. Significant applications in trade mark matching have been reported [1,24], as has experimental work in fabric design pattern matching [1]. Request: CRESTS: London, Brighton and South Coast Railway Subject Metadata [20] Title

Subject Caption

Keywords

National Railway Museum

Coat of arms of the Southern Railway on a hexagonal panel, 18231947. TRANSPORT > Railway Heraldry > Coats of Arms The coat of arms of the Southern Railway features a dragon and a horse on either side of a shield. 20th Century, Arm, Coat, Coat of arms, Coats of arms, Dragon, Horse, Industrial Revolution (1780-18, LOCO, Rail travel, Railway, Railway coat of arms, Shield, Southern, Southern Railway, SR, Train, Unattributed, United Kingdom

3.8 Complex Image Our research has uncovered many examples of images which are composites or sequences of simple images, where the latter may take any of the forms described above. In such cases the focus of interest would normally lie with the composite image as a single entity. Recovering the Desired Semantic Content For retrieval purposes, complex images would not appear to raise either challenges or opportunities different from those encountered with simple pictures.

186

P.G.B. Enser, C.J. Sandom, and P.H. Lewis

Request: Fish being sold Subject Metadata [25] Title Keywords

Two scenes depicting fishmongers (w/c) merchant scales fish monk buying selling business Medieval Garramand fishmonger

f.57 Two scenes depicting fishmongers (w/c) New York Public Library, USA / The Bridgeman Art Library

4 Conclusion The semantic gap is now a familiar feature of the landscape in visual image retrieval. The developing interest in bridging the semantic gap is a welcome response to the criticism directed at the visual image retrieval research community, one expression of which is that “the emphasis in the computer science literature has been largely on what is computationally possible, and not on discovering whether essential generic visual primitives can in fact facilitate image retrieval in ‘real-world’ applications” [4, p.197]. The project reported in this paper seeks to contribute towards this effort by locating ‘semantic image retrieval’ in the real world of client requests and metadata construction within commercially managed image collections. An analysis across the broad spectrum of image types has identified some of the challenges and opportunities posed by a CBIR-enabled approach to the realisation of semantic image content.

Acknowledgements The ‘Bridging the semantic gap in visual information retrieval’ project is funded by the Arts and Humanities Research Council (MRG-AN6770/APN17429) whose support, together with that of our various contributors, is gratefully acknowledged.

References 1. Eakins, John P. and Graham, Margaret E.: Content-based Image Retrieval. A report to the JISC Technology Applications Programme. Institute for Image Data Research, University of Northumbria at Newcastle, Newcastle upon Tyne (1999)

Surveying the Reality of Semantic Image Retrieval

187

2. Gudivada, V.N. and Raghavan, V.V.: Content-based image retrieval systems. IEEE Computer 28(9) (1995) 18-22 3. Greisdorf, H. and O’Connor, B.: Modelling what users see when they look at images: a cognitive viewpoint. Journal of Documentation 58(1) (2002) 6-29 4. JĘrgensen, C.: Image retrieval: theory and research. The Scarecrow Press, Lanham, MA and Oxford (2003) 5. Trant, J.: Image Retrieval Benchmark database Service: a Needs Assessment and Preliminary development Plan: a report prepared for the Council on Library and Information Resources and the Coalition for Networked Information (2004) 6. Lavrenko, V., Manmatha, R. and Jeon, J.: A model for learning the semantics of pictures. (undated) 7. Brodatz, P.: Textures: a photographic album for artists and designers. Dover, New York (1966) 8. Edina: Education Image Gallery 9. JĘrgensen, C.: Indexing images: testing an image description template. Paper given at the ASIS 1996 Annual Conference, October 19-24, 1996. 10. Rui, Y, Huang, T.S., Chang, S-F.: Image Retrieval: Current Techniques, Promising Directions, and Open Issues, Journal of Visual Communication and Image Representation, 10(4) (1999) 39-62. 11. Marr, David: Vision. Freeman, New York. (1982) 12. Enser, P.G.B.: Pictorial Information Retrieval. (Progress in Documentation). Journal of Documentation 51(2), (1995) 126-170. 13. Armitage, L.H, and Enser, P.G.B.: Analysis of user need in image archives. Journal of Information Science 23(4) (1997) 287-299 14. Ornager, S.: The newspaper image database: empirical supported analysis of users’ typology and word association clusters. In Fox, E.A.; Ingwersen, P.; Fidel R.; (eds): SIGIR 95, Proceedings of the 18th International AGM SIGIR ACM, New York, (1995) 212-218. 15. Enser, P. and Sandom, C.: Retrieval of Archival Moving Imagery - CBIR Outside the Frame? In: Lew, M.S., Sebe, N. and Eakins, J. P. (eds.): CIVR 2002 - International Conference on Image and Video Retrieval, London, UK. July 18-19, 2002, LNCS Series 2383. Berlin: Springer, (2002) 202-214. 16. Markkula, M.; Sormunen, E.: End-user Searching Challenges Indexing Practices in the Digital Newspaper Photo Archive. Information Retrieval 1(4), (2000) 259-285 17. Conniss, L.R; Ashford, L.R; Graham, M.E.: Information Seeking Behaviour in Image Retrieval. VISOR 1 Final Report. Library and Information Commission Research Report 95. Institute for Image Data Research, University of Northumbria at Newcastle’ Newcastle upon Tyne, (2000) 18. Wellcome Trust: Medical Photographic Library http://medphoto.wellcome.ac.uk 19. Hu, B., Dasmahapatra, S., Lewis, P. and Shadbolt, N.: Ontology-based Medical Image Annotation with Description Logics. In Proceedings of The 15th IEEE International Conference on Tools with Artificial Intelligence, Sacramento, CA, USA. (2003) 20. Science & Society Picture Library. < http://www.scienceandsociety.co.uk> 21. Hauptmann, A., Ng, T.D., and Jin. R. Video retrieval using speech and image information. In Proceedings of Electronic Imaging Conference (EI'03), Storage Retrieval for Multimedia Databases, Santa Clara, CA, USA (2003)

188

P.G.B. Enser, C.J. Sandom, and P.H. Lewis

22. Eakins, J.P.: Design criteria for a shape retrieval system. Computers in Industry 21 (1993) 167-184 23. Corporation of London: Talisweb. http://librarycatalogue.cityoflondon.gov.uk:8001/ 24. Eakins, J.P., Boardman, J.M. and Graham, M.E.: Similarity retrieval of trademark images. IEEE Multimedia 5(2), (1998), 53-63 25. Bridgeman Art Library: Bridgeman Art Library – a fine art photographic archive.

Too Much or Too Little: Visual Considerations of Public Engagement Tools in Environment Impact Assessments Ann Shuk-Han Mak, Poh-Chin Lai, Richard Kim-Hung Kwong, and Sharon Tsui-Shan Leung Department of Geography, The University of Hong Kong, Pokfulam Road, Hong Kong {annshmak, pclai, sharonts}@hkucc.hku.hk [email protected]

Abstract. Recently proposed reclamation works due to take place in the Victoria Harbor of Hong Kong have raised questions about their appropriateness and desirability. Although the plans for reclamation had gone through the Environmental Impact Assessment (EIA) process and the submitted report available online, its wordy and technical contents were not well received by the public. The report failed to offer the community at large a better understanding of the issues at hand and to visualize what would become of the proposed site upon project completion. Henceforth, the Environmental Protection Department stipulates that future EIA reports be presented in a format more readily comprehensible than written accounts. This requirement calls for more visual displays, including but not limited to, three dimensional models, maps and photo imageries. In compliance with the requirements and recognizing technological impetus, we structured a web-based platform that makes use of the Geographic Information System technology to explore alternative visual presentation, such as maps, graphics, photos, videos, and animations. Our research has demonstrated that visual resources are viable substitutes to written statements in conveying environmental problems albeit with limitations. This paper shares our knowledge and experience in compiling visual resources and hopes that our integrative effort is a step forward in the development of a more effective public engagement tool.

1 Introduction Environment Impact Assessment (EIA) is a statutory process in Hong Kong. It stipulates that proponents of construction projects must submit EIA reports before environmental permits would be considered by the Hong Kong Environmental Protection Department (EPD). EIA reports have traditionally been text-based and they are now posted on the Internet for open access and comments over a period of not less than three months. The EIA reports, though easily accessible over the Internet, are not well received by the public because of its wordy and technical nature. In an effort to foster continuous public involvement in which the public is kept informed and involved at all stages of development planning and implementation of a major construction project, the EPD has

requested that future EIA reports must be rendered in a format that is “interactive, S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 189 – 202, 2005. © Springer-Verlag Berlin Heidelberg 2005

190

A.S.-H. Mak et al.

dynamic and comprehensive to lay persons” [1]. The EPD has suggested that various visualization techniques (including photographs, three dimensional or 3D models, videos and animation) have enormous power in engaging stakeholders and the public to better appreciate environmental concerns of a development project for two reasons: (i) the lifelike presentation of the project site and environmental issues facilitates visual assessments, and (ii) the visual contents eliminate technical requirements in appraising the desirability or suitability of a construction project [2]. The above presuppositions have led us to develop a web-based platform1 based on the Geographic Information System (GIS) technology to attest the claims. Our system illustrates with the example of proposed reclamation works in the Victoria Harbor of Hong Kong and offers various visualization options (including maps, graphics, photos, videos, and animations) to view related information on the reclamation. Other than being a demonstration system, our project will undertake an evaluative assessment of the utility of various visualization techniques. We offer observations on our creative efforts and problems with various visual recourses in this paper. The system will be open for public assessment and opinion survey until the end of 2005 and the results will be documented in a separate paper.

2 Background of the Project Reclamation activities in the Victoria Harbor have been ongoing since the 1950s. However, recent proposals on the third phase of Central Reclamation (also known as CRIII) provoked overwhelming public opposition and even a lawsuit attempting to block the project [3]. The proposal, designated to ease traffic congestions in the financial district of Central, would reclaim 23 hectares (57 acres) of the Harbor for the construction of a promenade and an underground traffic bypass [4]. The public was concerned about the desirability and suitability of the reclamation given the premise that damages to the Harbor will be permanent and irreversible. It also appears that public opposition against the proposal may not subscribe to sound and educated judgment because of adverse publicity in the news. EIA originated in the United States in 1960s and aimed to consider environmental and social impacts of a project prior to decision-making. Specifically, it was meant to speculate possible environmental impacts early on in the project planning and design stage. Such speculations would help decision-makers and the general public to contemplate ways to reduce adverse impacts or shape projects to suit the local environment [7]. An EIA exercise usually covers such environmental issues as visual impact, air, noise, water, waste, heritage, and ecology. Our web-based platform is intended to enhance the public’s understanding of environmental matters through visual recourses like maps, photographs, videos, 3D models, and animations. These methods of representation, also termed information graphics [5], are most suited for describing landscape and environmental conditions as they seem more intuitive than textual descriptions [6]. The use of information graphics 1

This project was undertaken with financial support from the Hui Oi Chow Trust Fund of the University of Hong Kong and the Environment and Conservation Fund of the Environment, Transport and Works Bureau.

Too Much or Too Little: Visual Considerations of Public Engagement Tools

191

would hopefully make the EIA reports more amenable to the public which, in turn, would facilitate public engagement in informed decision making.

3 Essential Technologies The implementation of the EIA reporting system is made possible through two enabling technologies: GIS and the Internet. The Internet extends accessibility to information from anywhere and at anytime and needs no further introduction. GIS is a system for capturing, storing, checking, integrating, manipulating, analyzing and displaying data which are spatially referenced to the Earth [8]. GIS has been employed to prepare maps and spatial data required in the EIA process. The system has also evolved over the years to support multimedia displays, including videos, photos, graphics, and 3D images. Recent enhancements in GIS processing for 3D displays and web-based dissemination have improved greatly the efficiency of serving photorealistic imageries for ubiquitous access. 3.1 3D GIS Geographic or spatial data have been confined to two-dimensional (2D) portrayal for some time. The primary reasons for the lack of 3D models of sufficient realism were the requirements of massive and cumbersome data processing, as well as an intense appetite for computer storage. Rapid advances in the computing systems with increasing speed and computational efficiency have enhanced opportunities to present landform data in 3D formats. 3D GIS come with extended capabilities to build, visualize, and analyze data in three dimensions. The software has also built-in navigation functions to explore, display, and query the three-dimensional model in a few keystrokes or by clicking with a pointing device. We used the 3D Analyst Extension of ArcGIS (a product of ESRI Limited) to construct our 3D relief model from spot heights and elevation data. Buildings were then extruded to the required heights and added onto the landscape. Aerial photographs were draped onto the 3D model to project a more realistic look. The 3D GIS also supply analytical functions not commonly available in computer-aided drafting programs to evaluate viewshed, perform line-of-sight analysis, interpolate spot heights at any point, construct vertical profiles along linear features, and determine steepness of slopes and so forth. 3.2 Web-Based GIS Graham [9] noted that “the proliferation of the Internet has generated a new public sphere supporting interaction, debate, new forms of democracy and cyber cultures which feed back to support a renaissance in the social and cultural life of cities”. Web-based GIS extends the ease and convenience of accessing information of high visual qualities. The Web technology not only has made it easier to disseminate voluminous data of quite realistic 3D models with sufficient speed over the Internet but also afforded adequate protection against unauthorized data access and systems overwrite. This method of access allows the user flexibility to choose the types of displays residing in the client server by way of Java and HTML forms. The client server

192

A.S.-H. Mak et al.

processes the requests and returns the results in GIF or JPEG graphics formats for posting on the Internet. The resultant GIF or JPEG images is viewable with a standard web browser. We used the ArcIMS (also a product of the ESRI Limited) in this project, to author, to design and deploy the maps and graphics visualizations over the Internet. The web-based GIS also offers functions to interface with VRML models and animated video streams. The use of a web-based system has the potential to break down some of the psychological barriers to solicit viewpoints in public meetings or hearings.

4 System Implementation Our systems development began with data collection, followed by selective data conversion into visual displays that were integrated subsequently into a common interface. Much of our required data about the CRIII Harbor Reclamation was downloadable from the website http://www.epd.gov.hk/eia/english/register/aeiara/all.html maintained by the Environmental Protection Department of the government of Hong Kong. We then classified the data into categories [10] and assigned the method of visual presentation for each. Acknowledging the fact that the majority of public users are not map-readers or GIS professionals, multimedia data (such as text, sound, graphics, and images) linked to geographical features were built-in for much easier browsing and maneuvering over the Internet. For example, 3D representations enable the access of photorealistic models from different viewing directions and zooming levels. Video sequences animate displays of landscape changes over time to augment visual perception. Virtual Reality Modeling Language (VRML) models enable a truly interactive and 3D environment for viewers to walk through or fly to any location in the models. Panorama images permit 360° axial views of the surrounding environment from fixed locations. Other graphics media such as charts and tables are also exploited to provide useful summaries of data and comparisons among different display options. 4.1 The Interface We used the ArcIMS (Internet Map Server by the ESRI Limited) to publish maps, videos, and 3D relief models over the Internet. We limited our development efforts to a few essential and basic functions to prevent undue demand on operational skills of the users. We abided by the standard manner of accessing information on the Internet in the development of user interface and web-based functions. Specifically, each function is a map service fitted with display, query and analytical options for map viewing and manipulation. Hyper Text Markup Language (HTML) files are imbedded in the map services to recall multimedia representations for general access via the web browser. We hope the system will be notably superior in terms of ease of use, objectivity, visualization, and dynamic interactions. The homepage of our demonstration system (http://geog.hku.hk/gis-hr/) has six main options (Fig. 1). The first two options integrate words, numbers, and images

Too Much or Too Little: Visual Considerations of Public Engagement Tools

193

Fig. 1. The homepage contains six navigation options to obtain information about the Harbor Reclamation: (1) Harbor View (2) Environmental Impact Assessment, (3) About the project, (4) Data Sources, (5) Comments, and (6) Disclaimer. The EIA reports are incorporated in the first two options.

used to measure or represent impact. Option 1 focuses on an overview of harbor changes from 1952 straddling to 2012 (the proposed CRIII) and option 2 highlights various aspects of the environmental impact assessment (EIA). Constrained by page size, this paper only accounts for interfaces that help users to literally see implications of the reclamation activities in the Victoria Harbor. used to measure or represent impact. Option 1 focuses on an overview of harbor changes from 1952 straddling to 2012 (the proposed CRIII) and option 2 highlights various aspects of the environmental impact assessment (EIA). Constrained by page size, this paper only accounts for interfaces that help users to literally see implications of the reclamation activities in the Victoria Harbor. 4.1.1 Harbor Changes The Harbor Change option in the Homepage contains four choices representing different types of visual presentation to convey development impacts on the Harbor (Fig. 2). The first two choices depict, in 2D and 3D views respectively, historical changes to the Harbor over the past three decades and potential changes brought about by CRIII. The 2D Harbor Change shows each decade of reclamation in a map that can be turned on or off for individual or concurrent displays (Fig. 2). Each period of reclamation is shaded in distinct colors. The 3D Harbor Change contains a video presentation of time-series changes to the urban landscapes and coastlines along the Victoria Harbor (Fig. 3). 3D scenes enable average viewers to analyze and visualize elevation data in a more "natural" representation. The video sequences capture temporal changes and also highlight in distinct colors the reclaimed areas and buildings erected at each stage of the development. While the 2D displays offer a comprehensive summary of the reclamation works over the years, the 3D views are closer approximations of the real world and the visual impact more readily seen. These two types of displays complement each other to render a better understanding of the sequence of events.

194

A.S.-H. Mak et al.

Fig. 2. The 2D Harbor Change summarizes reclamation works from 1952 to the proposed CRIII in 2014. The panel on the right turns on or off corresponding maps of reclamation activities in selected periods/years. Illustrated here are three reclaimed layers: 1952, 1957- 1971 and 1971-1884 that have subsequently turned on. Icons located on the upper left corner of the map include functions to size the map display in different scales or query data for selected periods. The large icon at the bottom allows user to see a record on how many areas have been reclaimed so far.

Fig. 3. The 3D Harbor Change provides 2 options here. The Slide Show option displays a series of static images simulating different phases of the reclamation. The Movie Clip option uses video streams as illustrated here to show time-series changes of the Harbor from 1952 to beyond the proposed CRIII in the 3D perspective. Areas reclaimed in different reclamation periods and the corresponding buildings are shaded in different colors to highlight the changes in time.

Too Much or Too Little: Visual Considerations of Public Engagement Tools

195

4.1.2 Environmental Impact Assessment (EIA) The EIA option (see Fig. 1) leads viewers into a setting that describes environmental issues relevant to CRIII: landscape and visual, air quality, noise, water quality, waste, and heritage (Fig. 4). The first four environmental issues are selected as exemplary cases for discussion below.

Fig. 4. The Environment Impact Assessment page includes six options on Landscape / Visual, Air Quality, Noise, Water Quality, Waste and Heritage

Visual impact assessment is an exercise that attempts to evaluate “the sensitivity of the affected landscape and visual receptors and the magnitude of change that they will experience” [11]. The scenarios undertaken in our Internet platform attempt to inspect visual changes of the Harbor by comparing before-and-after events of CRIII. The Visual Impact Assessment puts forward several 3D VRML models representing different conditions to convey the degrees of impact upon existing buildings along the waterfront that would be affected (Fig. 5). Three VRML models were created to simulate the following conditions: (i) construction phase during CRIII, (ii) operation phase at the conclusion of CRIII, and (iii) mitigated condition 10 years later. Effects of mitigated situations (enhanced with buildings or new ventilation structures, more trees for enhanced greenery, or additional amenity areas) are also shown in the last model. The models allow viewers to walk through the virtual space to assess changes arising from various mitigated measures. This kind of interactive dialogue is comparable to the making of scaled models for most construction projects but its availability in the cyber environment can extend the virtual experience to a wider group of audience. The Air Quality link contains a series of air quality surfaces to estimate the distribution of selected pollutants in the air at the conclusion of CRIII (Fig. 6). Concentration levels of pollutants - including carbon monoxide, total suspended particulates, and respirable suspended particulates – are presented as choices on the right panel. Viewers may examine these map layers one at a time or cumulatively to assess air qualities by different pollutants.

196

A.S.-H. Mak et al.

Fig. 5. The Visual Impact Assessment uses VRML models to show visually outcomes of various scenarios of the reclamation project to stakeholders. The affected properties are shaded with patterns or colors to convey magnitudes of the visual impact in terms of severity or concentration.

Fig. 6. The Air Quality link shows patterns of diffusion of selected pollutants. The present snapshot displays the concentration levels of carbon monoxide at 1.5 meter above ground level. Function buttons are provided left of the display to facilitate map maneuvering (such as zooming in/out, panning left/right, and refreshing graphics to the original size). A link is also provided on the left below the buttons to retrieve documents explaining the measurement standards of these pollutants.

The Noise page presents a 3D model that simulates noise emission and propagation on the influenced areas from the proposed Central and Wanchai Bypass at its comple-

Too Much or Too Little: Visual Considerations of Public Engagement Tools

197

tion in 2007 (Fig.7). We input the required parameters (traffic network with lanes and widths, projected traffic data, building walls, terrain, population, and land use) into the Lima environmental calculation and noise mapping software (http://test.bksv. com/base/ 2498.asp) to construct the 3D model. Different color intensities were assigned to represent noise impact of different levels from the acoustic environment.

Fig. 7. The Noise page allows viewers to review noise maps displayed in 3D. Gradation of colors on the building facades signifies the corresponding noise levels as shown on the legend to the right.

Fig. 8. The Water Quality option presents water pollutant data gathered from seven monitoring stations. The bar charts illustrated here show levels of concentration of dissolved oxygen for 2003 and 2016. Other pollutant data such as total inorganic nitrogen and ecoli bacteria are also mapped.

198

A.S.-H. Mak et al.

The Water Quality maps highlight concentration levels of pollutants (including dissolved oxygen, total inorganic nitrogen, and ecoli bacteria) measured at seven monitoring stations (Fig. 8). The maps show observed concentrations in 2003 against projected concentrations in 2016 as bar charts to illustrate changes brought about by the proposed reclamation and development. Figures 1-8 summarize the multimedia and visualization options of our demonstration system. While we fully enjoy the aspirations brought by these resources, we are also aware of problems associated with the vivid and compelling displays that may deceive and mislead our audience or overwhelm their senses and judgments. We came across many questions concerning design options and user requirements in our attempts to transform textual and technical reports into maps and other media displays. First and foremost is the lack of guidelines and design principles even though multimedia and 3D displays have been employed widely in planning processes for many decades.

5 Some Points to Ponder in 3D Visualization Those who embark on 3D visualization often have to face the paradox of how much rendering is enough? There is widespread belief that the more realistic looking the model is, the more real world one can associate and the more sensibility is aroused in one’s mind to foster appreciation and intuitive reasoning. This belief has made some practitioners to go to extreme lengths to dress up their 3D models (e.g. put textures on four sides of the buildings; populate the milieu with agents such as people, vegetation and traffic). This approach, however, has a high price to pay. Notwithstanding the amount of efforts expended in the creation of a realistic representation, the resultant model inevitably succumbs to a large file size which poses a major challenge to many personal computers in data download and handle. Even for broadband users, one can expect a single video clip of 100Mb taking about 2 to 4 minutes to download. Such a long download time may erode the interests of some viewers who may quit prematurely. An estimated 20% of viewers were reportedly lost for every 10 seconds used in loading the graphics. Are there ways to keep the file size realistic for web dissemination? In this attempt, we exported graphics into GIF or JPEG and saved videos as MPEG (instead of AVI) to command a much smaller file size. We also employed several video compression technologies such as MPEG-1 or MPEG-2. Yet this is a challenging task, if not formidable, to find one right solution that fits a wide range of existing transmission and the desired image quality. A model is an abstract and often simplified conceptual representation, and it is constructed for the purpose of explaining a process or predicting the processes under hypothetical conditions [13, 14]. However, the 3D terrain model presented in the demonstration system can hardly be called a model of simplicity. Rather, it is a close approximation of the real world embroidered with complex details that may not be well perceived by the public at large. We conducted a simple test using two kinds of 3D models of the Victoria Harbor (Fig. 9). The 3D terrain model was created by superimposing aerial photographs of the scenes onto the 3D terrain model. The second was a quick rendition of a 3D ter-

Too Much or Too Little: Visual Considerations of Public Engagement Tools

199

rain model created from topographic data and upon which buildings were extruded to the required heights. We then asked a random group of individuals to indicate which of the two presentations is easier to read and to tell changes taking place in the harbor at different phases of the reclamation. Most of them (83 % of 42 individuals surveyed) selected the one without the aerial photographs.

Fig. 9. The 3D display on the left was made by superimposing aerial photographs on the terrain model. The 3D display on the right was constructed from terrain data upon which buildings were extruded to the required heights. The displays were presented to a group of viewers and most preferred the one on the right because of its uncomplicated scene which is more understandable and easier to detect change.

The test, though not entirely conclusive, has bestowed upon us the following insights: Firstly, models with photorealistic details thought to be a closer approximation to the real world may not necessarily be easy to read and understand. Perhaps too much detail will cloud perception and affect interpretation. Secondly, how can we determine what level of rendering is too much or too little? Should we attempt to model infrastructural and natural landscapes without stochastic events (such as adding vehicles on motorways and human figures in city centers)? Thirdly, are there methods to recognize redundant visualization and eliminate or generalize it to retain substantial contents on the one hand and increase efficiency on the other? Finally, is 3D visualization a better representation than 2D maps? Fisher et al. [15) argued that 3D graphs may be visually more pleasing but simple 2D graphs are better in terms of extracting information with respect to accuracy and ease. Aside from the above concerns, there is also misperception that images are immune from biases and inaccuracies because they are created by automated and objective processes requiring no human intervention. 3D models, like any other visual displays, can be tampered with misleading and deceptive information. For example, we have to exaggerate the vertical distance (or the height) of a landscape and the structural features therein so that these components would appear to be visually balanced against the horizontal distance on the ground. The vertical exaggeration factor is rather arbitrary and subjective but the results may be spectacularly different to impact one’s perception in the virtual setting. Strangely, the compelling and vivid rendering seems to sway the most skeptical audience from further doubts about the accuracy of the images. This observation is perhaps best summarized by Eiteljorg [16]: “As more and more people have realized the potential of the technology, some have

200

A.S.-H. Mak et al.

allowed the power of the presentation to overcome concerns about the accuracy of the information being presented.” Equally true are viewers who tend to ignore or undermine the message and information beneath the artful appeals of 3D models. These visual resources are also fashioned with pleasing and entertaining elements that easily captivate our viewers. Viewers are inclined to seek fun and entertainment from the interactive models. Public service officers have begun to exploit this inclination by making EIA reports more like video games to engage the more reluctant viewers into the EIA processes [17]. If EIA visualization were to endow with entertainment and visual delight, how can viewers be encouraged to look attentively for the information subscribed within the model, maintain their sense of doubt over possible errors, and keep a vigilant eye for missing details? We truly believe that 3D visualization makes a much better public engagement tool than the conventional text-based reports; however, we must not exaggerate its benefits beyond what it can offer. At the same time, we must admit there is so much we do not know about 3D representations. We need some good studies on how various visual techniques impact viewers’ perception. Research to develop visualization guidelines are needed to gauge the design of images that will not overwhelm our sensible judgment or dispel our intellectual inquisition.

6 Conclusion In most situations involving many stakeholders, a decision may not be made simply by a single person but by a group of people after reaching some agreement. In an event of EIA, encouraging public participation is particularly important during the process of open deliberation because of the following reasons. First of all, people may contribute vital environmental information to support decision making for the authority in charge. Furthermore, the process of participation will not only satisfy the right of public awareness but also establish consensus toward a feasible alternative in advance. Innovative applications of information technologies, including 3D and webbased GIS tools, are used in ways that are responsive to the complex and dynamic realities of urban environments. By integrating words, numbers and images in an information-rich environment, the impact and long-term sustainability of alternative scenarios of infrastructure development can be visualized and assessed as if they were actually built. Designing for the future through visualization help us make informed planning and design decisions. Our demonstration system employs the concept of interactive visualization for promoting public participation in EIA. To make the general public aware of EIA related information, our system takes full advantage of the 3D and web-based GIS to deliver messages in multimedia formats. The easy to use and intuitive systems interface would hopefully make more people willing to take part in EIA and thus put public participation in practice. In consequence, the authority in charge of EIA may make more appropriate decisions in light of the results from public participation. Through the project, we have proved that the 3D visualizations are effective means to describe and characterize the landscape and environmental information. However,

Too Much or Too Little: Visual Considerations of Public Engagement Tools

201

it also reveals some problems associated with its vivid and compelling appearance. The model can be overcharged with so many realistic details that it is no longer easy to read and comprehend. Its persuasive ability may overcome concerns about the accuracy of the information being presented. Moreover, its pleasure appeals may also lead to entertainment instead of scholarly information. This paper shares our knowledge and experience in compiling visual resources and hopes that our integrative effort is a step forward in the development of a more effective public engagement tool. We must also emphasize that our project is application-driven rather than technology-driven. Technology serves as a means to an end – not an end in itself. Our exploration is tailored to the uniqueness of the place, circumstances, and public needs. The many technological advances are relevant only because they were employed in response to the greater needs of public participation in the EIA processes.

References 1. EPD (Environmental Protection Department, HKSAR), Environmental Assessment & Noise Division). 3-Dimensional EIA Public Engagement Tools. (2001) http://www. epd.gov.hk/eia/3deia/3DEIA.pdf [accessed January 2005] 2. Au, E.: Opening address on Seminar on Enhancing Public Involvement in Noise Assessment, Seminar leaflet (2004). 3. Lord, P.: 12,000 in Harbor March. The Standard (2004) 4 July. http://www. thestandard.com.hk/news_detail_frame.cfm?articleid=47262&intcatid=42 [accessed January 2005] 4. Housing, Planning and Lands Bureau. Our Harbor Front. http://www.hplb.gov.hk/cr3/ eng/home/index.htm [accessed January 2005] 5. Monmonier, M. S. How to lie with maps. 2nd ed. University of Chicago Press, Chicago (1996) pg. 12. 6. Fonseca, A. and C.: Gouveia, Environmental Impact Assessment Using Multimedia GIS. (1994) http://gasa.dcea.fct.unl.pt/gasa/gasa97/gasa97/xana/papx3/papx3.htm [accessed January 2005] 7. UNEP (United Nations Environment Program): Environmental Impact Assessment (EIA). http://www.uneptie.org/pc/pc/tools/eia.htm [accessed January 2005] 8. AGI, Association for Geographic Information: GIS Dictionary: Definition of GIS. http://www.agi.org.uk/resources/index.htm [accessed January 2005] 9. Graham, S.D.N.: Flight to the Cyber Suburbs. The Guardian (1996) April 18, pp. 2-3. 10. Cartwright, W.: New media and their application to the production of map products. Computers & Geosciences 1997, 23, vol. 24, 447-456. 11. Institute of Environmental Assessment and The Landscape Institute (IEATLI): Guidelines for Landscape and Visual Impact Assessment (Second Edition), SPON Press, London, (1995) 12. EPD (Environmental Protection Department, HKSAR): Wan Chai Bypass and Island Eastern Corridor Link (EIA-057/2001), (http://www.epd.gov.hk/eia/english/register/aeiara/ all.html (as accessed January, 2005) 13. Gruber, T.R., A.: Translation Approach to Portable Ontology Specifications, Knowledge Acquisition (1993) vol. 5, 199-220. 14. Storey, V.C, et al.: An Ontology for Database Design Automation, Conceptual Modeling, (1997) 2-15.

202

A.S.-H. Mak et al.

15. Fisher, SH, Dempsey, JV and Marousky, RT.: Data visualization, preference and use of two-dimensional and three-dimensional graphs, Social Science Computer Review (1997) 15, vol. 3, 256-263. 16. Eiteljorg, H. II: Virtual Reality and Rendering. (1995) http://www.csanet.org/ newsletter/feb95/nl029508.html (as accessed January, 2005) 17. Oriental Daily: EIA Data Truckle in Video Game (Chinese) (2004) July 24.

Active Landmarks in Indoor Environments Beatrix Brunner-Friedrich and Verena Radoczky Department of Geoinformation and Cartography, TU Vienna, 1040 Vienna, Austria {brunner, radoczky}@cartography.tuwien.ac.at http://cartography.tuwien.ac.at/

Abstract. Landmarks are an important enhancement for pedestrian navigation systems. They are not only aids at decision points but they are also an affirmation to the user that he is still on the correct route. Especially in indoor environments where the density of conventional landmarks is rather low, the implementation of so called “Active Landmarks” is an enrichment to the system. This notation derives from the fact that information is actively sent to the handheld device without any user interaction. That way the user receives information from and about a specific landmark, especially concerning its position and consequently the user’s position. This is particularly important within buildings where the user needs detailed position information. Because of the lack of outstanding elements, there are many possibilities to get lost and orientation is much more difficult than outdoors. Nevertheless positioning techniques are scarcely offered and in case where they are available their usage is cost intensive or not accurate enough. In this article the importance of Active Landmarks and their implementation with the help of new technologies like RFID is discussed.

1 Introduction Within the last few years navigation systems have gained more and more importance. Especially in the outdoor environment location based services play an important role for supporting the wayfinding process. Car drivers have started to trust in the information provided by car navigation systems and even pedestrians are gaining interest in reliable guiding instructions. Most of these systems are limited to outdoor areas, whereas wayfinding within buildings has mostly been neglected so far. Even though some museums and exhibitions offer digital guiding services to their customers, this kind of giving directions is not very popular until today. Various reasons are responsible for this scarce usage: positioning sensors are rarely available within buildings and hardly fulfil the minimum conditions concerning range and cost. Because of the expensive infrastructure the user has to expect high costs when using the system. Moreover the communication of 3D indoor environments on twodimensional small screens is a difficult task that influences, if the user recognizes his surroundings and therefore if he successfully reaches the target. For that reason user acceptance has not reached a high level so far. Nevertheless, especially in complex buildings, visitors often need guidance. One of the main disadvantages inside of buildings affects the sense of orientation: because of the third S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 203 – 215, 2005. © Springer-Verlag Berlin Heidelberg 2005

204

B. Brunner-Friedrich and V. Radoczky

dimension height people tend to lose orientation a lot easier within buildings than outdoors [Fontaine and Denis 1999; Lemberg 1996; Radoczky 2003]. Therefore we can conclude that it is necessary to provide indoor navigation systems that are user friendly and easy to implement. To overcome the problem of expensive sensors cheaper technologies have to be investigated and their adoption as navigation aids needs to be tested.

2 Definition of Landmarks and Active Landmarks To communicate directions it is not enough to provide just a map or a list of left/right turns. Due to the results of several researches it is obvious that more information is necessary to fulfil the user’s needs [e.g. Denis and Michon 2001; Elias 2002; Tversky and Lee 1999]. The user feels more comfortable, if there are more aids given [Elias 2002]. Humans need directions with additional information – this means they need landmarks. A landmark is something of importance and it is placed in a central or outstanding position. It has a special visual characteristic and it is unique in its function or meaning [Sorrows and Hirtle 1999; Elias 2002]. Raubal and Winter [2002] define a landmark as “an object or structure that marks a locality and is used as a point of reference”. Therefore it is an aid for the user in navigating and understanding space [Sorrows and Hirtle 1999]. Landmarks can support the navigation-process. While these conventional landmarks have to be found, identified and verified by the user himself, Active Landmarks are based on a different approach. They search for the user and build up a spontaneous radio contact via an air-interface where they are identified by the navigation system. Due to this behaviour the following differences to conventional landmarks occur: • • • •

The kind of interaction between the landmark and the user is the main distinction between conventional and Active Landmarks. Active Landmarks do not need to fulfil landmark criteria (visual criteria, outstanding meaning or prominent location) all the time because the finding process does not rely on visual criteria. If they fulfil landmark characteristics, they can serve as conventional and as Active Landmarks. While conventional landmarks only act as an aid in the navigation process Active Landmarks are additionally capable of providing further position-based information.

3 Functions of Landmarks and Active Landmarks Both conventional landmarks and Active Landmarks dispose of several functions [Denis and Michon 2001; Elias and Sester 2002]: • • •

They are important for the user to form a mental model of the environment. Landmarks signalise where a crucial action should take place. Landmarks allow “re-orientation” at (potential) decision points.

Active Landmarks in Indoor Environments

205

According to Denis and Michon [2001] this is the main function of landmarks. They provide information about important maneuvers to be performed or not to be performed at points where it is possible that changes in direction appear. • With the help of Landmarks the user can approve that he is on the right way. Concerning the differences in the definition of landmarks and Active Landmarks further functions, which can only be accredited to Active Landmarks, can be derived [Brunner-Friedrich 2003a]: •

•

•

Positioning of pedestrians Some situations (e.g. at crucial situations) require a higher positioning accuracy than others, but unfortunately conventional positioning systems are not able to grant this. If the exact position of the pedestrian is needed, an additional aid has to be provided. Î Active Landmarks are an aid at decision points. Sometimes the positioning via GPS, e.g. in indoor environments, is too inaccurate, because in these areas the satellite reception is usually too poor to deliver the accurate position. Î Active Landmarks are needed in indoor areas. Verification of a landmark If the verification of a landmark is difficult, the user can be doubtful, if he has found the correct place. An Active Landmark reports the current location to the user. This is especially helpful, if no significant peculiarities can be found. This aspect is very important for indoor environments. For example staircases or corridors tend to be hardly distinguishable in most cases. In the example (see Fig. 1) it would be very difficult to identify the correct corridor. So the user can hardly verify, if he has found the correct location. Î Active Landmarks are an aid to verify the users assumption of his position. Further information about sights A route can be close to important places of interest, for example a festival room. If the pedestrian passes such sights, he could be interested in information

Fig. 1. Similar corridors within the same building

206

B. Brunner-Friedrich and V. Radoczky

about them. It would be more comfortable to the pedestrian, if he does not need a tourist guide to look for this information but to obtain the information automatically by passing the room or the object. This also enables an easier way to keep the information up-to-date than in a printed form. Î Active Landmarks should be placed at points of interest. Active Landmarks can also serve as conventional orientation aids. But this is not their main task. So it does not matter, if they have no outstanding visual characteristics. This leads to consequences in their derivation because it simplifies the process of the selection.

4 Derivation of Landmarks For indoor navigation the user needs a higher density of landmarks because a shift in direction occurs more often than outdoors, but there is less variety of landmarks indoors. To derive landmarks for indoor navigation some methods can be applied: Method 1: Raubal and Winter [2002] base the gathering of landmarks on the aspect of the structural distinctiveness. This method examines the nodes, edges and regions in the meaning of their outstanding position in relation to their environment. The following ratings are done: • Node degree Node degree = number of edges adjacent to one node, for example the following node has the node degree three:

Fig. 2. Node degree three

Whereas the next example shows a node with degree four: Edges Edge degree = sum of the node degrees of the terminal points minus two and is therefore the sum of the adjacent edges to the respective edge. For example the edge between the two nodes mentioned above shows an edge degree of five (three + four – two). The difference of a degree to the average of this degree in the net is named Salience [Raubal and Winter 2002]. It is assumed that a node or an edge with a high degree is suitable as a landmark. This means that central points within a building where stairs and elevators lead to should function as landmarks. •

Active Landmarks in Indoor Environments

207

Fig. 3. Node degree four

Fig. 4. Edge degree five

Method 2: This method is adapted to the user’s needs [Brunner-Friedrich 2003b]. Here the process of landmark derivation can be divided into two steps: •

• •

On the basis of directions which have been given by pedestrians, landmarks are extracted. The results of this analysis are common aids and common descriptions which are often used in directions. With these specifications a landmark catalogue can be established. The results are universally valid. Potential landmarks: elevator, stairs, floor, door, swing door, crossing corridor. In a second step the catalogue is applied to a certain area. The aim of this method is to use data that can be derived without on-site inspection but instead uses maps and floor plans.

Because the landmark derivation for indoor navigation is short on experience method 2 seems to be a good way to assure that the user’s needs are respected. If the results of this derivation are contemplated, it can be realized that most of the potential landmarks are edges, nodes or barriers. This is similar to Winter’s method (method 1). So the best possibility to derive landmarks indoor would be a combination of method 1 and method 2. In a first step the network of all possible indoor paths is analysed and salience nodes and edges are calculated. Afterwards this result is assured or disproved by applying method 2. The following example shows the derived landmarks within the Vienna University of Technology.

208

B. Brunner-Friedrich and V. Radoczky

Fig. 5. Derived landmarks in the Vienna University of Technology – method 1

Fig. 6. Derived landmarks in the Vienna University of Technology – method 2

5 Derivation of Active Landmarks The derivation of Active Landmarks differs from the methods described above because it is not necessary for this kind of landmarks to fulfil landmark characteristics, but both methods, which are described above, show where users expect and need additional help in the navigation process. This means, based on these two methods, important locations where Active Landmarks should be placed can be discovered. Furthermore the derivation should concern the functions of Active Landmarks mentioned above.

Active Landmarks in Indoor Environments

209

Beside the functions the allocation of Active Landmarks should be considered (see chapter “Distribution of Active Landmarks”). Before this step can be done it is necessary to choose an adequate technique which is able to establish a spontaneous connection between an Active Landmark and the user. Based on the chosen technique different ranges occur and different distributions are asked. 5.1 Indoor Positioning Techniques Due to obstruction effects the achievable accuracy of conventional GPS systems, which are a common method for positioning in outdoor environments, can hardly be used indoors, so different techniques have to be investigated and new possibilities for indoor positioning have to be explored. To enable the communication between Active Landmarks and the user a spontaneous wireless connection is necessary. There are several possibilities for communication via an air-interface: Bluetooth, WLAN (Wireless Local Area Network), IrDA (Infrared Data Association) or RFID (Radio Frequency Identification). All these options differ in their range, in their transfer rate and in their transmission reliability, but they all have in common that the user needs an adequate handheld device. •

•

•

•

Bluetooth: Bluetooth provides a way to connect and exchange information between different devices. Usually it is used to build up connections between personal digital assistants (PDAs), mobile phones, laptops, PCs etc. and as such functions as a replacement of cable, infrared and other connection media. It works quietly, unconsciously and automatically in the background (Mahmoud 2003). Bluetooth devices, which are within a range of 10 meters, are able to communicate with each other. While this technology is an enhancement for outdoor navigation, an application within buildings is not advisable because of overlay effects. WLAN: A wireless LAN is a flexible data communication system. Using electromagnetic waves, WLANs transmit and receive data via an air interface. Similar to Bluetooth, WLAN can be used as a location determination tool. Unfortunately its range is up to 100 meters, which again makes it unsuitable for pedestrian navigation systems. IrDA: Infrared Data Association is a standard communication protocol for short range exchange of data via infrared light. It can be used to build up personal area networks. The range of this method, which lies between one and two meters, would be ideal for indoor navigation. A main disadvantage of this system is the necessity to turn the handheld device such that it faces the directional sender. RFID: Radio Frequency Identification (RFID) is a method to read and save data without touch and intervisibility. Mostly it is used in the consumer goods industry for the contactless transmission of information for product identification [Retscher and Thienelt, 2004]. The term RFID describes a complete technical system that comprises of the transponder, the reader with antennas and the system it is used for (e.g.: billing system, warehouse organisation system, …). Data is stored on the transponders which receive and provide information via radio waves. The range differs depending on the size of the antenna and the method of construction. The use of senders with a short range of up to two meters would be ideal for indoor environments. Moreover a system with these transponders causes very low costs (20 to 40 U.S. cent per tag) [www.rfidjournal.com, 17. 03. 2005].

210

B. Brunner-Friedrich and V. Radoczky

5.2 Distribution of Active Landmarks According to the chapters above landmarks should be placed at (potential) decision points. This means, that at each door and at corridor crossings landmarks are necessary in order to guarantee wayfinding success. Therefore the density of landmarks is usually rather high and their range does not need to be very high. Actually it is even better, if the range is low, because if Active Landmarks are located too close to one another, an overlay of their transmission range is caused [Brunner-Friedrich 2003b].

Fig. 7. Overlay of the transmission range

This overlay must be avoided because in the intersection area problems can occur. It can be difficult to detect to which Active Landmark the connection is actually established. This problem is not only limited to horizontal overlays, because also vertical ranges have to be considered. Most indoor positioning systems are able to build up connections through walls and ceilings, which can cause problems when potential Active Landmark positions are situated above each other. This will usually be the case because the structure of different floors in one building often looks alike. Under consideration of the aspects discussed above (functions of Active Landmarks, accuracy of positioning, avoidance of overlays), an optimum range of 1-2 meters seems logical. If situated in the timber set of the door, which offers the advantage of being available from both sides of the door, a range of one meter is sufficient in most cases. Only if the user moves along wide corridors, it can not be guaranteed that he passes the doors within a distance of one meter. In these situations the range should be adapted up to two or even three meters, depending on the height of the ceiling and the room structure behind the doors. In the following example both possibilities are visualised (Fig. 8 and 9): The figures clearly show, that problems occur at adjacent doors (see room No. 10, 14, 16, 16A and 18). In these areas a range of one meter seems to be optimal whereas everywhere else a higher range should be applied provided that some landmarks are slightly moved.

Active Landmarks in Indoor Environments

Fig. 8. Active Landmarks with a range of one meter

Fig. 9. Active Landmarks with a range of two meters

211

212

B. Brunner-Friedrich and V. Radoczky

6 Communication of Active Landmarks Beside some specific functions (see “Functions of Landmarks and Active Landmarks“) Active Landmarks can serve as conventional landmarks. So communication (visualization) in a navigation service is usually advisable. In case the Active Landmark does not fulfil conventional landmark criteria, it might be sufficient to solely visualize it while the user actually passes by. Depending on the cartographic communication of directions there are several possibilities to display landmark information. The presentation of route directions ranges from a 3D walkthrough guided by a virtual scout accompanied by spoken text down to simple text explanations indicating the directions [Thakkar et al. 2001]. Most of these methods address the visual sense: maps, schematic maps, sketch maps, written text, pictures and video-sequences. As an enhancement also acoustic elements can be used (e.g. spoken text). But disadvantages have to be considered: surrounding noise (e.g. from traffic) interferes with the perception of sounds delivered by the guiding-system [Scharlach and Müller 2002]. The other way round can also be a problem. If there is a rather silent environment (e.g. in a streetcar), the loud speaking guiding-system may disturb other people. Nevertheless, because of small displays and low resolution acoustic elements become more important [Buziek 2002]. When maps are used to show landmarks it is necessary to highlight these features for example by marking with a pointing arrow or with strong colors. This kind of visualization is rather easy to understand. If the landmark is the only object colored, it is not necessary to add a legend [Elias 2002]. The user understands that this feature is remarkable. Instead of highlighting also symbols can be used.

Fig. 10. Examples of self-explanatory symbols (indoor environment)

They should be simple and self-explanatory because the use of a legend should be avoided. Because of the small display a legend cannot be placed next to the map. So it would have to be shown as an add-on information (e.g. similar to a tooltip). But this enhancement means another “mouse click” in the system and therefore it is rather inconvenient for the user. Self-explanatory symbols can facilitate a wayfinding process [Brunner-Friedrich 2003b]. Geometric symbols would be a further possibility. In some cases they are more adequate for small displays. The user can read them more easily, but he will not be able to interpret them without a legend. Photos on the other hand are usually easy to interpret but might take too much time to compare to reality [Radoczky 2003]. As mentioned above also text can be used to communicate landmarks. Either the whole navigation system is represented without graphic which means directions are

Active Landmarks in Indoor Environments

213

given in written form and are enhanced with textually represented landmarks or the system is based on maps and landmarks are labeled with text. Audible presentation of information is similar to the written text. The information content about the landmark is similar, the only difference is the mode of communication. Spoken text should be used as an additional enhancement. It is not a good support when it is used as the only presentation form because it can’t give an overview of the environment. Beside this, most of the users are used to visual information [Thakkar et.al. 2001].

7 Results Several researches show that guiding instructions should be enhanced by the use of landmarks [e.g. Denis and Michon 2001; Elias 2002; Tversky and Lee 1999]. They are necessary to give an efficient aid for the wayfinding process and they also facilitate the creation of a mental map. Recapitulating it can be said that Active Landmarks are an improvement in a pedestrian navigation service. If the technical problems are solved, an implementation seems easy to realize and useful for a support in the navigation process. The derivation of Active Landmarks is easier than of conventional ones because they need not to fulfill landmarks characteristics. It is not necessary to acquire Active Landmarks with big effort. They can be determined by analyzing their functions. As a basis of their derivation a map which contains sights can be used. As the main functions of Active Landmarks the improvement of the positioning function, the simplification of the verification of a landmark, and the possibility to serve enhanced information can be mentioned. Beside these tasks Active Landmarks can serve as conventional landmarks. So visualization in a navigation service is advisable. The implementation of Active Landmarks in indoor environments is already offered in museums where users are guided through an exhibition. Active Landmarks are placed at exhibits where a spontaneous connection is established the moment a visitor passes by. That way the user’s attention is called to the exhibit and the visitor is supplied with additional information. A prototype of such a system has been tested by Oppermann [2003] and results showed that Active Landmarks were widely accepted and approved by visitors.

8 Conclusions The use of Active Landmarks is an enhanced support in a pedestrian navigation system. As the main functions of Active Landmarks the improvement of positioning, the supported verification of landmarks, and the possibility to provide additional information can be mentioned. Beside these tasks Active Landmarks can serve as conventional landmarks. Their distribution is dependent on their functions, the users’ needs and the structure of the building. Moreover it should avoid overlays. Another problem is the need for a big amount of active devices to cover a certain area and to offer an adequate density for the navigation support. The resulting costs

214

B. Brunner-Friedrich and V. Radoczky

could be lowered and requirements could be fulfilled by the implementation of an RFID system.

References 1. Brunner-Friedrich, B. (2003a): Modellierung und Kommunikation von Active Landmarks für die Verwendung in Fußgängernavigationssystemen. In: Strobl, J. et.al. (Hrsg.): Angewandte Geographische Informationsverarbeitung, AGIT-Symposium. Herbert Wichmann, Heidelberg. 2. Brunner-Friedrich, B. (2003b): Reports within the project: „Modellierung und Kommunikation von Landmarks für die Verwendung in Fußgängernavigationssystemen“; unpublished. 3. Buziek, G. (2002): Geoinformation im mobilen Internet – Aspekte der Kommunikation und Wahrnehmung. In: Kelnhofer, F., M. Lechthaler, K. Brunner (Hrsg.): TeleKartographie & Location Based Services. Geowissenschaftliche Mitteilungen, Schriftenreihe der Studienrichtung Vermessungswesen und Geoinformation Technische Universität Wien, Heft Nr. 58, Wien. 4. Denis, M., P.-E. Michon (2001): When and Why are Visual Landmarks Used in Giving Directions. In: Montello, D. R. (Hrsg.), Spatial Information Theory: Foundation of Geographic Information Science. Lecture Notes in Computer Science, Springer Verlag, Berlin. 5. Elias, B. (2002): Erweiterung von Wegbeschreibungen um Landmarks. In: Seyfart, E. (Hrsg), Publikationen der Deutschen Gesellschaft für Photogrammetrie und Fernerkundung, Band 11, S. 125 - 132, Potsdam. 6. Elias, B.; M. Sester (2002): Landmarks für Routenbeschreibungen, In: GI-Technologien für Verkehr und Logistik, IfGI prints, Band 13, Institut für Geoinformation, Münster. 7. Fontaine, S., M. Denis (1999): The Production of Route Instructions in Underground and Urban Environments. In: Frewka, C. & D. M. Mark (Eds.), Proc. Of COSIT99, LNCS 1661, Springer, Berlin, Heidelberg, New York. 8. Lemberg, D. (1996): Wayfinding in Caves – A Proposed Curriculum for a Short Course in Self Rescue. In: Proceedings of the 1996 National Speleological Society Convention, Salida, Colorado, August 5th, 1996. 9. Mahmoud, Q.H. (2003): Wireless Application Programming with J2ME and Bluetooth; http://wireless.java.sun.com/midp/articles/bluetooth1/. (17.03.2003) 10. Oppermann R. (2003): Ein Nomadischer Museumsführer aus Sicht der Benutzer. In: Szwillus G., J. Ziegler (Hrsg.): Mensch & Computer 2003: Interaktion in Bewegung. Stuttgart: B. G. Teubner; S. 31-42. 11. Radoczky, V. (2003): Kartographische Unterstützungsmöglichkeiten zur Routenbeschreibung von Fußgängernavigationssystemen im In- und Outdoorbereich. Diplomarbeit am Institut für Kartographie und Geo-Medientechnik, TU-Wien. 12. Raubal, M., S. Winter (2002): Enriching Wayfinding Instructions with Local Landmarks. In: GISscience 2002; Lecture Notes in Computer Science, Springer, Berlin. 13. Retscher G., M. Thienelt (2004): Die Zukunft des Einkaufens - Was können Geodäten dazu beitragen?. Allgemeine Vermessungs-Nachrichten (AVN), 111, 11-12; S. 387 - 393. 14. Scharlach, H., J. Müller (2002): Multimediale thematische Kartographie auf Handheld PCs: Potentiale und Grenzen. In: Kelnhofer, F., M. Lechthaler, K. Brunner (Hrsg.): TeleKartographie & Location Based Services. Geowissenschaftliche Mitteilungen, Schriftenreihe der Studienrichtung Vermessungswesen und Geoinformation Technische Universität Wien, Heft Nr. 58, Wien.

Active Landmarks in Indoor Environments

215

15. Sorrows, M. E., S. C. Hirtle (1999): The Nature of Landmarks for Real and Electronic Spaces. In: Freksa, C., D. M. Mark (Hrsg.), Spatial Information Theory: Cognitive and Computational Foundation of Geographic Information Science. Lecture Notes in Computer Science, Springer Verlag, Berlin. 16. Thakkar, P., I. Ceaparu, C. Yilmaz (2001): Visualizing Directions and Schedules on Handheld Devices. A Pilot Study of Maps vs. Text and Color vs. Monochrome. Project at CMSC 838S - Seminar Advanced Usability (for Mobile Devices) by Ben Shneiderman; University of Maryland, Department of Computer Science; http://www.cs.umd.edu/ class/fall2001/cmsc838s/CMSC838Project.html. (10.12.2002) 17. Tversky, B., P. U. Lee (1999): Pictorial and Verbal Tools for Conveying Routes. In: Freksa, C., D. M. Mark (Hrsg.), Spatial Information Theory: Cognitive and Computational Foundation of Geographic Information Science. Lecture Notes in Computer Science, Springer Verlag, Berlin.

Image Annotation for Adaptive Enhancement of Uncalibrated Color Images Claudio Cusano, Francesca Gasparini, and Raimondo Schettini Dipartimento di Informatica Sistemistica e Comunicazione, Università degli studi di Milano-Bicocca, Via Bicocca degli Arcimboldi 8, 20126 Milano, Italy {cusano, gasparini, schettini}@disco.unimib.it http://www.ivl.disco.unimib.it/

Abstract. The paper describes an innovative image annotation tool, based on a multi-class Support Vector Machine, for classifying image pixels in one of seven classes - sky, skin, vegetation, snow, water, ground, and man-made structures - or as unknown. These visual categories mirror high-level human perception, permitting the design of intuitive and effective color and contrast enhancement strategies. As a pre-processing step, a smart color balancing algorithm is applied, making the overall procedure suitable for uncalibrated images, such as images acquired by unknown systems under unknown lighting conditions.

1 Introduction The new generation of content-based image retrieval systems is often based on whole image classification and/or region-based image classification [1]. Several works have been therefore proposed for labeling the images with words describing their semantic content. These labels are usually exploited for increasing retrieval performance. Among these works, Fan et al. [2] have proposed a multi-level approach to annotate the semantics of natural scenes by using both the dominant image components and the relevant semantic concepts. Jeon et al. [3] have proposed an automatic approach to annotating and retrieving images assuming that regions in an image can be described using a small vocabulary of blobs generated from image features using clustering. Our interest in image annotation is here related to the application need of performing automatically selective color and contrast enhancement of amateur digital photographs. Color Management converts colors from one device space to another; however, originals with bad color quality will not be improved. Since interactive correction may prove difficult and tedious, especially for amateur users, several algorithms have been proposed for color and contrast enhancement. Among these, some authors have suggested selective color/contrast correction for objects having an intrinsic color such as human skin or sky [4-6]. Fredembach et al. [7] have presented a framework for image classification based on region information, for automatic color S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 216 – 225, 2005. © Springer-Verlag Berlin Heidelberg 2005

Image Annotation for Adaptive Enhancement of Uncalibrated Color Images

217

correction of real-scene images. Some authors proposed a region based approach only for color correction on a limited set of image classes (sky, skin and vegetation). Among these, Fredembach et al. [8] have tested the performance of eigenregions in an image classification experiment, where the goal was to correctly identify semantic image classes, such as “blue sky,” “skin tone,” and “vegetation.” Taking this state of affairs as our point of departure, we propose a pixel classification tool to be used to tune image-processing algorithms in intelligent color devices, namely color balancing, noise removal, contrast enhancement and selective color correction. The tool is capable of automatically annotating digital photographs, by assigning the pixels to seven visual categories that mirror high-level human perception, and permitting the design of intuitive and effective enhancement strategies. Image feature extraction and classification may be not completely reliable when acquisition conditions and imaging devices are not known a priori, or are not carefully controlled. While in many cases the human observer will still be able to recognize the skin colors in the scene, we can only guess to what extent a classification algorithm, trained on high quality images, can perform the same task on typical unbalanced digital photos. To this end, we explore here the hypothesis that preprocessing the images with a white balance algorithm could improve the classification accuracy.

2 Data Selection and Pre-processing Our annotation tool is capable to classify image pixels into semantically meaningful categories. The seven classes considered - sky, skin, vegetation, snow, water, ground, and man-made structures - are briefly defined in Table 1. Table 1. Description of the classes

Class Ground Man-made structures Skin Sky Snow Vegetation Water

Description Sand, rocks, beaches … Buildings, houses, vehicles, … Caucasian, asian, negroid Sky, clouds and sun Snow and ice Grass, trees, woods and forests Sea, oceans, lakes, rivers …

The set of the images used for our experiments consisted of 650 amateur photographs. In most cases, imaging systems and lighting conditions are either unknown or else very difficult to control. As a result, the acquired images may have an undesirable shift in the entire color range (color cast). We have approached the open issues of recovering the color characteristics of the original scene, designing an adaptive color cast removal algorithm [9]. Traditional methods of cast removal do not discriminate between images with true cast and those with predominant colors, but are applied in the same way to all images. This may result in an unwanted distortion of

218

C. Cusano, F. Gasparini, and R. Schettini

the chromatic content with respect to the original scene. To avoid this problem, a multi-step algorithm classifies the input images as i) no-cast images; ii) evident cast images; iii) ambiguous cast images (images with feeble cast or for which whether or not the cast exists is a subjective opinion); iv) images with a predominant color that must be preserved; v) unclassifiable images. Classification makes it possible to discriminate between images requiring color correction and those in which the chromaticity must, instead, be preserved. If an evident or ambiguous cast is found, a cast remover step, which is a modified version of the white balance algorithm, is then applied. The whole analysis is performed by simple image statistics on the thumbnail image. Since the color correction is calibrated on the type of the cast, a wrong choice of the region to be whitened is less likely, and even ambiguous images can be processed without color distortion. After this color balancing step, the dataset was randomly subdivided into a training set of 350 photographs, and a test set composed of the remaining 300 images. The salient regions of the images in the training and in the test set have been manually labeled with the correct class. Our tool has not been designed to classify directly image pixels, but it works selecting and classifying several image tiles, that is, square subdivisions of the image the size of a fixed fraction of the total area of the image. The length l of the side of a tile of an image of width w and height h, is computed as:

l=

p⋅w⋅h ,

(1)

meaning that the area of a tile is p times the area of the whole image (here p=0.01). We randomly selected two sets of tiles from the training set and the test set, and used respectively for the learning and for the validation of the classifier. For both the training and the test set, we first drew an image, then a region of that image, and, finally, we selected at random a tile that was entirely inside that region. This process was repeated until we had 1800 tiles for each class extracted from the training set, and 1800 for each class extracted from the test set. At the end of the selection process we had two sets of tiles, consisting of 1800 x 7 = 12600 tiles each.

3 Tiles Description We compute a description of each tile by extracting a set of low-level features that can be automatically computed from the tile simply on the basis of its color and luminance values. We have used histograms because they are very simple to compute and have given good results in practical applications, where feature extraction must be as simple and rapid as possible [10]. Two kinds of histograms are used: color and edge-gradient histogram. The first allow us to describe the pictorial content of the tile in terms of its color distribution and the second gives information about the overall strength of the regions edge in the tile by computing edge gradient statistics. To combine the histograms in a single descriptor, we have used what is called a joint histogram [11]. This is a multidimensional histogram which can incorporate different information such as color distribution, edge and texture statistics and any other kind

Image Annotation for Adaptive Enhancement of Uncalibrated Color Images

219

of local pixel feature. Every entry in a joint histogram contains the fraction of pixels in the image that are described by a particular combination of feature values. We used a simple color histogram based on the quantization of the HSV color space in eleven bins [12]. Color histograms, due to their properties of efficiency and invariance in rotation and translation, are widely used for content-based image indexing and retrieval. 3.1 Edge-Gradient Histogram Two edge-gradient histograms are computed: vertical edge-gradient histogram and horizontal edge-gradient histogram. The horizontal and vertical components of the gradient are computed by the application of Sobel’s filters to the luminance image. For each component, the absolute value is taken and then quantized in four bins on the basis of comparison with three thresholds. The thresholds have been selected taking the 0.25, 0.50, and 0.75 quantiles of the distribution of the absolute value of gradient components, estimated by a random selection of over two million pixels in the images of the training set. The joint histogram combines information regarding color histogram and the two edge-gradient histograms, and thus is a three-dimensional histogram with 11 x 4 x 4 = 176 total number of bins. Although the dimension of this joint histogram is quite small (compared with the typical joint histogram of thousands of bins), we think that the information conveyed can describe the pictorial content of each tile for applications of image annotation.

4 Support Vector Machines Here, we briefly introduce the Support Vector Machine (SVM) framework for data classification. SVMs are binary classifiers trained according to the statistical learning theory, under the assumption that the probability distribution of the data points and their classes is not known [13-15]. Recently, this methodology has been successfully applied to very challenging and high dimensional tasks such as face detection [16], and 3-D object recognition [17]. Briefly, a SVM is the separating hyperplane whose distance to the closest point of the training set is maximal. Such a distance is called margin, and the points closest to the hyperplane are called support vectors. The requirement of maximal margin ensures a low complexity of the classifier, and thus, according to the statistical learning theory, a low generalization error. Obviously, often the points in the training set are not linearly separable. When this occurs, a non-linear transformation Φ(⋅) can be applied to map the input space ℜd into a high (possibly infinite) dimensional space H which is called feature space. The separating hyperplane with maximal margin is then found in the feature space. Since the only operation needed in the feature space is the inner product, it is possible to work directly in the input space provided a kernel function K(⋅,⋅) [18], which computes the inner product between the projections in the feature space of two points (2) K : ℜ d × ℜ d → ℜ, K (x1 , x 2 ) = Φ(x1 ) ⋅ Φ(x 2 ) .

220

C. Cusano, F. Gasparini, and R. Schettini

Several suitable kernel functions are known, the most widely used are the polynomial kernels

K (x1 , x 2 ) = (x1 ⋅ x 2 + 1) k , (3) with k natural parameter, and the Gaussian kernel § x −x 2 · 2 ¸, (4) K ( x1 , x 2 ) = exp¨ − 1 ¨ ¸ σ © ¹ with σ real parameter. The label f(x) assigned by the SVM to a new point x is determined by the separating hyperplane in the feature space § f (x) = sgn ¨ b + ¨ © § = sgn ¨ b + ¨ ©

N

·

¦α y (Φ(x )⋅ Φ(x ))¸¸ , i i

i

(5)

¹

i =1 N

· α i yi K (x, x i )¸ ¸ i =1 ¹

¦

where N is the size of the training set, xi ∈ℜd are the points in the training set and yi ∈{-1,+1} their classes. The real bias b and the positive coefficients αi can be found solving a convex quadratic optimization problem with linear constraints. Since the only points xi for which the corresponding coefficient αi can be nonzero are the support vectors, the solution is expected to be sparse. If training set is not linearly separable even in the feature space, a penalization for the misclassified points can be introduced. This strategy can be achieved simply bounding the coefficients αi to some value C. 4.1 Multi-class SVM Although SVMs are mainly designed for the discrimination of two classes, they can be adapted to multi-class problems. A multi-class SVM classifier can be obtained by training several classifiers and combining their results. The adopted strategy for combining SVMs is based on the “one per class” method [19, 20]. It consists in training one classifier for each class to discriminate between that class and the other classes. Each classifier defines a discrimination function g(k) that should assume positive values when the points belong to the class k and negative values otherwise. These values are then compared and the output of the combined classifier is the index k for which the value of the discriminating function g(k) is the largest. The most commonly used discrimination function is the signed distance between the case to classify and the hyperplane, obtained discarding the sign function from equation (5), and introducing a suitable normalization N

b (k ) + g

(k )

( x) =

¦α

(k ) i

yi K (x, x i )

i =1

N

.

N

¦¦α i =1 j =1

αj

(k ) i

(k )

(

yi y j K x i , x j

)

(6)

Image Annotation for Adaptive Enhancement of Uncalibrated Color Images

The label c, assigned to a point x, by the multi-class SVM is c(x) = arg max g ( k ) (x) ,

221

(7)

k∈{1,...,K }

where K is the number of classes.

5 Experimental Results For classification, we used a multi-class SVM, constructed according to “one per class” strategy. The binary classifiers are non-linear SVMs with a Gaussian kernel. In order to train each of them, we used the 1800 tiles of one class, and a random selection of 1800 tiles of the other classes; all the tiles used in the learning phase are taken from the images of the training set. Each SVM was thus trained to discriminate between one class and the others. To evaluate the performance of the classification engine, we used all the 12600 tiles extracted from the test set. For each tile in the training and in the test set a joint histogram is computed combining information related to color and gradient statistics, as described in Section 3. The results obtained on the test set by the multi-class SVM are reported in Table 2. Table 2. Confusion matrix of a non-linear multi-class SVM trained with the joint-histogram features vector. The penalization coefficient C is set to 25, and the kernel parameter σ is set to 0.1. The values are computed only on the test set.

True class

Predicted class

Man-made structures Ground Skin Sky Snow Vegetation Water

Man-made structures 0.814 0.035 0.016 0.010 0.017 0.035 0.008

Ground

Skin

Sky

Snow

Vegetation

Water

0.063

0.022

0.012

0.026

0.046

0.017

0.841 0.025 0.002 0.011 0.056 0.018

0.033 0.928 0.000 0.011 0.014 0.009

0.009 0.011 0.899 0.074 0.007 0.042

0.014 0.006 0.036 0.832 0.005 0.040

0.064 0.006 0.002 0.010 0.869 0.009

0.004 0.008 0.051 0.045 0.014 0.874

Table 3. Confusion matrix of a non-linear multi-class SVM trained with the joint-histogram features vector, and applying color balancing pre-processing. The penalization coefficient C is set to 25, and the kernel parameter σ is set to 0.1. The values are only computed on the test set.

True class

Predicted class

Man-made structures Ground Skin Sky Snow Vegetation Water

Man-made structures 0.83 0.030 0.001 0.003 0.014 0.039 0.009

Ground

Skin

Sky

Snow

Vegetation

Water

0.055

0.017

0.011

0.024

0.047

0.016

0.857 0.006 0.002 0.008 0.055 0.019

0.035 0.989 0.000 0.001 0.013 0.008

0.008 0.000 0.920 0.058 0.005 0.040

0.011 0.002 0.032 0.890 0.007 0.036

0.056 0.002 0.000 0.007 0.870 0.008

0.003 0.000 0.043 0.022 0.011 0.880

222

C. Cusano, F. Gasparini, and R. Schettini

The overall accuracy is about of 86%. The best recognized class was the skin class (more than 92%), the worst results are obtained classifying the Man-made structures class (about 81%). Typical errors involve classes with overlapped color distributions, ground tiles in particular have often been misclassified as vegetation tiles (6,4% of cases) and vice-versa (5,6%), snow tiles have been misclassified as sky tiles (7,4%), and man-made structures tiles as ground tiles (6,3%). Table 3 reports confusion matrix using the same classification strategy on the images of the dataset previously preprocessed by color balancing. Note that with or without color correction the results are worse than those obtained in a previous experimentation where all the images considered were free of color cast [21]. We also believe that a richest description of the tiles would improve the performances of the classifiers significantly. This will be the main topic of our future research in the area. After the satisfactory training of a classifier for image tiles, we designed a strategy for annotating the pixels of whole images. In order to label each pixel of the image as

Sky Man-made structures Other classes Unknown

(a)

(b) Fig. 1. Original (a) and annotated image (b)

Sky Man-made structures Unknown

(a)

(b) Fig. 2. Original (a) and annotated image (b)

Image Annotation for Adaptive Enhancement of Uncalibrated Color Images

223

belonging to one of the classes, we needed a way to select the tiles and then combine multiple classification results. In our approach, the tiles are sampled at fixed intervals. Since several tiles overlap, every pixel of the image is found in a given number of tiles. Each tile is independently classified, and the pixel’s final label is decided by majority vote. The size of the tiles is determined by applying Equation (1). Frequently an area of the image cannot be labeled with one of the seven classes selected, and in this case different classes are often assigned to overlapping tiles. To correct this kind of error and to achieve in general a more reliable annotation strategy, we introduced a rejection option: when the fraction of concordant votes related to overlapping tiles lies below a certain threshold, the pixels inside these tiles are labeled as belonging to an unknown class. In practice, the rejection option selects the pixels that cannot be assigned to any class with sufficient confidence. Our method of annotation has performed with success on the 300 images in the test set. The tiles are sampled at one third of their size in both the x and y dimensions. As a result, the same pixels are found in 9 tiles. Pixels on the image borders are classified using only the available tiles. Figures 1-4 show examples of annotated images (right), with the identified classes visually represented, compared with the corresponding originals (left). Note the rejected pixels, labeled as unknown. Figures 5 and 6 show the improvement in the annotation performance after a color correction pre-processing of the original images. Although the accuracy of the tool is quite satisfactory, we plan to further refine the whole strategy. For instance, the rejection option can be improved introducing a rejection class directly in the SVMs. Furthermore, we plan to introduce new application-specific classes. We are also considering the application of our annotation tool in content based image retrieval systems.

Snow Man-made structures Other classes Unknown

(a)

(b) Fig. 3. Original (a) and annotated image (b)

Snow Man-made structures Vegetation Other classes Unknown

(a)

(b) Fig. 4. Original (a) and annotated image (b)

224

C. Cusano, F. Gasparini, and R. Schettini

Sky

(a)

(b)

(c)

(d)

Snow

Water

Man-made structures

Unknown

Fig. 5. Example of annotation in the presence of a color cast before (a) and after (b) color correction. In the first case the blue cast confuses the classification process which erroneously detects regions of water (c). The enhanced image has been correctly annotated (d).

(a) Sky

(b) Skin

Man-made structures

(c)

(d) Other classes

Unknow

Fig. 6. Example of annotation in the presence of a color cast before (a) and after (b) color correction. Due to the red cast almost all pixels have been classified as skin (c). Significantly better results have been obtained on the corrected image (d).

Image Annotation for Adaptive Enhancement of Uncalibrated Color Images

225

References 1. A.W.M. Smeulders, M. Worring, S. Santini, A. Gupta and R. Jain: Content-based image retrieval at the end of the early years. IEEE Trans. on PAMI vol.22 (2000). 2. J.Fan , Y. Gao, H. Luo and G. Xu: Automatic image annotation by using concept-sensitive salient objects for image content representation. Proceedings of the 27th annual international conference on Research and development in information retrieval, July 2529, Sheffield, United Kingdom (2004). 3. J. Jeon, V. Lavrenko and R. Manmatha: Retrieval using Cross-Media Relevance Models. Proceedings of the 26th Intl ACM SIGIR Conf. (2003) 119–126. 4. H. Saarelma, P. Oittinen: Automatic Picture Reproduction. Graphics Art in Finland Vol. 22(1) (1993) 3-11. 5. K. Kanamori, H. Kotera: A Method for Selective Color Control in Perceptual Color Space. Journal of Imaging Technologies Vol. 35(5) (1991) 307-316. 6. L. MacDonald: Framework for an image sharpness management system. IS&T/SID 7th Color Imaging Conference, Scottsdale (1999) 75-79. 7. C. Fredembach, M. Schröder, and S. Süsstrunk, Region-based image classification for automatic color correction. Proc. IS&T/SID 11th Color Imaging Conference, pp. 59-65, 2003. 8. C. Fredembach, M. Schröder, and S. Süsstrunk, Eigenregions for image classification. Accepted for publication in IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004 9. F. Gasparini and R. Schettini: Color Balancing of Digital Photos Using Simple Image Statistics. Pattern Recognition Vol. 37 (2004) 1201-1217. 10. M. Stricker and M. Swain: The Capacity of Color Histogram Indexing. Computer Vision and Pattern Recognition (1994) 704-708. 11. G. Pass and R. Zabih: Comparing Images Using Joint Histograms. Multimedia Systems Vol. 7(3) (1999) 234-240. 12. I. J. Cox, M. L. Miller, S. M. Omohundro, and P.N. Yianilos: Target Testing and the PicHunter Bayesian Multimedia Retrieval System. Advances in Digital Libraries (1996) 66-75. 13. V. Vapnik: The Nature of Statistical Learning Theory. Springer (1995). 14. T. Hastie, R. Tibshirani, J. Friedman: The Elements of Statistical Learning. Springer (2001). 15. C. Cortes and V. Vapnik: Support-Vector Networks. Machine Learning Vol. 20(3) (1995) 273-297. 16. E. Osuna, R. Freund, and F. Girosi: Training support vector machines. An application to face detection. Proceedings of CVPR'97 (1997). 17. V. Blanz, B. Schölkopf, H. Bülthoff, C. Burges, V. Vapnik, and T. Vetter: Comparison of view-based object recognition algorithms using realistic 3D models. Artificial Neural Networks ICANN'96. (1996) 251-256. 18. K. R. Muller, S. Mika, G. Ratsch, K. Tsuda, and B. Scholkopf: An introduction to kernel-based learning algorithms. IEEE Transactions on Neural Networks Vol. 12(2) (2001) 181-201. 19. J. Weston and C. Watkins: Support vector machines for multiclass pattern recognition. Proc. Seventh European Symposium On Artificial Neural Networks (1999). 20. K. Goh, E. Chang, and K. Cheng: Support vector machine pairwise classifiers with error reduction for image classification. Proc. ACM workshops on Multimedia: multimedia information retrieval (2001), 32-37. 21. C. Cusano, G. Ciocca and R. Schettini: Image annotation using SVM. Proc Internet imaging V, Vol. SPIE 5304 (2004), 330-338, 2004.

Automatic Redeye Removal for Smart Enhancement of Photos of Unknown Origin Francesca Gasparini and Raimondo Schettini Dipartimento di Informatica Sistemistica e Comunicazione, Università degli studi di Milano-Bicocca, Via Bicocca degli Arcimboldi 8, 20126 Milano, Italy {gasparini, schettini}@disco.unimib.it http://www.ivl.disco.unimib.it/

Abstract. The paper describes a modular procedure for automatic correction of redeye artifact in images of unknown origin, maintaining the natural appearance of the eye. First, a smart color balancing procedure is applied. This phase not only facilitates the subsequent steps of processing, but also improves the overall appearance of the output image. Combining the results of a color-based face detector and of a face detector based on a multi-resolution neural network the most likely facial regions are identified. Redeye is searched for only within these regions, seeking areas with high “redness” satisfying some geometric constraints. A novel redeye removal algorithm is then applied automatically to the red eyes identified, and opportunely smoothed to avoid unnatural transitions between the corrected and original parts. Experimental results on a set of over 450 images are reported.

1 Introduction The redeye effect is a well known problem in photography. It is often seen in amateur shots taken with a built-in flash, but the problem is also well known to professional photographers. Redeye is the red reflection of the blood vessels in the retina caused when a strong and sudden light strikes the eye. Fixing redeye artifacts digitally became an important skill with the advent of digital technologies, which permit to acquire digitalized images either directly with a digital camera or converting traditional photos by scanners. Also the widespread use of small devices with built-in flash, including cell phones and handheld computers, produces a large amount of digital photographs potentially redeye affected. Currently, many image processing software applications in the market offer redeye removal solutions. Most of them are semi-automatic or manual solutions. The user has to either click on the redeye or draw a box containing it before the removal algorithm can find the redeye pixels and correct them [1-4]. Also several companies, such as Hewlett Packard, Kodak, Fuji, Agfa, Nikon, and others developed ad hoc algorithms and tools [5-13]. A typical problem of most of these algorithms is poor pupil segmentation that leads to unnatural redeye correction. Even with user interaction, these algorithms S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 226 – 233, 2005. © Springer-Verlag Berlin Heidelberg 2005

Automatic Redeye Removal for Smart Enhancement of Photos

227

sometimes correct redeye pixels too aggressively, darkening eyelid areas, or too conservatively, leaving many redeye pixels uncorrected. In this context, we have designed a procedure for automatic detection and correction of the redeye effect for images of unknown origin, i.e. images acquired by unknown imaging systems under unknown imaging conditions, such as images downloaded from the web or received by friends with e-mails or cell phones. The proposed method is modular so that each step can be removed and substituted with a more efficient one in future work, without changing the main structure. Also, it can be improved by simply inserting new modules. In the first module, a smart color balancing algorithm is applied to correct the color photo. This phase not only facilitates the subsequent steps of processing, but also improves the overall appearance of the output image. In the following, as several redeye removal algorithms, the method proposed looks for redeye within the most likely face regions. The localization of these candidate regions is here obtained combining, through a scoring process, the results of a colorbased face detector and a face detector based on a multi-resolution neural network, working only on the intensity channel. In the final phase, red eyes are automatically corrected exploiting a novel algorithm that has been designed to remove the unwanted effects maintaining the natural appearance of the processed eyes.

2 Image Color Correction As a first module of our system, a smart color balancing algorithm is applied to perform image color correction. We have designed a reliable and rapid method to classify and remove color cast (i.e. a superimposed dominant color) in a digital image without any a priori knowledge of its semantic content, [14]. First a cast detector, using simple image statistics, classifies the input images as presenting no cast, evident cast, ambiguous cast, a predominant color that must be preserved (such as in underwater images or single color close-ups) or as unclassifiable. A cast remover, a modified version of the white balance algorithm, is then applied in cases of evident or ambiguous cast. Since the color correction is calibrated on the type of the cast, even ambiguous images can be processed without color distortion.

3 Combined Face Detectors Face detection in a single image is a challenging task because the overall appearance of faces ranges widely in scale, location, orientation and pose, as well as in facial expressions and lighting conditions [15-16]. Our objective therefore was not to determine whether or not there are any faces, but instead to generate a score map, reflecting the confidence of having a facial region that might contain red eyes. This score map is obtained here combining the scores from a color-based face detector and a neural network face detector, working only on the intensity channel.

228

F. Gasparini and R. Schettini

3.1 The Color-Based Face Detector A skin detector, structured as follows, is applied on the color balanced images: ƒ color corrected images (in sRGB) [17-18] are mapped in the CIELAB color space [19], and the segmentation is performed on skin-like regions analyzing the following variables:

(

)

(

2

* * hue = tan −1 b* / a* ; K = a + b *

L

2

)

12

=

C* L*

where C*/L*, the ratio between the chroma radius and the lightness, is correlated to color saturation. ƒ a multi modal analysis of the hue histogram is performed, taking into account the cyclic nature of hue, and a first order skin-like region mask is obtained from all the peaks with maxima (MAXhue) in the interval: 0 ≤ MAX hue ≤ 90 and 340 ≤ MAX hue ≤ 360

ƒ a second order refinement of the mask is obtained with a multi modal histogram analysis of the ratio K of these first order skin-like regions. Among all the histogram peaks, those satisfying the following relations (in terms of mean values over the peak: Kmean, L*mean) correspond to skin regions:

Fig. 1. Top left, input image, top right the corresponding skin mask. Bottom left, score of the color-based face detector, bottom right, score of the neural network face detector.

Automatic Redeye Removal for Smart Enhancement of Photos

K mean < 0.75 and and

229

L*mean ≥ 40

L mean ≥ −215 × K mean + 95 *

ƒ The final skin-like region mask, used to segment the original image is obtained with morphological operators, to fill holes and gaps that could be eyes, or noses, or lips, and to remove small areas that are unlikely to be faces. In Fig. 1, top row, an input image and its corresponding skin mask are shown. The candidate faces are finally obtained as follows. Instead of a classification of the skin regions as a binary output (face, no-face), a score map, scoreCB, is generated taking into account: 1) discontinuities in the intensity channel that may correspond to eyes, nose and lips (scoreD), 2) the shape of the region that should fit an oval (scoreS), and 3) the ratio between the area of the region and the area of its filled version (scoreA). These scores are all normalized to one. The final score is defined as scoreCB = (scoreD + scoreS + score A ) 3

(1)

An example of this score map is depicted in Figure 1, bottom left. 3.2 The Neural Network Face Detector We have trained an autoassociative neural network [15] to output a score map reflecting the confidence of the presence of faces in the input image. It is a three layer linear network, where each pattern of the training set is presented to both the input and the output layers, and the whole network has been trained by a backpropagation sum square error criterion, on a training set of more than 150 images, considering not only face images (frontal faces), but non-face images as well. The network processes only the intensity image, so that the results are color independent. To locate faces of different sizes, the input image is repeatedly scaled down by a factor of 15%, generating a pyramid of subsampled images. A window of 20x21 pixels is applied at every point of each scaled image. This 20x21 portion of the image, properly equalized, forms the input of the network. The output is obtained with a feedforward function, and the root mean square error ε, between output and input is calculated. The performance of the network is evaluated analysing the True Positive, versus the False Positive varying the error ε. The score map of the input image, scoreNN, is obtained collecting the confidence of being a facial region of each single 20x21 window in the pyramid with root mean square error ε, evaluated as 1-FP(ε). 3.3 The Final Score Map The final score map is obtained combining these scores and normalizing the result so that it ranges between 0 and 1:

score = (score NN + score CB ) 2

(2)

The most likely facial regions correspond to the regions with the highest values of the score. In this work we consider as face all the regions with score greater than 0.6. In

230

F. Gasparini and R. Schettini

Figure 2, on the left, the final score map of the input image of Figure 1 is shown; while on the right the most likely facial regions are reported. Even if both face detectors show high values of false positives to guarantee a good face detection rate, (especially when applied to search faces different in scale, location, orientation and pose, as well as in facial expressions and lighting conditions), the union of the two methods through the definition of a score process permits to improve the performance of the final system significantly, as shown in Table 1. Table 1. Comparison among the performances of the face detectors

Face detector Color-based (CB) Neural Network (NN) CB+NN

TP 90% 95% 96%

FP 25% 27% 2%

Fig. 2. Left, final score map of the input image of Figure 1; right, the most likely facial regions for score> = 0.6

4 Redeye Detection Within the most likely face regions the algorithm looks for red eyes, applying a first step based on color analysis, and a second based on geometric constraints. 1.

Color analysis: the algorithm looks for the regions with high value of the “redness”: redness=(4×R-(G+B)-min(G,B)-max(G,B))/R

(3)

the color image is thus converted into a monochrome (redness), in which a redeye is highlighted as a bright spot. 2.

Geometric analysis: to limit the number of false hits, the algorithm exploits the following geometric constraints: 9 Ratio between the redeye area and the face bounding box area: P > 1.5% 9 Ratio between minimum and maximum dimension of the eye: F > 0.4 9 Roundness: ratio between the redeye area and the area of the ellipse that has the same second-moment as the region.: O > 0.7

Automatic Redeye Removal for Smart Enhancement of Photos

231

5 Redeye Removal The last step of the algorithm is the color correction of the redeye artifact. If a pixel has been detected as belonging to a redeye, it is replaced with a substantially monochrome pixel, applying the following equation to all the three channels R,G and B:

Rnew = Rold × (1 − Masksmooth ) + Masksmooth × Rmch

(4)

The coordinates of the monochrome pixel are Rmch, Gmch and Bmch, evaluated considering the intensity equal to the mean of (G,B) and the color correction is weighted with the smoothing mask (Masksmooth) of the incriminated area, to avoid unnatural transitions between corrected and original parts of the eyes. The pixels involved are modified to preserve the bright specular reflection of the eyes. In Figure 3, the original image the smoothed area corresponding to the red eyes and the correction of the artifacts with equation 4 are reported.

Fig. 3. Left, original image, middle the smoothing mask of the redeye areas; and right, the corrected image

6 Experimental Results and Conclusions We have tested our algorithm on a data set, composed of 450 color images of different sizes (from 120 × 160 to 2272 × 1704 pixels), resolutions and quality. These images were downloaded from personal web-pages, or acquired using various digital cameras and scanners. Different pre-processing was applied to these images and was in most cases unknown. In order to evaluate the proposed method quantitatively, we divided images with redeye artifacts into two groups: 1) portraits and high resolution images, about 250 images and 2) low resolution images, or images with several people, about 200 images. The results are reported in Table 2 in terms of true positives, and false positives per image, (non red eyes erroneously corrected). The step of redeye detection within the most probable facial regions has a very high percentage of success in the case of portraits, and of high-resolution images, where the geometric features of the eyes are easily detected, while with images of several people or poor quality images, it is less successful. In both cases the number of false hits is very low, making the proposed method quite useful for automatic systems.

232

F. Gasparini and R. Schettini Table 2. Performance of the method

Image class

TP

FP

Portrait or high quality

95%

0.1%

75%

0.35%

Groups or low quality

The removal algorithm has been heuristically evaluated in terms of number of automatically corrected eyes that do not require further interactive editing to achieve “optimal results”. This number, resulting in 96%, has been evaluated by a panel of five observers having a good image processing skill, thus only the 4% of the corrected eyes could be better corrected interactively using Photoshop. In conclusion, the proposed tool has several features in terms of effectiveness, friendliness and robustness that make it an ideal candidate to be included within software for the management and enhancement of digital photo albums by non expert, amateur photographers.

References 1. C. M.Dobbs, R. Goodwin: Localized image recoloring using ellipsoid boundary function. US Patent 5,130,789, July (1992). 2. P. Benati, R. Gray, P. Cosgrove: Automated detection and correction of eye color defects due to flash illumination. US patent 5,748,764 5 May (1998). 3. A. Patti, K. Kostantinides, D. Tretter, Q. Lin: Automatic Digtal Redeye Reduction. Proceedings of the IEEE International Conference on Image Processing (ICIP-1998), Chicago, IL, vol.3 (1998) 55-59. 4. .J. Y. Hardeberg: Red eye removal using digital color image processing. Proceedings of the Image Processing, Image Quality, Image Capture System Conference, Montreal, Canada (2001) 283-287. 5. J. Y. Hardeberg: Digital red eye removal. Journal of Imaging Science and Technology, Vol. 46 (4) (2002) 375-381. 6. A. Patti, K. Kostantinides, D. Tretter and Q. Lin: Apparatus and a method for reducing red-eye in a digital image. U.S patent 6,016,354, Jan. (2000). 7. J. Schildkraut, R. Gray, J. Luo: Computer program product for redeye detection. US. Patent 6,292,574 Sept. (2001). 8. J. Wang, H. Zhang: Apparatus and a method for automatically detecting and reducing redeye in a digital image. U.S patent 6,278,491, August (2001). 9. M. Gaubatz, R. Ulichney, ’Automatic red-eye detection and correction’ Proceedings of the IEEE International Conference on Image Processing (ICIP-2002), Rochester, NY, vol.1 (2002) 804-807. 10. H. Luo, J. Yen, D. Tretter: An Efficient Automatic Redeye Detection and Correction Algorithm. Proceedings of the 17th International Conference on Pattern Recognition (ICPR-2004), Cambridge, UK, Vol. 2 (2004) 883-886. 11. K. Czubin, B.Smolka, M. Szczepanski, J. Y. Hardeberg, K.N. Plataniotis: On the Redeye Effect Removal Algorithm. Proc. CGIV 2002, Poitiers, France (2002) 292-297.

Automatic Redeye Removal for Smart Enhancement of Photos

233

12. B.Smolka, K. Czubin, J. Y. Hardeberg, K.N. Plataniotis, M. Szczepanski, K. Wojciechowski: Towards Automatic Redeye Effect Removal. Pattern Recognition Letters Vol. 24 (11) (2003) 1767-1785 13. J. Y. Hardeberg: Red-eye removal using color image processing. US Patent 6,728,401, Apr. (2004). 14. F. Gasparini and R. Schettini: Color Balancing of Digital Photos Using Simple Image Statistics. Pattern Recognition Vol. 37, (2004) 1201-1217. 15. M. H. Yang, D.J. Kriegman, N. Ahuja: Detecting Faces in Images: A Survey. IEEE Trans. on Pattern Analysis and Machine Intelligence Vol. 24(1) (2002). 16. H. Rowley, S. Baluja, T. Kanade: Neural Network-Based Face Detection. IEEE Trans. on Pattern Analysis and Machine Intelligence Vol. 20 (1) (1998). 17. http://www.srgb.com. 18. International Color Consortium (ICC). http://www.color.org. 19. M. D. Fairchild, Color Appearance Models, Addison Wesley, (1997).

Analysis of Multiresolution Representations for Compression and Local Description of Images Fran¸cois Tonnin, Patrick Gros, and Christine Guillemot Irisa, Campus Universitaire de Beaulieu, 35042 RENNES Cedex, France

Abstract. Low level features of images are often extracted from their representations in a Gaussian scale space. These representations satisfy the desired properties of covariance under a set of transformations (translations, rotations, scale changes) as well as of causality. However, the corresponding image representations, due to their redundant and non sparse nature, are not well suited for compression purposes. This paper aims at characterizing a set of multiresolution representations from the joint perspective of feature point and descriptor extraction and of compression. This analysis leads to the design of a feature point detector and of a local descriptor in signal representations given by oversampled steerable transforms. It is shown that the steerable transforms due to their properties of covariance under translations, and rotations as well as of angular selectivity provide signal representations well suited to address the signal description problem. At the same time, techniques such as iterative projection algorithms (POCS - projection on convex sets) are used to reduce the coding cost induced by the corresponding oversampled signal representation. The robustness and the discriminative power of extracted features are rated in terms of the entropy of the quantized representation. These results show the tradeoff that can be found between compression and description.

1

Introduction

During the last two decades, image representations obtained with various transforms, e.g., Laplacian pyramid transforms, separable wavelet transforms, curvelets, and bandelets have been considered for compression and denoising purposes. These critically sampled or low redundant transforms do not allow the extraction of low level features such as geometric features (points, edges, ridges, or blobs) or local descriptors. Yet, some point detectors [8,3] have been designed in the dyadic wavelet, but their robustness is much lower than the ones designed in the Gaussian scalespace. Many visual tasks such as segmentation, motion detection, object tracking and recognition, content based image retrieval, require prior extraction of these low level features. The Gaussian scale space is almost the unique image reresentation used for this detection [2,4,7,10,9]. Management of large databases are therefore uneasy, as the extraction of new features requires to de-compress the whole database and then to convert it in the Gaussian scale S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 234–246, 2005. c Springer-Verlag Berlin Heidelberg 2005 

Analysis of Multiresolution Representations for Compression

235

space. It is thus desirable to find representations suitable for both problems. However, their design criteria can be seen as antagonist. Feature extraction requires the image representation to be covariant under a set of admissible transformations, which ideally is the set of persective transformations. Reducing this set of transformations to the group of translations, rotations and rescalings, and adding the constraint of causality, the image representation is uniquely characterized by the Gaussian scale space [13]. In a compression perspective, one is concerned by reconstructing the image from a minimal amount of information, provided by quantized transform coefficients. Thus, the image representation should be sparse, critically sampled (or minimally redundant), and transform coefficients should be maximally independent. Sparsity is achieved by adapting the projection functions onto the transformed space to the statistics of natural images. Independency between transform coefficients can not be achieved, then one prefer orthogonal basis because they decorrelate these coefficients. Yet, critically sampled representations suffer from shift variance and thus are not adapted for feature extraction. Some multiresolution representations have been designed specifically to prevent this drawback, such as the complex wavelet transform [5] and the steerable transform [12]. The price to pay is a certain amount of redundancy in the representation, but, in some cases, as shown in [1,14], this redundancy does not penalize the compression. This paper aims first at characterizing a set of multiresolution representations from the joint perspective of feature point and descriptor extraction and of compression. This analysis leads to the design of a feature point detector and of a local descriptor in signal representations given by oversampled steerable transforms. Section 2 defines the protocols for the evaluation of feature points and local descriptors, section 3 characterizes the extraction of feature points in the Gaussian scale space and in wavelet domains, and section 4 the extraction of local descriptors in the same domains. In the section 5 are evaluated the quality of the features extracted in the compressed domain in function of the entropy of the representation. It is shown that a tradeoff between compression and description can be found. Some concluding remarks and future work are discussed in the last section.

2

Evaluation of Low Level Features

Many visual applications require prior detection of points that are robust to the above-mentioned set of admissible image transformations T . These points are called feature points. Given a natural image I0 ∈ l2 (ZZ 2 ) (the space of square summable sequences of ZZ 2 ), they are extracted from an image representation R0 of I0 . When requiring the image representation to be covariant under translations and dilations, it is necessarily provided by the convolution of the image with a set of filters parameterized in scale: R0 (x, y; s) = I0 (x, y)  hs (x, y) .

(1)

236

F. Tonnin, P. Gros, and C. Guillemot

Pyramidal or decimated representations do not belong to this set of representations, but will be considered. When a feature point is extracted at a given scale, the analysis of energy at finer scales allows to localize the point onto the sampling grid of the original image. The set of n0 feature points extracted from R0 is denoted P 0 = {(x0i , yi0 ; s0i ), 1 ≤ i ≤ n0 } , (2) where (x0i , yi0 ) are the coordinates of the ith extracted point and s0i its scale. In order to evaluate the robustness of these points to a given admissible transformation T ∈ T , we construct the synthetic image I1 = T ◦ I0 , and the image representation R1 of I1 . The set of n1 feature points extracted from R1 up to the transformation T is denoted   P 1 = { T −1 (x1i , yi1 ); si1→0 , 1 ≤ i ≤ n1 } , (3) where the scale si1→0 is the scale s1i divided by the factor ratio of the transformation T . The robustness of feature points to an admissible transformation T ∈ T is evaluated through the notion of repeatability introduced in [11], and defined by the percentage of matching points r=

max(|C 01 |, |C 10 |) , min(n0 , n1 )

(4)

where |C 01 | denotes the cardinal of C 01 , and C 01 (resp. C 10 ) is the subset of points in P 1 (resp. in P 0 ) that match a point in P 0 (resp. in P 1 ). In [11], C 01 is defined relatively to a given precision ε, as   C 01 = {(x1 , y 1 ; s1 ) ∈ P 1 : ∃(x0 , y 0 ; s0 ) ∈ P 0 | d (x0 , y 0 ), (x1 , y 1 ) ≤ ε} (5) where d is the L2 distance. Note that this notion of repeatability referred to as space repeatability increase with the number of extracted points and decrease with the size of the image. The expectation of the space repeatability of n points 2 randomly extracted from an image composed of N pixels is r(ε) = nε N . In the sequel, repeatability is evaluated by extracting 400 feature points from images of size 480 × 320. One application of feature points detection is the local description of images. The goal of local description is to extract in the neighborhood of every feature point a feature which ideally uniquely characterizes the set of neighborhoods covariant under perspective transforms. In practice, given a feature point (xki , yik ; ski ), one looks for a distinctive feature mki , (k ∈ {0, 1}) invariant to the admissible transformations. We will evaluate both the invariance and the discriminative power of the local descriptors by the repeatability achieved by matching feature points according to their closest descriptor. The subset of points in P 1 that match a point in P 0 relatively to a given precision ε is defined as   {(x1 , y 1 ; s1 ) ∈ P 1 : d (x0j , yj0 ), (x1 , y 1 ) ≤ ε, j = arg min ||m1 − m0p ||} . (6) 1≤p≤n0

The ratio between this notion of repeatability and the space repeatability is referred to as descriptor repeatability.

Analysis of Multiresolution Representations for Compression

3

237

Feature Points Detection

Feature points are defined as salient points, i.e. points where a transient phenomena occur in the image. This transient phenomenum can be modeled geometrically by providing a shape of saliency (corners, blobs, T-junctions), or energetically by defining a saliency measure in scale space. 3.1

Detection in the Gaussian Scale Space

The Gaussian scale space L(x, y; s) of I(x, y) is the image representation as de2 +y 2 1 exp − x 4s . fined by Eqn. 1 where the expansion kernel is given by hs (x, y) = 4πs Different scale normalized differential operators have been proposed in [7]. The operator used for blob detection is the scale normalized Laplacian E(x, y; s) = s∇2 L(x, y; s) .

(7)

The normalization in scale guarantees that scale space local maxima are robust to scale changes. The scale s at which the local maximum is reached is the intrinsic scale of the point (x, y). Feature points are defined as strongest local extrema of the saliency measure E(x, y; s). These local extrema are detected in a discrete set Θ of scale pace. Lowe [9] proposed: j

j

Θ = {(m2 nv −1 , n2 nv −1 ; 2j/nv s0 ), (m, n) ∈ ZZ 2 , 0 ≤ j ≤ nv no + 1}

(8)

as a good tradeoff between the complexity of extraction and the repeatability of feature points. The number of voices per octave nv , the number of octaves no , and the scale s0 of initial smoothing, are determined experimentally. Note that the image is interpolated to double size and then subsampled by a factor four at every This leads to a redundancy factor of E(x, y; s) given by  v nonew−joctave. v 4 ∼ 16n , which is much too high for compression purposes. Space nv . nj=−1 3 repeatability, as defined by Eqn. 5 and shown on the top left corner of Fig. 1, is increased by a sub-pixel refinement and by keeping only the points where the two principal curvatures are high [9]. Another definition of feature points has been proposed by Harris [4]. Considering that edges (resp. corners) are characterized by one (resp. two) high principal curvature(s), feature points can be extracted by the mean of a saliency measure defined as a function of principal curvatures. This detection can be done in the Gaussian scale space, where derivation is well posed. The partial derivatives of order i + j with respect to (x, y) of L(x, y; s) are denoted Lxi yj i+j

and defined by Lxi yj (x, y; s) = ∂ ∂xL(x,y;s) . The eigenvectors of the matrix of i ∂y j autocorrelation of first order partial derivatives

Lx 2 (x, y; s) Lx (x, y; s)Ly (x, y; s) M (x, y; s, s˜) = g(x, y; s˜)  (9) Ly 2 (x, y; s) Lx (x, y; s)Ly (x, y; s) provide the principal curvatures, where s and s˜ are respectively the integration and the derivation scales. Feature points are local maxima of the Harris measure defined by:

238

F. Tonnin, P. Gros, and C. Guillemot

E(x, y; s, s˜) = |det (M (x, y; s, s˜)) − α trace2 (M (x, y; s, s˜)) |

(10)

In order to reduce the complexity of extraction, local maxima are searched in a hyperplane defined by a proportionality relation between s and s˜. Robustness to rescaling is improved by keeping only the points that are detected at their intrinsic scale [10]. The top right corner of Fig. 1 shows the space repeatability of this Harris-Laplacian detector, which is not as high as Lowe’s repeatability. Gaussian scale-space is uniquely determines by th constraint of covariance under the group of translations, rotations and dilations, and the constraint of causality. When relaxing the latter constraint, a large set of image representations is allowed, and specifically wavelet representations, which are of particular interest to the joint problem of compression and description. 3.2

Detection in Wavelet Domains

Instead of causal kernels, one can look for kernels hs given by Eqn. 1 that are of null moment. The choice of the kernel, such as the shape, the number of null moments, will determine the type of singularities that can be detected. Considering temporarily that the image I is continuous, i.e. I ∈ L2 (IR2 ), the continuous wavelet transform is defined over IR2 × IR+ × [0, 2π] by:



1 −1 Rψ (x; s; θ) = s−2 rθ (u − x) d2 u I(u)ψ (11) s IR2 where ψ is the complex conjugate of ψ, rθ the rotation matrix of angle θ, and ψ ∈ L2 (IR2 ) a wavelet, i.e. it is a complex valued function satisfying the admissibility condition:

Cψ = (2π)

ˆ ||ψ(u)||

2 IR2

2

||u||

2

d2 u < ∞ ,

(12)

where ψˆ denotes the Fourier transform of ψ. In particular, the finitude of Cψ implies that ψ is of null moment. Thus, contrary to the Gaussian scale space, image representations provided by wavelet transforms do not satisfy the property of causality. Yet, they satisfy the properties of covariance under the group of translations, rotations and dilations, and it preserves norms, that is,

2π 2 2 ds 2 −1 |I(u)| d2 u . Cψ ||Rψ (x; s; θ)|| d x dθ = (13) 2 + s IR IR 0 2

The quantity ||Rψ (x; s; θ)|| is an energy density in scale space, which turns out to be a natural choice for the saliency measure E(x; s; θ). When ψ is chosen to be isotropic, there is no more dependency in θ in Eqn. 11. The continuous wavelet transform of discrete images I ∈ l2 (ZZ 2 ) is adapted from Eqn. 11 by replacing the integral by a discrete sum:  I(u, v)ψx,y,s,θ (u, v) , (14) Rψ (x, y; s; θ) = u,v

Analysis of Multiresolution Representations for Compression

239

  where ψx,y,s,θ (u, v) = s−2 ψ s−1 r−θ (u − x, y − v) . Some tests have shown that the choice of the mother wavelet much impacts the repeatability, and that both the isotropic and the directional versions of the mexican hat wavelet provide the best results. The sampling grid of oriented scale space used in the tests is defined by Θ = {(m, n; 2j/nv ;

kπ ); (m, n) ∈ ZZ 2 , 1 ≤ j ≤ nv no , 0 ≤ k ≤ 17} , 18

(15)

where the number n0 of octaves and the number nv of voices per octave are both equal to three. A finer discretization in scale or in orientation does not increase the repeatability. The middle left part of Fig. 1 shows the repeatability of (oriented) scale space local maxima of the saliency measure |Rψ (x, y; s; θ)|2 , for the directional mexican hat. It can be seen that non Gaussian scale space may allow robust extraction of features. Yet, the continuous wavelet transform is highly redundant, and thus is not suited for compression. The purpose of discrete wavelet transforms is to eliminate or reduce this redundancy. The dyadic wavelet transform is the most widely used wavelet transform, both because it is critically sampled, and there exists fast and simple algorithms to compute it. It is slightly different from the definition given by Eqn. 11, since it is not a directional neither an isotropic transform. The image can be reconstructed from its wavelet coefficients by: I=

nb  no  

Rψ(k) (xm , yn , sj )ψx(k) + m ,yn ,sj

j=0 k=1 m,n



Rφ (xm , yn , sno )φxm ,yn ,sno (16)

m,n

where nb = 3 and {ψ (k) }1≤k≤3 and φ are respectively three separable wavelets and a separable scaling function constructed from the tensor product beween a monodimensional wavelet and a monodimensional scaling function. The sampling grid is dyadic, i.e. defined by: Θ = {(m2j , n2j ; 2j ); (m, n) ∈ ZZ 2 , 1 ≤ j ≤ no }

(17)

The wavelet coefficients are dependent of their position relatively to the dyadic grid, making this transform highly shift variant. The space repeatability corresponding to the saliency measure: 2

E(x, y; s) = max |Rψ(k) (x, y; s)| , 1≤k≤nb

(18)

is shown by the middle left part of Fig. 1 for various admissible image transformations. The best repeatability has been reached when the mother wavelet and the mother scaling function are the bi-orthogonal “9-7” filters. The number of octaves is equal to three, and higher levels of decomposition do not allow to reach higher repeatability. For the given set of admissible transformations, repeatability at coarse precision is comparable to the one obtained by traditional extractors in the Gaussian scale space. Yet, shift variance of the transform makes the repeatability lower at fine precision. Note that the repeatability for

240

F. Tonnin, P. Gros, and C. Guillemot

translations of odd number of pixels is much lower than for translations of even number of pixels. This comes from the dependency of the transformed coefficients to their position relatively to the sampling grid. The high repeatability for rescaling of factor two also comes from the dyadic sampling. The non decimated wavelet transform is the simplest way to adapt the previous transform to be shift invariant. It is the same separable wavelet transform as the dyadic wavelet transform, except that it is defined over a cubic sampling grid: (19) Θ = {(m, n; 2j ); (m, n) ∈ ZZ 2 , 1 ≤ j ≤ no } The redundancy of this transform is then equal to 3no +1. The bottom left corner of Fig. 1 shows the space repeatability when choosing the saliency measure as defined by Eqn. 18. Space repeatability is much better than in the dyadic case, and is close to the one obtained in the Gaussian scale space. This transform is suitable for feature points detection, but its lack of orientation selectivity does not allow to design local description schemes. The shiftable multiscale transform, designed in [12], satisfies several properties which make it adapted for the joint problem of compression and description. It is not exactly shift invariant, but “power shiftable”, meaning that the energy in every band remains constant when shifting the input image. This transform also allows a fine angular analysis suitable for description. Finally, some techniques [14,1] allow to significantly reduce the coding cost due to its redundancy. This transform is built in the Fourier domain, where the kernel hs given by ˆ s (ρ, θ) = Us (ρ)V (θ). The kernel Us (ρ) is bandpass Eqn. 1 is polar separable h and power shiftable. The kernel V (θ) is steerable, i.e. there exists a set of angles {αk }k and a set of interpolating functions {fk }k so that: ∀θ ∈ [0, π], V (θ) =

N −1 

fk (θ)V (αk )

(20)

k=0

At a given location and a given scale, N coefficients are sufficient to interpolate the image representation at any angle. A pyramidal representation is constructed across scales, according to the sampling grid Θ = {(m2j , n2j ; 2j ;

kπ ); (m, n) ∈ ZZ 2 , 0 ≤ j ≤ no 0 ≤ k ≤ N − 1}. N

(21)

The redundancy of this transform is then around 4N 3 . The bottom right corner of Fig. 1 shows the space repeatability when choosing the saliency measure defined by Eqn. 18, a number N of oriented bands equal to four, and a number n0 of octaves equal to three. It can be seen that high repeatability is achieved for every admissible transformation. This transform also allows to perform a fine orientation analysis, which is particularly adapted to the design of local description schemes as described in the next section.

Analysis of Multiresolution Representations for Compression Harris−Laplace points detector

1

1

0.9

0.9

0.8

0.8

0.7

0.7

Repeatability

Repeatability

Lowe’s points detector

0.6 0.5 0.4

translation 5 pixels translation 10 pixels rescaling x0.5 rescaling x1.1 rescaling x2 rotation 5 degrees rotation 30 degrees

0.3 0.2 0.1 0 0.5

1

1.5

2

2.5

3

3.5

0.6 0.5 0.4 0.3 0.2 0.1 0 0.5

4

1

1.5

Precision in pixels

0.9

0.9

0.8

0.8

0.7

0.7

0.6 0.5 0.4

0.2

0.1

0.1

3

3.5

0 0.5

4

1

1.5

Precision in pixels

0.9

0.9

0.8

0.8

0.7

0.7

0.6 0.5 0.4

0.2

0.1

0.1

2.5

4

0.4 0.3

2

3.5

0.5

0.2

Precision in pixels

3

0.6

0.3

1.5

2.5

Extraction dans les representations "steerable" 1

Repetabilite

repeatability

Points detector in the non decimated wavelet domain

1

2

Precision in pixels

1

0 0.5

4

0.4 0.3

2.5

3.5

0.5

0.2

2

3

0.6

0.3

1.5

2.5

Points detector in the dyadic wavelet domain 1

repeatability

repeatability

Points detector in the continuous wavelet domain

1

2

Precision in pixels

1

0 0.5

241

3

3.5

4

0 0.5

1

1.5

2

2.5

3

3.5

4

Precision en pixels

Fig. 1. Space repeatability of feature points extracted in the Gaussian scale space (first row), and in wavelet domains (second and third row). The repeatability is calculated for various admissible transforms, for a fixed number of extracted points equal to 400, with images of size 480 × 320.

242

4

F. Tonnin, P. Gros, and C. Guillemot

Local Description

Considering that the image representation is covariant under the set of admissible transforms, and that feature points are perfectly repeatable, every local descriptor in this image representation is invariant to translations and dilations. In this ideal case, the problem is then to design a distinctive feature which is invariant to rotations. 4.1

Description in the Gaussian Scale Space

The most widely used local descriptors are the differential invariants introduced in [6]. They consist in linear combinations of partial derivatives of the Gaussian scale space of the image at a given extracted point. These partial derivatives convey much information about the local geometry. They are combined in order to create distinctive features invariant to rotations. In [10], these differential invariants have been adapted in order to be invariant to scale changes, and by extension to the group of isometries. The middle part of Fig. 2 shows the descriptor repeatability of these invariants, as defined by Eqn. 6. differential invariants

SIFT transposed in steerable domain 1

0.8

0.8

0.8

0.6 translation 5 pixels translation 10 pixels rescaling x0.5 rescaling x1.1 rescaling x2 rotation 5 degres rotation 30 degres

0.4

0.2

0 0.5

1

1.5

2 2.5 precision in pixels

3

3.5

descriptor repeatability

1

descriptor repeatability

descriptor repeatability

SIFT descriptor 1

0.6

0.4

0.2

4

0 0.5

0.6

0.4

0.2

1

1.5

2 2.5 precision in pixels

3

3.5

4

0 0.5

1

1.5

2 2.5 precision in pixels

3

3.5

4

Fig. 2. Descriptor repeatability of SIFT invariants, differential invariants, and our transposition of SIFT in the steerable pyramid. The repeatability is calculated for various admissible transforms, for a fixed number of extracted points equal to 400, with images of size 480 × 320.

Another way to achieve invariance to rotation is to extract a robust orientation at every extracted point and then to compute a descriptor relatively to this orientation. Such a robust orientation extraction has been proposed in [9]. In the 16 × 16 neighborhood {xi , yj }i,j of an extracted point (x, y) are computed the orientations

∂I(xi , yj ) ∂I(xi , yj ) / θi,j = tan−1 (22) ∂y ∂x and the magnitudes of these orientations by:  ∂I(xi , yj ) 2 ∂I(xi , yj ) 2 + (23) mi,j = ∂x ∂y

Analysis of Multiresolution Representations for Compression

243

The robust orientation that is assigned to the feature point (x, y) is the one maximizing the histogram of local orientations weighted by their magnitudes and a gaussian window. This principal orientation is used to partition the 16×16 neighborhood into 4 × 4 squares of size 4 × 4. An histogram of 8 orientations is computed for every square. SIFT descriptor is constructed by concatenation of these local histograms. The left part of Fig. 2 shows the high descriptor repeatability of SIFT descriptor. 4.2

Description in the Steerable Pyramid

SIFT descriptor is easily transposed into steerable representations. Using the notations of Eqn. 20, an orientation can be calculated at every point {xi , yj ; s}i,j in the 16 × 16 spatial neighborhood of an extracted point as follows: θi,j = arg

max R(xi , yj , s;

0≤k≤35

kπ ). 18

(24)

The magnitudes of these orientations are directly given by mi,j = R(xi , yj ; s; θi,j )

(25)

Given these orientations and these magnitudes, a SIFT descriptor can be calculated at every extracted point. The descriptor repeatability, shown by the right part of Fig. 2, is better than the differential moments used in the Gaussian scale space and lower than the original SIFT descriptors. The next table shows, for a database of 16000 images of size 480 × 320, the number of votes for the “good” image and the largest number of votes among the other images of the database. It is given for two random requests belonging to the database. These request images are transformed by three image transformations. request 1 request 2 rotation of angle 5 degrees 735/15 1011/4 rotation of angle 30 degrees 660/9 825/4 rescaling of factor 0.5 461/8 157/33

5

Extraction of Features in the Compressed Domain

The goal of this section is to evaluate the effect of quantizing and compressing the steerable representation on the robustness and the discriminative power of the extracted features. When quantizing the coeficients of the steerable transform, two antagonist phenomena appear in terms of robustness of extracted points. On one hand, quantization makes local maxima sharper, as the minimal step with the neighborhood correspond to the quantization step. This tends to make local maxima more robust to admissible image transformations. On the other hand, many local

244

F. Tonnin, P. Gros, and C. Guillemot POCS from 10 to 100 iterations for three different values of threshold 38 number of non zero coeficients from 30500 to 24500 number of non zero coeficients from 18500 to 15000 number of non zero coeficients from 11000 to 8500

37

36

PSNR

35

34

33

32

31

30 −1 10

0

10

entropy in bits per pixel of the original image

Fig. 3. PSNR vs entropy using POCS with the redundant steerable transform. Three different thresholds to define the initial mask hev been tested. These different values lead to different numbers of non-zero coeficient in the final representation. Effect of quantization on repeatability

Effect of quantization on descriptor repeatability 1 0.9

0.8

0.8

Descriptor Repeatability

1 0.9

Repeatability

0.7 0.6 0.5 0.4

no quantization uniform quantization at 8 bits per coeficient uniform quantization at 5 bits par coeficient

0.3 0.2

0.6 0.5 0.4 0.3 0.2

0.1 0 0.5

0.7

0.1

1

1.5

2

2.5

Precision in pixel

3

3.5

4

0 0.5

1

1.5

2

2.5

3

3.5

4

Precision in pixel

Fig. 4. Effect of quantization on repeatbility and descriptor repeatability. The image transform is a rotation of angle 10 degrees.

maxima disappear because they have a neighbor belonging to the same bin. The left part of Fig 4 shows the effect of a uniform quantization of steerable coeficients with five and eight bits per coeficient. The right part shows that these steps of quantization are sufficiently fine to preserve robustness and discriminative power of descriptors. The technique used for compression, called POCS (Projection On Convex Sets) is the main problem. As the steerable transform is highly redundant, there exists many representations corresponding to the same original image. In a compression perspective, the problem is to find a sparse one. POCS technique consists in iteratively projecting the redundant representation on two convex sets. This ensures to converge to the pair of points of these two sets of minimal distance [14]. As a first convex set, we choose the set of representations whose non-zero coeficients are provided by a fixed mask of booleans. The booleans equal to one correspond to the largest local maxima of energy, their neighborhoods, and the largest coeficients of initial representation. The second convex set is the set of representations corresponding to the original image. Using this technique, Fig 3 shows the PSNR that can be achieved for a given entropy. When iterating the projections on the two sets, the distribution of the energy in the steerable

Analysis of Multiresolution Representations for Compression repeatability in the compressed domain

Descriptor repeatability in the compressed domain

1

1

10 iterations

repeatability at two pixels

0.8

20 iterations 40 iterations

0.7

80 iterations 0.6 0.5 0.4 0.3 0.2 0.1

descriptor repeatability at two pixels

5 iterations

0.9

0

245

0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1

0

10

entropy in bits per pixel of the original image

0

0

10

entropy in bits per pixel of the original image

Fig. 5. Effect of POCS on repeatability and space repeatability. These repeatabilities are evaluated for a fixed precision of two pixels, and for a fixed image transformation, a rotation of angle 10 degrees.

representation changes significantly. Local maxima are therefore not robust to these projections. Fig 5 shows that repeatability of extracted points decrease with the number of iterations. It also shows that the repeatability does not increase with the entropy of the representation. This comes from the fact that the robustness of local maxima decrease with the number of non-zero coeficient. This figure also shows the descriptor repeatability in function of the entropy of the representation. It shows that robustness and discriminative power of descriptors remain high. This should be checked with real tests of retrieval in large databases of images.

6

Conclusion

We have analyzed various multiresolution image representations for the joint perspective of feature point and descriptor extraction and of compression. As expected, the critically sampled transforms widely used in image compression do not allow to extract robust features. A certain amount of redundancy is needed to perform such an extraction. Considering the extraction of feature points, the non decimated transform and the shiftable multiscale transform both allow to reach a repeatability comparable to the one obtained from traditional extractor used in the Gaussian scale space. This shows that low level features extraction should not be uniquely considered in a causal scale space. The steerable pyramid is of particular interest, since it fulfills the requirements for joint compression and description. It is not penalyzed by its redundancy, as there exist efficient coding schemes as POCS allowing to significantly reduce the entropy of the representation. The steerable pyramid also allows the extraction of distinctive invariant features, as shown by the tests of retrieval done in a database of 16000 images. The most important contribution of this paper is the evaluation of these invariant features extracted in the compressed domain. It has been shown that the steerable pyramid compressed by POCS and uniformely quantized is adapated to the joint problem of compression and description. Future work first consists in

246

F. Tonnin, P. Gros, and C. Guillemot

evaluating, in terms of image retrieval, the quality of local descriptors extracted in the compressed domain.

References 1. B. Beferull-Lozano and A. Ortega. Coding techniques for oversampled steerable transforms. In Proc. of thirty-third Intl. Asilomar Conf. on Signals, Systems and Computers, 1999. 2. J. Canny. A computatinal approach to edge detection. Trans. on Pattern Analysis and Machine Intelligence, pages 679–698, 1986. 3. C.H. Chen, J.S. Lee, and Y.N. Sun. Wavelet transformation for gray level corner detection. Pattern Recognition, 28(6):853–861, 1995. 4. C. Harris and M. Stephens. A combined corner and edge detector. In Proceedings of Fourth Alvey Vision Conf., pages 147–151, 1988. 5. N. Kingsburry. The dual-tee complex wavelet transform: a new efficient tool for image restoration and enhancement. In European Signal Processing Conf., pages 319–322, 1998. 6. J.J. Koenderinck and A.J. Van Doorn. Representation of local geometry in the visual system. Biological Cybernetics, pages 367–375, 1987. 7. T. Lindeberg. Scale-space theory in computer vision. Kluwer Academic Publisher, 1994. 8. E. Loupias and N. Sebe. Wavelet-based salient points for image retrieval. Technical Report RR 99.11, Laboratoire Reconnaissance de Formes et Vision, INSA Lyon, 1999. 9. D.G. Lowe. Object recognition from local scale-invariant features. In Intl. Conf. on Computer Vision, pages 1150–1157, 1999. 10. C. Schmid and R. Mohr. Local grayvalue invariants for image retrieval. IEEE Trans. on Pattern Analysis and Machine Intelligence, pages 530–534, 1997. 11. C. Schmid, R. Mohr, and C. Bauckhage. Comparing and evaluating interest points. In Intl. Conf. on Computer Vision, pages 230–235, 1998. 12. E.P. Simoncelli, W.T. Freeman E.H. Adelson, and D.J. Heeger. Shiftable multiscale transforms. IEEE Trans. on Information Theory, 38:587–607, 1992. 13. A.P. Witkin. Scale space filtering. In Intl. Joint Conf. on Artificial Intelligence, pages 1019–1022, 1983. 14. D.C. Youla. Generalized image restoration by the method of alternating orthogonal projections. IEEE Trans. Circuits and Systems, 25:694–702, 1978.

Presenting a Large Urban Area in a Virtual Maquette: An Integrated 3D Model with a ‘Tangible User Interface’ Irene Pleizier1 and Evert Meijer2 Abstract. Providing information and combining information to make a decision on the most suitable location for locating a new building or office is not a new concept. However the approach which was used in the Holland Promotion Circle (HPC) is entirely new. Instead of paper maps and analogue 3D models, a tangible interface, augmented reality and a 3D virtual environment was used to provide information for foreign investors that are considering locating their business in the Randstad area of the Netherlands. To make a selection of suitable data on the database, profile recognition through RFID was used. The Pilot period of the completion of the HPC opened the possibilities to find out the advantages of using a TUI and to experiment with the use of an enormous dataset. How to cope with the fast amounts of data and how to present the appropriate data to the visitors of the HPC.

1 Introduction The information technology improves many aspects of the communication that is needed in the process of designing, making decisions on the design, and implementing the design of office buildings, dwellings or even a new city centre. The same applies to the decision making process of finding the best suitable location for a company. For example an architect can make several variants of a design using CADtype software. The different designs can then be displayed in attractive 3D views and if necessary also on the web. This way the different designs can be compared and be shown to a large public to allow discussion and public participation. The traditional way of showing an analogue representation of the design will not be replaced by using 3D computer models, but will add to the current way of presenting spatial plans. However with the use of 3D computer models it is possible to present a larger area, because of the ability to zoom in from a greater distance. The analogue model is only useful when separate buildings and vegetation can be distinguished. The main advantage of having an analogue model is that stakeholders, planners and civilians can gather around the model to discuss the plans presented by that model. People can interact with each other and point at the model to indicate the location subjected to discussion. The disadvantage of the model is that it is static and can not be altered easily. The combination of the table surface interface and the digital representation of the plans could bring the solution. A tangible interface to allow interaction with the 1

2

Irene Pleizier finished her study in Geo Ecology. in 2003 and is currently working on her PhD on the use of 3D visualisation in spatial planning issues, in a joint project of the Spinlab, VU, Amsterdam and Geodan, Amsterdam (contact [email protected]). Evert Meijer is director and co-founder of Geodan, an Amsterdam based independent Geo-ICT Company.

S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 247 – 258, 2005. © Springer-Verlag Berlin Heidelberg 2005

248

I. Pleizier and E. Meijer

presented plans seems ideal for group discussion. The fast advancing technology brings a lot of geo-visualisation opportunities (MacEachran, 2005) The promotion of a large urban environment like in the western part of the Netherlands the Randstad, is a challenging task. On international fairs and congresses this is done by showing images, maps and mock-up models of areas of interest. Leaflets, flyers and brochures are handed out to provide information on possible building sites, ground prices, available office space and information on housing and education. The four cities of the western part of the Netherlands, jointly called ‘Randstad Holland’3, decided to combine forces and investigate the possibilities of a booth where the information is presented in a spectacular and appealing, digital way. The result should be used for several years and should therefore be based on existing databases in order to update the information regularly. The concept of the HPC was created and a first pilot version has been realised. This paper describes the various aspects of the resulting virtual maquette and gives conclusions on the usability of the concept.

2 The Holland Promotion Circle (HPC) The purpose of the HPC pilot was to provide customised and complete information about the possible locations for investment within the Randstad area. All possible relevant information was collected and stored into the central database. Information on the possible office or industrial locations as well as statistical information on employment and housing was stored in the same central database. The choice of datasets included in the database was based on the experiences of the investment offices of the four major cities of the Randstad. They are familiar with the questions that foreign investors have, when considering to invest in a new location or want to move their office to Western Europe. Because it is difficult to guess what information the investors are missing in the database, the database is designed in such a way that other and updated information can easily be added. The information stored in the database was presented on three different interfaces which presented the data on entirely different levels to provide a different visual approach to the same dataset. Not only is the data presented on another display device the presentation also takes place in another dimension. The user interfaces used for the HPC pilot can be seen in figure 1. 1. 2D display The 2D presentation of the data is the first approach to the dataset. The whole dataset is presented in a GIS-environment. The purpose of the application is to make a selection in the huge data set in such a way that the person making the enquiries is only presented with relevant data. For this purpose, profile recognition was used the working of which is described further on in this paper. 3

Based on the number of inhabitants, Randstad Holland, with its 6.6 million inhabitants, makes up about 41% of the Netherlands. In this sense the region is the fifth conurbation of Western Europe. The four main cities are Amsterdam, Rotterdam, The Hague, Utrecht.

Presenting a Large Urban Area in a Virtual Maquette

249

Fig. 1. HPC pilot setup. The Tangible interface is situated on top of the table surface With a frame holding the web-cam and projector. Behind the table two plasma screens are showing either the 2D or 3D visualisation of the project area.

The GIS application has three predefined zoom levels. The highest zoom level shows the map of the Randstad area and the location of the four major cities. Once the most suitable city has been selected on basis of the type of business the user can zoom to the next level. The second level shows the extend of the whole city with polygons showing the available investment areas of the city. The user can call all kinds of information onto the screen to find out more about the different regions on the map. When a suitable or preferred region is selected the map zooms in to the last zoom level. The third level shows the region in more detail. The streets and building blocks are visible. Now the user can see what kind of facilities are available in the area, and can find information on available office space or available building sites. Once the preferred and most suitable region is selected the user can move onto the next visualisation method. 2. 3D display Once a selection has been made in the 2D GIS-environment the chosen area of interest is visualised in 3D to be able to explore the region in more detail. The user can choose to take a guided tour through the area or to navigate manually and freely. The 3D environment is as realistic as possible to give a good impression of the atmosphere and image of the region. It is not only possible to fly over a detailed aerial photographs of the area, but is will also be possible to select buildings to find out more details, like age of the building, available square meters of office space etc. When selecting a building a link can be made to the web-site on that building or a film or sound can be played. A 2D map in the corner of the screen shows the location on the map to prevent the user from getting lost.

250

I. Pleizier and E. Meijer

Fig. 2. Two people using the tangible interface. The three coloured disks on the table are used to activate themes projected on the table. The image is projected on the table by the beamer mounted in the frame and the location of the coloured disks is registered by the web-cam hanging next to the projector.

3. Tangible interface The tangible user interface is the third element in the virtual maquette4. Instead of looking at a vertical screen using the mouse to navigate, the tangible interface is a large horizontal interface and is displayed in figure 2. Navigation is done by placing coloured disks on the table surface. The user can place a coloured disk on the table on a specific location. A web-cam hanging over the table registers the location and colour of the disk. The projection on the table changes according to the location of the coloured disk. In the situation of the HPC a green disk represented a query tool. When the disk was placed on the table, specific information on the location was projected on the table. The second colour represented travel distance and routing. When the disk was placed on the map, the shortest travel-route to a couple of points of interest such as the central station and the airport was visualised on the table. The third colour was used as a fly-over tool. When the disk was moved over the table surface, a flyover situation of the location of the disk was projected on the wall.

3 Datasets For the 2D GIS display several datasets were used. The selection of the most fundamental datasets about possible locations for establishing a new business was based on the experience of the people already involved in promoting their city to foreign investors. These datasets include direct location variables like average ground-prices, the location of investment areas defined by the city itself, the highway pattern, the railway layout, the location of the airport and the average educational 4

Earlier at MIT experiments with a tangible user interface have been conducted, see BenJoseph c.s. (2001) and Ratti c.s. (2004). Also at Ydreams (Lisbon) some experience with tangible user interfaces have been gained.

Presenting a Large Urban Area in a Virtual Maquette

251

level of the population. Based on this information companies will be able to find a suitable location. Secondly, several datasets on the socio-cultural environment were added as well. These datasets include cultural aspects such as the location of theatres and restaurants, as well as datasets on facilities such as schools and housing. Datasets for example on schools and cultural facilities are based on point locations. The exact coordinates are included in the database. This is not the case with the statistical datasets. The statistical details are averaged per 4-level postcode area of the Netherlands. There is a great variety in the way the data is represented. For indicating the specific location of for example a restaurant, an icon showing a knife and fork seems to be the most recognisable option. This counts for all the facilities that have a specific location and a specific function. An icon placed on the map will tell what can be expected at that location. For the statistical data this does not work. For example employment can not be indicated using a specific icon. Therefore the data is represented in a way that numerical values can be expressed. This can be done using coloured fields with an accompanying legend (choropleth maps) or using pie and bar charts. These charts are projected on the map over the middle of the statistical area. The 3D display application existed of flat aerial photographs in high resolution (50m and 20m) with 3D models of the built up areas placed on top of the images. For large areas like the Randstad detailed 3D models are not part of standard available geo datasets. Therefore different datasets that can be upgraded to 3D models were used. To illustrate the different levels of detail possible in the 3D content, 4 different datasets were used. The simplest one was the Top10 vector map, this is the standard 1:10.000 topographical map made by the national survey (Topografische Dienst Kadaster). The polygons of the buildings were selected and the rest of the legend items were removed. The polygons were then extruded to blocks, creating the simplest version of the buildings. A second data set, which was also part of the Top10NL, contained the footprints of individual houses. The difference between the previous dataset is that the second data set contained the footprint of individual houses and buildings and the first datasets contained only blocks of built up areas. Therefore the first dataset was a lot less precise than the first one. The dataset containing the houses could also be extruded to form blocks to represent the individual houses. The main disadvantage of these two datasets is that the height of extrusion was the same for all the polygons, which means that all the houses get the same height. This was not the case for the third dataset that was used. This was a dataset created by Tele Atlas, one of the leading geo-data providers. Tele Atlas has a huge database with detailed geo-data on all continents. For future use in car navigation, Tele Atlas is currently expanding their database with 3D information. This 3D expansion of the dataset contains several polygons per building, and a specific height per polygon. For each 4 meters of height difference within a building, a new polygon was created. This makes it possible to distinguish height differences within buildings and to recognise buildings by the shapes. The three datasets mentioned so far do not have any surface texture, which means that the buildings have only one colour. It is possible to select individual buildings and assign a different colour to it, but as this needs to be done by hand it is not very practical.

252

I. Pleizier and E. Meijer

The last type of data on buildings showing the highest level of detail is also the hardest to make on a large scale. The buildings are modelled by hand and a texture is applied to each building and can even be applied to individual building faces. The buildings can be designed in 3D modelling software such as 3D Studio MAX or Maya. These modelling programs do not have a GIS component and therefore the buildings are not geo-referenced and need to be put in the right place on top of the aerial photographs. The datasets described are illustrated in figure 3. The first and second type of dataset (the Top10NL) can be seen in the background of the image. The square blocks representing apartment buildings were made by extruding polygons are the simplest form of building representation. The third type of dataset described (the Teleatlas dataset) can be seen in front of the images. The buildings are made up of several polygons, but do not contain any textures. The shape of the building itself is

Fig. 3. Two images showing the used datasets and the potential usage of 3D visualisation. The top image shows the current situation of the ‘zuid-as’ area in Amsterdam, the Netherlands. The bottom image shows the situation in the same area in the year 2030, when the plans “Dokmodel” has been completed. The Dokmodel includes making a tunnel for the highway and railroad and placing office buildings and residential buildings on top of the tunnel.

Presenting a Large Urban Area in a Virtual Maquette

253

recognisable and makes it therefore more useful than the Top10 dataset. The fourth and last type of data described is the hand modelled buildings which can be seen in the centre of the images of figure 3. These buildings show detailed shapes and also contain textures like windows and a view on the roof of the buildings. The more detail is required for the visualisations the longer it will take to prepare the data for the various interfaces. Especially buildings and projects that still have to be realised have to be modelled by hand as no geographical data on these future projects is available. Architects and project developers do not standard model their designs and plans in 3D and even if they do create a model of their design they are not willing to share these models with others. So unless the architect has modelled their project or building already in 3D and are willing to share this, the models will have to be created manually by using the sketches and artistic drawings that have been made of the building. This is a time consuming job, but is definitely worth it when a realistic visualisation is required.

4 Profile Recognition The number of datasets in the database is so large, that making sense of the incredible amounts of data seems a difficult task. An inexperienced user will not know what data is relevant and what is not, or it will take a long time to make a useful selection. To overcome this effort and to save time when requesting information a pre-selection of the data is made according to a profile of the visitor on the booth. For the profile recognition the Geodan Movida platform for indoor location awareness (based on RFID) is used. The badge that visitors of the virtual maquette receive contains an RFID tag. All information about the visitor, which is available from registration (on the booth itself or before the visit through a website) is real time available the moment the visitor approaches the virtual maquette. The working of Movida will be explained by sketching a scenario of a foreign investor visiting the HPC. For example a foreign bank wants to open a new office in Western Europe. The representative of the bank will visit a fair where the major cities of Western Europe have a booth and the HPC is present. The representative will register his visit to the various booths via internet. On the web-site of the HPC it will be possible to leave personal information as well as information on the type of business and the necessary resources of that business. The information is stored in the database and coupled with an RFID tag, which is part of the badge. When the investor arrives at the HPC and registers his presence, he will receive the tagged badge which is coupled with his personal information. Now the visitor can start exploring the possibilities of the Randstad area. The visitor will first be presented with the 2D application to find out which location within the Randstad area will be most suitable for starting a new office. Because of the Movida system, this selection is done automatically. When he approaches the 2D map he is being recognised by Movida and his personal profile is used to make a selection on the database. The selection is presented on the screen and the most suitable location for starting a new building is indicated straight away. Of course it is possible for the investor to change his profile in case more or other criteria

254

I. Pleizier and E. Meijer

need to be considered. A clear menu can be called to the screen presenting a checklist to alter the desired information. Once the profile is changed a new customised selection on the database is presented on the screen. Once the first filter has been applied on the database, the visitor can switch on to other map layers as he desires, to find out more about the possible locations for locating a business. Once a city has been chosen and explored in 2D application, the foreign investor can move on to the 3D application to take a closer and more realistic look at the possibilities and facilities in the area of interest. The user can choose to take a guided flying tour through the area or to fly through the region to find out more about specific buildings or locations himself. Moving film, voice-overs and sounds are integrated in the 3D environment to provide complete information on the area.

5 Innovative Aspects The newest and most innovative aspects of the HPC are the Movida profile recognition, the integration of large amounts of data, the use of a tangible interface, the implementation of augmented reality and the approach towards providing and presenting aerial information to foreign investors. Especially the approach towards data presentation seems to be a totally new and promising concept. The approach creates a new vision on providing customised information in a new and appealing manner to a broad target group. In the next section we will discuss the tangible user interface (TUI) and augmented reality in more detail.

6 Tangible Interface The tangible interface is a horizontal surface on which items can be placed. The items placed interact with the projection on the table, causing the projection to change. In other words a tangible interface is a “hands-on” interface, which changes when objects on the table are moved. The system works as follows. A web-cam and a projector are hanging vertically over a table surface and are connected to a computer. The software is programmed in such a way that colours and real shapes can be recognised. In the application of the HPC, the software could recognise coloured disks. The web-cam detects the disks when they are placed within the range of the camera. The beamer projects an image on top of the table surface. The projection changes according to the location of the disks placed on the table. The application for the HPC was built with the software called Virtools, which is meant for building computer games. The system is a new and unknown way of presenting and interacting with geographical data, but it appears that users do not hesitate to interact and start to use the system instantly. The question is why the tabletop interface is so different? Moving objects on the table seems a natural thing to do, and therefore it takes away the fear of interacting with a computer system (Ratti et al; 2004). The items on the table and table itself are recognisable objects and therefore users do hardly hesitate to interact with the system. Also the response of the system to the manually moved objects is generally experienced as enjoyable and interesting. It is easy to understand the response of the system to the movement of the objects.

Presenting a Large Urban Area in a Virtual Maquette

255

Further, a discussion between several people around a table on which participants can point at, is easier and more natural than the situation where all participants are on one side of a vertical screen. This was clearly seen during the pilot presentation of the HPC and is registered in a short film that was made during the pilot. This film can be observed on the internet at www.geodan.nl/VirtualMaquette.

7 Augmented Reality An important innovative aspect of the HPC system is the implementation of Augmented Reality (AR). Augmented reality is a mix between reality and virtuality. (Milgram et al, 1994). Like the TUI the system is made up of a web-cam and a display

Fig. 4. Picture showing the combination of using the tangitable and the AR application. The tangitable and its disks can be seen in the top half of the picture. A person is holding a pattern over the table surface. In the bottom half of the image the picture captured by a simple webcam can be seen. In the filmed image, instead of the pattern, a 3D model of an office building is visualised.

256

I. Pleizier and E. Meijer

device, coupled to a computer. The web cam registers objects, icons or patterns in the real world. The computer replaces the recognised pattern with a 3D computer model. The combination of the real world captured by the web-cam and the 3D computer generated model is shown on the display device. Preferably the combined image is put out on a head mounted display: a helmet with a screen in front of the eyes. In the case of spatial planning the icons used for the application can represent buildings. An example of this can be seen in figure 4. Several buildings can be used as building blocks for spatial planning activities. The spatial configuration of the icons can be altered to experiment with the layout of the planning area. For example the impact of the placement of a large structure can be visualised and different alternatives can be compared. At the pilot demonstration of the HPC the concept of AR was demonstrated using the Geodan logo as icon. The icon was placed on the table and registered by the web cam. On the computer screen a 3D model of the Geodan building appeared on the place where the logo was put down. This concept is illustrated in the short film which can be found at www.geodan.nl/VirtualMaquette. It is not difficult to imagine a combination of the TUI and the augmented reality, meaning that the objects or disks used for the table interaction placed on the table are at the same time the objects to be recognised for the AR. We expect that this will prove to be a very powerful tool in spatial planning activities. The AR icons can be placed on the table to represent structures that not yet exist in reality. It offers the possibility to explore the consequences of a new spatial object in its environment.

8 Possibilities of the Tangible Interface In the specific case of the HPC demonstration the tangible interface seemed to be a good medium for presenting geo-information to a selective user group. However the tangible interface also shows potential for urban planning activities and public participation. Advantages of using a tangible interface for urban planning and involving the public in the planning process seems to be the possibility to stand around the table and therefore giving a good overview for everybody about the plans being presented on the table. The interface proved to be an easy to understand concept and therefore there is no hesitation to interact with the table. Planners and civilians can discuss the spatial plans together by gathering around the tangible interface and alter or modify the plans until a compromise and final plan is made. A disadvantage of using the tangible interface could be the lack of overview on what is actually presented on the table. The scale on which the plans are presented is of crucial importance. The table as it was used for the HPC did not have the ability to zoom or pan in the image. Therefore the details of the spatial plans can not be visualised. A solution to this is to include a zoom and pan function in the table. This again causes problems when more users are interacting with the table. Changing the extend of the projection on the table may be confusing for other users around the table. This effect of panning and zooming will become clear in the follow-up of the HPC project, when the tangible interface will

Presenting a Large Urban Area in a Virtual Maquette

257

have additional functionalities like projecting the 2D and 3D application on the table and the zoom and pan functionality.

9 Conclusions Not only did the tangible table provide general information of the regions projected on the table, it proved to be a tool to attract people to the table and to invite them to interact with the table itself. A positive response to the table was immediately visible, because there seemed to be no hesitation to interact with the tangible interface. The tangible interface did also allow open discussion between the visitors of the HPC. Before starting the construction of the final HPC booth, the pilot had to prove that we are capable of meeting the needs for the final HPC. This pre-result was very promising. Not only did we find out how to deal with the technical infrastructure, but also about the necessary datasets and data conversions. We learned that it is extremely important that all data conversions from ‘normally’ used databases to the virtual maquette environment have to be limited. Converting datasets can be very time consuming and for example bringing geographical data into a virtual environment on the right scale and in the right location currently needs to be done manually. In the next phase we will streamline the updating process and data conversions will be automated to prevent time consuming manual adjusting of datasets.. The way of presenting customised information is a step towards a new successful concept In the next phase it will also be necessary to experiment with other tools such as Virtools and the equivalent open source options. The experimenting will include testing level of detail management techniques. Only with a good and smooth level of detail management it is possible to show a continuous large dataset with acceptable speed. When a sufficient level of detail management can be implemented it will also be possible to include more detail in the visualisation and therefore reaching a better representation of the real world. With Virtual Netherlands we will try to take the next step towards the creation of the HPC booth and towards improving the communication and presentation in the field of choice of location. We expect that the approach towards providing information to a large target group using innovative tools such as a TUI, AR, 3D visualisation and profile recognition will be of increasing importance in the near future.

Acknowledgements The HPC pilot has been created by a team from RoVorm (communication and design, Amsterdam), Geodan (Amsterdam) and Ydreams (Lissabon). The authors like to thank the members of the pilot team: Peter Lodewijks, Peter Fokker, Brian Drescher (RoVorm), Henk Scholten, Valik Solorzano Barboza, Lex de Jong, Joost Maus (Geodan) João Serpa and João Batalha (Ydreams).

258

I. Pleizier and E. Meijer

References Eran Ben-Joseph, Hiroshi Ishii, John Underkoffler, Ben Piper and Luke yeung, 2001. Urban Simulation and the Luminous Planning Table; Journal of Planning Education and Research. P195-202 Mark Billinghurst, Hirkazu Kato, Ivan Poupyrev, 2001, The MagicBook- Moving seamlessly between reality and virtuality. IEEE Computer Graphics and Applications. pp. 6-8 Dykes, J., A.M. MacEachren & M. J. Kraak, 2005, Advancing geovisualization, Exploring Geovisualisation, Chapter 36. A MacEachren, 2005, Moving geovisualization toward support for group work, Exploring Geovisualisation, Chapter 22. Paul Milgram, Haruo Takemura, Akira Utsumi, Fumio Kishina, 1994. Augmented Reality: A Class of displays on the reality-virtuality continuum. Telemanipulator and Telepresence Technologies, Volume 2351. Carlo Ratti, Yao Wang, Hiroshi Ishii, Ben Piper, Dennis Frenchmam, 2004. Tangible user interfaces (TUIs): A Novel Paradigm for GIS. Transactions in GIS, 8(4): 407-421. Blackwell Publishing, Oxford.

Perceptual Image Retrieval Noureddine Abbadeni University of Sherbrooke, Dept. of Computer Science, Sherbrooke QC J1K 2R1, Canada [email protected]

Abstract. This paper addresses the problem of texture retrieval by using a perceptual approach based on multiple viewpoints. We use a set of features that have a perceptual meaning corresponding to human visual perception. These features are estimated using a set of computational features that can be based on two viewpoints: the original images viewpoint and the autocovariance function viewpoint. The set of computational measures is applied to content-based image retrieval (CBIR) on a large image data set, the well-known Brodatz database, and is shown to give better results compared to related approaches. Furthermore, results fusion returned by each of the two viewpoints allows significant improvement in search effectiveness.

1

Introduction

Texture is one of the most important image features that have been used in various image analysis applications such as classification, segmentation, shape from texture and more recently image retrieval [16], [13], [12]. One important property of texture is the fact that it plays a very important role in human visual perception. There is a class of texture analysis methods, in which we are particularly interested us in this paper, that are attached to use textural feature that have a perceptual meaning for users. In fact, it is widely admitted that there is a set of textural features that human beings use to recognize and categorize textures. Among these features, we can mention coarseness, contrast and directionality [5], [15]. In order to simulate the human visual perception system, we must dispose of computational techniques that allow a quantitative and computational estimation of such perceptual textural features. There are some works published in literature related to this problem since the early studies done by Julesz [11] and Bergen et al. [7] on the subject of human visual perception. Tamura and al. [15] and Amadasun and al. [5], each, proposed computational measures for a set of textural features. The work of Tamura and al. [15] was based on the cooccurrence matrix and the work of Amadasun and al. [5] was based on a variant of the co-occurrence matrix called NGTDM (neighborhood grey-tone difference matrix). The results obtained by both of them were good comparatively to human perception. Let us cite another work done by Ravishankar and al. [14] and in which the authors present what they call a texture naming system: They have S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 259–268, 2005. c Springer-Verlag Berlin Heidelberg 2005 

260

N. Abbadeni

made a tentative to determine the pertinent dimensions of the texture as in the case of color (RGB, HSI, etc). We, Abbadeni and al. [3], [4], have also proposed a set of computational measures to simulate a set of perceptual textural features, namely coarseness, directionality, contrast and busyness. Our method was based on the autocovariance function. Comparisons we have done with the works of Tamura and al. [15] and Amadasun and al. [5], by using a psychometric method based on the Spearman coefficient of rank-correlation, show that our method gives better results. In this paper, we propose to apply the set of computational measures we have proposed to CBIR. Such an approach contrasts with the most of CBIR works that, generally, use a set of mathematical image features that have no perceptual meaning for users. Furthermore, rather than to compute the computational measures from one image representation, we compute them from two image representations: original images themselves and the autocovariance function associated with images. This means that we use two different viewpoints [10]. Search results returned by each viewpoint for a given query are merged using an appropriate results fusion model. Experimental results show that such an approach allow significant improvement in search effectiveness (relevance). The rest of this paper is organized as follows: in section 2, we will briefly remind the set of computational measures used; In section 3, we define briefly both the similarity measure and the results fusion model used; In section 4, experimental results and benchmarking, based on the precision/recall measures, applied on the well-known Brodatz database of textures is shown and discussed; And, finally, in section 5 a conclusion is given.

2

Computational Measures for Perceptual Textural Features

We can find a long list of perceptual textural features in literature. However, only a small list of features is considered as the most important. This list comprises coarseness, contrast and directionality. Other features of less importance are busyness, complexity, roughness and line-likeness [15], [5]. In this study, we have considered four perceptual features, namely coarseness, directionality, contrast and busyness. We can define briefly the chosen perceptual features as follows: - Coarseness is the most important feature and, in a certain sense, it is coarseness that determines the existence of texture in an image. Coarseness measures the size of the primitives that constitute the texture. A coarse texture is composed of large primitives and is characterized by a high degree of local uniformity of grey-levels. A fine texture is constituted by small primitives and is characterized by a high degree of local variations of grey-levels. - Directionality is a global property in in an image. It measures the degree of visible dominant orientation in an image. An image can have one or several

Perceptual Image Retrieval

261

dominant orientation(s) or no dominant orientation at all. In the latter case, it is said isotropic. The orientation is influenced by the shape of primitives as well as by their placement rules. - Contrast measures the degree of clarity with which one can distinguish between different primitives in a texture. A well-contrasted image is an image in which primitives are clearly visible and separable. Among the factors that influence contrast, we cite: the grey-levels in the image; the ratio of white and black in the image; and the intensity change frequency of grey-levels. - Busyness refers to the intensity changes from a pixel to its neighborhood: a busy texture is a texture in which the intensity changes are quick and rush; a non-busy texture is a texture in which the intensity changes are slow and gradual. One can say, therefore, that busyness is related to spatial frequency of the intensity changes in an image. If these intensity changes are very small, they risk to be invisible. Consequently, the amplitude of the intensity changes has also an influence on busyness. We must note also that busyness has an inverse relationship with coarseness. We have proposed computational measures for four perceptual textural features, namely coarseness, directionality, contrast and busyness. Briefly, coarseness was estimated as an average of the number of extrema; Contrast was estimated as a combination of the average amplitude of the gradient, the percentage of pixels having the amplitude superior to a certain threshold and coarseness itself; Directionality was estimated as the average number of pixels having the dominant orientation(s); And finally, busyness was estimated based on coarseness since the two features are related to each other. The computational measures proposed for each perceptual textural feature were evaluated by conducting a set of experimentations taking into account human judgments and using a psychometric method. Thirty human subjects were asked to rank a set of textures according to each perceptual feature. Then, for each perceptual feature, we consolidate the different human rankings into one human ranking using the sum of rank values. For each feature, the consolidated human ranking obtained was compared to the ranking given by the corresponding computational measure using the Spearman coefficient of rank-correlation. Experimental results show very strong correspondence between the proposed computational measures and human rankings. Values of Spearman coefficient of rank-correlation rs found are as follows: for coarseness, rs = 0.913; for directionality, rs = 0.841; for contrast, rs = 0.755; and finally, for busyness, rs = 0.774. Comparatively to related works, our results were found better [3], [4]. Finally, note that the set of computational measures proposed can be based on two different viewpoints: the original images viewpoint and the autocovariance function viewpoint associated with images. The autocovariance function was chosen as a second viewpoint because it is known to have some desirable characteristics [3]. The use of two viewpoints results in two parameters vectors, each of size 4, to model textural content of each image. Using computational measures on one or the other of the two viewpoints (original images or the au-

262

N. Abbadeni

tocovariance function associated with images) does not give the same results for the different features, and this is the main secret behind the potential of their results merging as we will see in experimental results later in this paper.

3 3.1

Similarity and Results Fusion Similarity Measure

The similarity measure used is based on the Gower coefficient of similarity we have developed in our earlier work [2]. The non-weighted similarity measure, denoted GS, can be defined as follows: n

(k)

k=1

GSij = n

Sij

(k) k=1 δij

(1)

(k)

where Sij , the partial similarity, is the score of comparison of images i and j (k)

according to feature k and δij represents the ability to compare two images i (k)

and j on the feature k. δij = 1 if images i and j can be compared on feature n (k) (k) k and δij = 0 if not. k=1 δij = n, the number of features, if images i and j can be compared on all features k, k = 1..n. (k)

Quantity Sij is defined as follows: (k)

Sij = 1 −

|xik − xjk | Rk

(2)

where Rk represents a normalization factor. Rk is computed on the database considered for experimentations and is defined as follows : Rk = M ax(xik ) − M in(xik )

(3)

The weighed version of the similarity measure can be defined as follows: n

(k)

k=1

wk Sij

k=1

wk δij

GSij = n

(k)

(4)

where wk corresponds to a weight associated with feature k. Weighting allows to give more importance to a feature compared to another. We used two approaches of weighting. We will discuss them briefly in section 4. 3.2

Results Fusion

In order to merge the results returned by each of the two viewpoints, we have experimented several results merging model. The model which gives the best results, denoted F usCL can be defined as follows:

Perceptual Image Retrieval

K k=1

263

GSMijk

(5) K where M k denotes the viewpoint k (K in the number of viewpoints used), GSM k denotes the similarity value (score) using viewpoint M k , i denotes a query image and j denotes an image returned as similar to the query image i. Equation (5), based on the similarity value, expresses the merging of results returned by different viewpoints as an average of the scores obtained by an image in its different rankings returned by these different viewpoints. The F usCL model exploits two effects: 1. The first one known as the chorus effect in the information retrieval community [17], that is, when an image is returned as relevant to a query by several viewpoints, this is a more stronger evidence of relevance than if it is returned by only one viewpoint; 2. The second one known as the dark horse effect [17], that is, when a viewpoint ranks exceptionally an image, which is not relevant to a query, in top positions, this can be attenuated by the fused model if the other viewpoints do not rank it in top positions. Actually, it is very rare for a non-relevant image to be ranked at top positions by several viewpoints. F usCLij =

4

Application to CBIR

Computational measures, based on the two considered viewpoints, were applied in CBIR. For each of the two viewpoints, we considered two variants of the perceptual model: 1. In the first variant, we weighted each feature with the inverse of its variance, that is a feature with the smallest variance is the one who has most important weight; 2. In the second variant, we used the Spearman coefficient of rank-correlation, found when the correspondence of computational measures and perceptual features was studied, as weight for the corresponding feature. In the rest of the paper we use the following notations: - PCP-COV-V: Perceptual model based on the autocovariance function viewpoint in which each feature is weighted with the inverse of its variance. - PCP-COV-S: Perceptual model based on the autocovariance function viewpoint in which each feature is weighted with the Spearman coefficient of rank-correlation. - PCP-V: Perceptual model based on the original images viewpoint in which each feature is weighted with the inverse of its variance. - PCP-S: Perceptual model based on the original images viewpoint in which each feature is weighted with the Spearman coefficient of rank-correlation. Note that experimental results show that PCP-COV-V gives the best results when using the autocovariance function viewpoint and PCP-S gives the best results when using the original images viewpoint. So these are the two models that will be considered for results fusion.

264

4.1

N. Abbadeni

Experimental Results

We have applied the computational features presented in this paper in a large image retrieval experience on Brodatz database [8]. Each of the 112 images of Brodatz database were divided into 9 tiles to obtain 1008 128 × 128 images (112 images × 9 tiles per image). Thus, we have 112 classed of 9 images each. Images that are considered relevant to a query image are those which are in the same class as the query. Figure 1 gives a sample of images from the Brodatz database. Figure 2 shows search results obtained for query image D68-1 using the PCPCOV-S model. Figure 3 shows search results obtained for query image D110-1 using the PCP-S model. Results are presented in a decreasing order based on the score of similarity to the query image. These results show that the considered models give very good results.

D1

D21

D24

D32

D9

D109

D84

D68

D17

D29

D93

D4

Fig. 1. Different types of textures from Brodatz database

D68-1: 1.000

D68-3 : 0.992

D68-5 : 0.991

D68-8 : 0.991

D68-9 : 0.990

D68-2 : 0.990

D50-8 : 0.986

D68-4 : 0.983

D68-6 : 0.983

D68-7 : 0.983

Fig. 2. Results returned for query D68-1 using the PCP-COV-S model

Perceptual Image Retrieval

D110-1: 1.000

D110-8 : 0.978

D110-2 : 0.989

D110-4 : 0.987

D110-9 : 0.984

D50-6 : 0.973

D110-3 : 0.967

D78-5 : 0.957

265

D110-6 : 0.980

D110-5 : 0.953

Fig. 3. Search results returned for query image D110-1 using the PCP-S model

4.2

Precision and Recall Measures

Figures 5 and 6 give precision/recall graphs for different separated perceptual models and for the fused model (by fusing PCP-COV-V and PCP-S separated models). The precision and recall plotted in this figure are an average computed over 83 classes among 112 classes in Brodatz database. That is, we have rejected 29 highly non homogeneous classes since images within such classes are not visually similar in order to avoid misleading conclusions. These 29 highly nonhomogeneous classes from Brodatz database are as follows: D2, D5, D7, D13, D19, D23, D30, D31, D36, D41, D42, D43, D44, D45, D58, D59, D61, D63, D67, D73, D74, D88, D89, D90, D97, D98, D99, D100 and D108. Figure 4 shows an example of such a class of images.

D59-1

D59-2

D59-6

D59-3

D59-7

D59-4

D59-8

D59-5

D59-9

Fig. 4. An example of a highly non-homogeneous image: D59. D59-1 is the query.

266

N. Abbadeni

Fig. 5. Recall graph for different perceptual models (separated and fused): Recall = f(Retrieved images)

Fig. 6. Precision with respect to recall graph for different perceptual models (separated and fused): Precision = f(Recall)

Perceptual Image Retrieval

267

Table 1. Average retrieval rate at positions 9, 18, 50, 100 using different separated models and the fused model compared to Tamura’s model Model

P9 P18 P50 P100

Tamura’s model PCP-COV-V (112 classes) PCP-S (112 classes) FusCL (112 classes) PCP-COV-V (83 classes) PCP-S (83 classes) FusCL (83 classes)

.32 .328 .417 .549 .377 .493 .644

.46 .451 .551 .677 .529 .648 .783

.65 .626 .73 .806 .71 .825 .906

.75 .765 .842 .894 .831 .924 .959

We can point out from the precision and recall graphs that the perceptual model based on the original images viewpoint gives better performance than the perceptual model based on the autocovariance function viewpoint in both variants (weighted variant using Spearman coefficient of correlation and weighted variant using the inverse of variance). The fused model allows significant improvement compared to the separated model by exploiting both the chorus effect and the dark horse effect as mentioned earlier. These results can be also pointed out from TABLE 1 which gives the average recall rate computed over 83 classes (by excluding the 29 highly non homogeneous classes) and also over all of the 112 classes (by including the 29 highly non homogeneous classes). Including the 29 highly non-homogeneous classes, as we have mentioned, has a negative influence on search effectiveness since images within such classes are not visually similar. Table 1 shows also the average recall rate for Tamura’s model [15], which is the main work published in literature that is closely related to our work and which was applied to content-based image retrieval in the well-known QBIC system [6], [9], as benchmarked in [12] using recall measure only. From this table, it is clear that our model, in almost all of its variants, outperforms Tamura’s model. The only variant that has a similar performance as Tamura’s model is the PCP-COV-V model. The perceptual fused model FusCL performs largely better than Tamura’s model.

5

Conclusion

A multiple viewpoints perceptual approach to image retrieval was presented in this paper. A set of computational measures, corresponding to perceptual textural features, based on two different viewpoints were used to represent texture content and were applied to CBIR using a large image database, the well-known Brodatz database. Experimentations show very good results compared to related works and benchmarking based on the precision and recall measures shows a significant improvement in search performance with the fused model in particular. Further research related to this work concerns mainly possible development of semantically-meaningful features based on the set of perceptually-meaningful features used in this paper as well as application to invariant texture retrieval.

268

N. Abbadeni

References 1. Abbadeni, N.: Content representation and similarity matching for texture-based image retrieval. Proceedings of the 5th ACM International Workshop on Multimedia Information Retrieval, Berkeley, CA, USA. (2003) 63–70 2. Abbadeni, N.: A New Similarity Matching Measure: Application to Texture-Based Image Retrieval. Proceedings of the 3rd International Workshop on Texture Analysis and Synthesis (Joint with ICCV), Nice, France. (2003) 1–6 3. Abbadeni, N., Ziou, D., and Wang, S.: Computational measures corresponding to perceptual textural features. Proceedings of the 7th IEEE International Conference on Image Processing, Vancouver, BC. 3 (2000) 897–900 4. Abbadeni, N., Ziou, D., and Wang, S.: Autocovariance-based Perceptual Textural Features Corresponding to Human Visual Perception. Proceedings of the 15th IAPR/IEEE International Conference on Pattern Recognition, Barcelona, Spain. 3 (2000) 3913–3916 5. Amadasun, M., King, R.: Textural Features corresponding to textural properties. IEEE Transactions on Systems, Man and Cybernetics 19 (1989) 1264–1274 6. Ashley, J., Barber, R., Flickner, M., Hafner, J., Lee, D., Niblack, W., and Petkovic, D.: Automatic and Semi-Automatic Methods for Image Annotation and Retrieval in QBIC. Proceedings of the SPIE Conference on Storage and Retrieval for Image and Video Databases. 2420 (1995) 24–35 7. Bergen, J. R., Adelson, E. H.: Early Vision and Texture Perception. Nature 333/6171 (1988) 363–364 8. Brodatz, P.: Textures: A Photographic Album for Artists and Designers. DoverNew York (1966) 9. Flickner, M., Sawhney, H., Niblack, W., Ashley, J., Huang, Q., Dom, B., et al.: Query by Image and Video Content: The QBIC System. IEEE Computer 28 (1995) 23–32 10. French, J. C., Chapin, A. C., and Martin, W. N.: An Application of Multiple Viewpoints to Content-based Image Retrieval. Proceedings of the ACM/IEEE Joint Conference on Digital Libraries. (2003) 128–130 11. Julesz, B.: Experiments in the Visual Perception of Texture. Scientific American 232 (1976) 34–44 12. Liu, F., Picard, R. W.: Periodicity, Directionality and Randomness: Wold Features for Image Modeling and Retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence 18 (1996) 722–733 13. Mao, J., Jain, A. K.: Texture Classification and Segmentation Using Multiresolution Simultaneous Autoregressive Models. Pattern Recognition 25 (1992) 173–188 14. Ravishankar, A. R., Lohse, G. L.: Towards a Texture Naming System: Identifying Relevant Dimensions of Texture. Vision Research 36 (1996) 1649–1669 15. Tamura, H., Mori, S., and Yamawaki, T.: Textural Features Corresponding to Visual Perception. IEEE Transactions on Systems, Man and Cybernetics 8 (1978) 460–472 16. Tuceryan, M., Jain, A. K.: Texture Analysis. In the Handbook of Pattern Recognition and Computer Vision, Edited by Chen, C.H., Pau, L.F. and Wang, P.S.P. World Scientific (1993) 17. Vogt, C. C., Cottrell, G. W.: Fusion via a linear combination of scores. Information Retrieval Journal 1 (1999) 151–173

Analyzing Shortest and Fastest Paths with GIS and Determining Algorithm Running Time Turan Erden and Mehmet Zeki Coskun ITU, Civil Engineering Faculty, Department of Geodesy and Photogrammetry, Division of Surveying Techniques, 34469 Maslak Istanbul {erdentur, coskun}@itu.edu.tr

Abstract. In this paper, a few tests have been performed for determining optimum and faster path in the networks. Determining the shortest or least cost route is one of the essential tasks that most organizations must perform. Necessary software based on CAD has been developed to help transportation planning and rescue examinations in the scope of this research. Two analyses have been performed by using Dijkstra algorithm. The first one is the shortest path which only takes into account the length between any two nodes. The second is the fastest path by introducing certain speeds into the paths between the nodes in certain times. A case study has been carried out for a selected region in Istanbul to check the performance of the software. After checking the performance of the software, running time of used algorithm was examined. According to the established networks, which have different nodes, the behaviors of algorithm running time were determined separately. Finally, the most appropriate curve was fitted by using CurveExpert 1.3 program according to algorithm running times in different networks.

1 Introduction The effective use of information is getting importance and the information is increasing very rapidly. Due to the size and intensity of information volume they become excessive and complex. This information must be carefully managed by organizing. The concept of Information System appears as the result of this need. A kind of information systems, which have wide area applications, is Geographic Information System. It is provided for the best product from information by Geographic Information System (GIS) [4],[13]. GIS can perform health, culture, environment, life and security procedures, which everybody can meet. For instance, in a traffic accident on a highway, reaching to accident area at the shortest time by an ambulance and making the first aid depend on a lot of parameters. To reach accident area, precautions that take into consideration for transportation network, road information, the traffic intensity, the location of the hospital and life safety must be organised efficiently. The more important thing than the above information is to best evaluate the ‘time’ information. GIS can help gathering these types of information. At the most simple meaning, from the digital road map which is in the S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 269 – 278, 2005. © Springer-Verlag Berlin Heidelberg 2005

270

T. Erden and M.Z. Coskun

ambulance, according to traffic intensity at the accident moment alternative routing paths are chosen and it is provided for being reached to accident area [14].

2 Developed GIS Software In this study, network analysis procedure is applied as following; • Determining network analysis • Determining address information GIS Software is developed for determining optimal path and address information. The shortest path between two points is determined by means of developed software and is represented on the screen. The codes of the developed software are based on DELPHI (PASCAL based language) and it runs under Windows environment. In addition, the software has every kinds of editing procedures. The interface of the software is shown in Fig. 1. [6], [7].

Fig. 1. The interface of the software

The maps used by the software have been obtained a digital form. The digital maps used in this study include road, building, and address information. These information are derived from 1/1000 digital maps produced by photogrammetric methods. Commonly, the excessive information is not included in the application data because these maps contain too much information. First of all, 1/1000 standard topographic maps are converted into DXF format and then unnecessary layers, i.e., tree, edge, and electricity pole layers are not taken into account for this study. After that, the maps are combined together for study areas not fitting into only one section of a map. Data loses are possibly tried to prevent by ending to ending the corners of the maps and the conflicts at maps edges are removed.

Analyzing Shortest and Fastest Paths with GIS

271

One part of the Bahçelievler district of Istanbul is used as data. The area studied is given in figure 1. First, the road information on the digital maps belonging to the selected area is taken into consideration and at the intersected points of the roads, nodes and vertex is constituted respectively. The nodes are numbered starting from somewhere of the selected area and all the streets are connected together. After that, these nodes are joined and another road information is obtained. The new data obtained are represented in a different layer. The address information belonging to the studied area is entered manually. The address information is connected with the nodes which are close to them. The connected address information and the nodes are stored in a database. The constituted database and its related nodes are represented in Fig. 2.

Fig. 2. The relation between node and its address information

3 Shortest and Fastest Paths Problems The shortest paths problem is one of the most fundamental network optimization problems. This problem comes up in practice and arises as a subproblem in many network optimization algorithms. Algorithms for this problem have been improved for a long time [1], [5]. And advances in the theory of shortest paths algorithms are still being made [3], [12]. In fastest path problems, the cost of a link is the travel time of that link. In minimum-cost path problems, link costs can be of general form. While the fastest paths problem is a particular case of the minimum cost paths problem, the distinction between the two is particularly important for design of solution algorithms [2].

4 Dijkstra Algorithm and Shortest and Fastest Paths Analyzing Procedure The shortest path algorithm is applied for finding the shortest path between the start and the end points. Dijkstra algorithm is used for finding the shortest path. Dijkstra algorithm which is known as single-source shortest path algorithm is one of the

272

T. Erden and M.Z. Coskun

efficient algorithms used in network analysis [9]. Let G= (V,E) be a graph, |V|=n, |E|=m, let s be a distinguished vertex of the graph, and let c be a function assigning a nonnegative real weight to each edge of G. The objective of the single source shortest path procedure is to compute, for each vertex v reachable from s, the weight dist(v) of a minimum-weight (shortest) path from s to v; the weight of a path is the sum of the weights of its edges [11]. Data preparation for this algorithm is the most important part, which takes time. Dijkstra algorithm data structure is shown in Fig. 3.

Node

Neighbors nodes to the node

Cost

Fig. 3. The data structure of Dijkstra Algorithm

The approximate speeds for roads are introduced because real values for traffic situation on the roads and average speeds of the roads are not obtained. With this goal, roads are divided into four types. As the first-degree road, a highway road is considered and highway speed is determined according to this. As the second-degree road, main streets are considered and as the third degree road, streets joined with main streets are considered. Finally, fourth degree roads are defined. According to this, road types based on our study and determined speeds in respect of time range are given in Table 1. Table 1. Road types and average speeds

Road Types 1 2 3 4

21:00-06:00 90 km/h 50 km/h 30 km/h 20 km/h

Time range 06:00-10:00 10:00-16:00 50 km/h 80 km/h 35 km/h 40 km/h 20 km/h 25 km/h 10 km/h 20 km/h

16:00-21:00 40 km/h 30 km/h 20 km/h 10 km/h

Time is taken into account as the cost of the Dijkstra data and from the t = road length/speed equation, the value of t is determined. The algorithm is processed by using two different criteria, one of them by taking into consideration of time and the other by ignoring the time. Here, the time concept is getting importance but in this study the turns and the time loses at turns are not taken into consideration.

5 Storing Road Information in the Developed Software The road data in the software are established based on an arc-node data model. The connection between main streets and streets joined with main streets is provided with nodes. The study data are stored in the software as follows and shown in Figure 4 [6], [7].

Analyzing Shortest and Fastest Paths with GIS

• • • •

273

Every road has an Identification Number (ID). The starting point and the end point of the line information are stored in the software as Point[1] and Point[2]. In addition to that, the coordinates of the nodes are available in the software. The speeds of the roads are entered into software as road information. In the selected region in the study area, an area is determined and this area is connected with the nodes, which are close to the area.

Fig. 4. Storing the road information in software

6 Finding the Shortest and Fastest Path by Means of Road Information Stored in the Software A module, which finds the fastest path using Dijkstra algorithm, is written. The fastest path analysis can be applied for every road types in the time ranges which are determined before. Moreover, the time range is monitored instantly by means of the clock which is shown on the module. When querying from the starting point to ending point, the found nodes and paths are represented in the module. The interface of the module is represented in Fig. 5. In this study two analyses are performed: • The shortest path which only takes into account the length between any two nodes • The fastest path by introducing certain speeds into the paths between the nodes in certain times

274

T. Erden and M.Z. Coskun

The shortest and the fastest paths are examined separately. Although the total length of the path in Fig. 7 is longer than the path in Fig. 6, it can be seen that the target point is reached in a short time period.

Fig. 5. Finding the fastest path by means of road information

Fig. 6. The shortest path which only takes into account the length between any two nodes

Analyzing Shortest and Fastest Paths with GIS

275

Fig. 7. The fastest path by introducing certain speeds into the paths between the nodes in certain times

7 Complexity of Algorithm and Algorithm Running Time Tests In order to justify an algorithm, we are mostly interested in its behavior for very large problems, since these are going to determine the limits of the applicability of the Table 2. The relation between the nodes number and the running time Number of nodes 100 150 200 250 300 350 400 450 500 550 600 650

Time(second) 0.036 0.066 0.198 1.360 1.060 2.420 3.384 4.444 5.036 7.450 9.474 11.120

Number of nodes 700 750 800 850 900 950 1000 1250 1500 1750 2000

Time(second) 14.006 17.180 21.018 25.126 30.564 35.192 41.150 81.444 140.006 223.336 332.144

276

T. Erden and M.Z. Coskun

Fig. 8. The third degree polynomial fit

Fig. 9. The relation between nodes number and residuals

Analyzing Shortest and Fastest Paths with GIS

277

algorithm. Thus, the essential item is the fastest growing term [10]. In this study, according to the program coded, the running time of Dijkstra Algorithm is investigated in different networks, which have different nodes. Nodes number in different networks and running times are given in the Table 2. Algorithm is running on the Pentium II platform and the time data are obtained according to this platform. The most appropriate curve is fitted by using CurveExpert 1.3 program according to algorithm running times in different networks. The third degree polynomial fit is applied for running time data and the most appropriate fit is obtained (Fig. 8). The nodes number and running times are shown with X and Y respectively. In the result of the most appropriate fit, standard error and correlation coefficient are obtained as 0.4334844 and 0.999987, respectively. Moreover, it is observed that residuals are in the interval of 0.1 and 0.6 Fig. 9.

8 Conclusion In this paper, one part of the Bahçelievler district in Istanbul is used as a test area. In order to provide the relation between the road information together, an arc-node data model is applied. Moreover, the speeds of roads in certain time ranges are taken into account. The shortest and the fastest paths are examined separately. A program is coded in PASCAL for shortest and fastest paths analysing procedure. In practice, more comprehensive results can be obtained by introducing “stops” and “turning points”. In this study, the running time of Dijkstra Algorithm is tested in different networks. As the result of this, algorithm-running time of Dijkstra algorithm is determined. The relation between node number and algorithm running time are investigated. And the most appropriate curve is fitted.

References 1. Bellman, R., E.: On a routing Problem, vol. 16. Quart. Appl. Math. (1958) 87-90 2. Chabini, I.: Discrete Dynamic Shortest Path Problems in Transportation Applications: Complexity and Algorithms with Optimal Run Time, Transportation Research record 1645 (1998) 170-175 3. Cherkassky, B., V., Goldberg, A., V., Radzik, T.: Shortest Paths Algorithms: Theory and Experimental Evaluation, Symposium on Discrete Algorithms Proceedings of the fifth annual ACM-SIAM symposium on Discrete Algorithms, Arlington Virginia United States (1994) 516-525 4. Davis, B.: GIS: A Visual Approach. OnWord Press. Santa Fe USA (1996) 5. Dijkstra, E., W.: A note on Two Problems in Connection with Graphs, Numer. Math. (1959) 269-271 6. Erden, T: Emergency Planning in Metropolitan Cities by GIS, Master Thesis. I.T.U. Institute of Science and Technology Istanbul (2001) 7. Erden T., Coskun, M.Z.: Emergency Planning in Metropolitan Cities by GIS, International Symposium on Geodetic, Photogrammetric and Satellite Technologies-Development and Integrated Applications, 08-09 November Sofia (2001) 367-374

278

T. Erden and M.Z. Coskun

8. Husdal, J.: Fastest Path Problems in Dynamic Transportation Networks, University of Leicester UK (2000) 9. John Morris’s homepage http://ciips.ee.uwa.edu.au/~morris/courses/PLD210/dijkstra.html (1998) 10. Kreyszig, E, Advanced Engineering Mathematics, New York: John Wiley&Sons, Inc. 7th ed (1993) 11. Meyer, U.: Average-case complexity of single-source shortest paths algorithms: lower and upper bounds. Journal of Algorithms 48 (2003) 91-134 12. Shibuya, T.: Computing the n x m Shortest Paths Efficiently, the ACM Journal of Experimental Algorithmics. Vol. 5/9 ISSN 1084-6654 (2000) 13. Star., J., Estes, J. 1990. Geographical Information Systems: An Introduction, Printice Hall New Jersey (1990) 14. Yomralıoglu, T.: GIS: Fundamental Terms and Applications. Istanbul (2000)

Multimodal Data Fusion for Video Scene Segmentation Vyacheslav Parshin, Aliaksandr Paradzinets, and Liming Chen LIRIS, Ecole Centrale de Lyon, 36, Av. Guy de Collongue, 69131 Ecully, France {vyacheslav.parshin, aliaksandr.paradazinets, Liming.Chen}@ec-lyon.fr

Abstract. Automatic video segmentation into semantic units is important to organize an effective content based access to long video. The basic building blocks of professional video are shots. However the semantic meaning they provide is of a too low level. In this paper we focus on the problem of video segmentation into more meaningful high-level narrative units called scenes aggregates of shots that are temporally continuous, share the same physical settings or represent continuous ongoing action. A statistical video scene segmentation framework is proposed which is capable to combine multiple mid-level features in a symmetrical and scalable manner. Two kinds of such features extracted in visual and audio domain are suggested. The results of experimental evaluations carried out on ground truth video are reported. They show that our algorithm effectively fuses multiple modalities with higher performance as compared with an alternative conventional fusion technique.

1 Introduction Segmentation of video into semantic units provides important indexing information that can facilitate browsing and navigation in it. The basic building blocks of professional video are the shots – contiguous sequences of frames recorded from a single camera. However, the semantic meaning they provide is of a too low level. Common video of about one or two hours, e.g. a full length movie film, contains usually hundreds or thousands of shots – too many to allow for efficient browsing. Moreover, individual shots rarely have complete narrative meaning. Users are more likely to recall whole events and episodes which usually consist of several contiguous shots. In this work we are concerned with the problem of automatic movie video segmentation into more meaningful high-level narrative units called scenes. Scenes are considered as aggregates of shots that are temporally continuous, share the same physical settings or represent continuous ongoing action performed by the actors. We suppose that an input video is segmented into shots using one of conventional techniques and our task is to group them into scenes. Both the image sequence and audio track of a given video can be used to distinguish scenes. Since the same scene of movie video is usually shot in the same settings by the same cameras that are switched repeatedly, it can be detected from the image track as a group of visually similar shots. The similarity is established using the low level visual features such as color histograms or motion vectors [1]-[3]. On the other hand, a scene transition in movie video usually entails abrupt change of some audio features caused by S. Bres and R. Laurini (Eds.): VISUAL 2005, LNCS 3736, pp. 279 – 289, 2005. © Springer-Verlag Berlin Heidelberg 2005

280

V. Parshin, A. Paradzinets, and L. Chen

a switch to other sound sources and, sometimes, by film editing effects [4]-[6]. Hence, sound analysis provides useful information for scene segmentation as well. Moreover, additional or alternative features can be applied. For example, semantic classification would reveal transitions from long silence moments to music which are often accompany scene transitions [7]; classification of shots into exterior or interior ones would allow for their grouping into the appropriate scenes [8] etc. To provide reliable segmentation, there is a need to properly combine these multiple modalities so as to compensate for their inaccuracies. The common approach uses a set of rules according to which one source of information is usually chosen as the main one to generate initial scene boundaries while the others serve for their verification [4], [7] or further decomposition into scenes [8]. Rules based techniques, however, are convenient for a small number of modalities, generally do not take into account fine interaction between them and are hardly extensible. In this work we propose a statistical segmentation framework which allows us to combine multiple informational sources in a symmetrical and flexible manner and is easily extensible for new ones. The paper is organized as follows. First, in section 2 we propose our scene segmentation algorithm which is derived based on a statistical framework. Then intermediate features suitable for this framework are described in the next section. In section 4 we report the results of segmentation performance tests conducted on a database of ground truth video. Final conclusions are then given in section 5.

2 Segmentation Framework The segmentation framework proposed in this section supposes that for a given video there are several observable features which provide the evidence about the presence or absence of a scene boundary at each time point under examination. The aim is to combine them so as to yield an optimal (in some sense) sequence of scene boundaries. At each candidate point of scene boundaries (in this paper they are shot change moments) let’s consider a posterior probability p(s|D), where s ∈ {0,1} is a random variable corresponding to the presence (s=1) or absence (s=0) of a scene boundary, D – a vector of video and audio features observable in the surrounding region and providing information about s. According to the Bayesian rule

p ( s = 1 | D) =

p( D | s = 1) p ( s = 1) = p( D | s = 1) p( s = 1) + p( D | s = 0) p ( s = 0)

1 , 1 p ( s = 0) 1+ L p( s = 1)

(1)

p( D | s = 1) is likelihood ratio, p(s) – the prior probability of s. Assuming p ( D | s = 0) that feature vectors are observable locally and conditionally dependent only from the value s and that the prior probabilities of scene boundaries are fixed for a given video, the posterior probabilities defined by expression (1) can be estimated independently at each candidate point. We propose a segmentation algorithm that selects N distinct candidate points of a given video as scene boundaries so as to provide the maximal expected number of

where L ≡

Multimodal Data Fusion for Video Scene Segmentation

281

correct ones (i.e. coinciding with an actual boundary) given the sequence of observable features. Since the posterior probability value of a scene boundary p(s=1|D) is an expected number of its observations at the corresponding point, the algorithm must choose N points with maximal value of posterior probability. As it can be easily seen from expression (1), the posterior probability of a scene boundary is increasing function of likelihood ratio L and, hence, it can be selected N points with maximal value of likelihood ratio L instead. The more is N, the more points of low probability are generally selected and, hence, the less is the relative expected number of correct boundaries among them. On the other hand, the value N should be high enough to provide an acceptable level of misses (a miss is considered as an actual boundary which does not coincide with any of claimed ones). So, this value controls the tradeoff between the expected number of false alarms (selected boundaries which do not coincide with any of actual ones) and that of misses. Let’s denote by nc and nb the number of correct boundaries and the number of actual boundaries respectively. Then, supposing that each correctly detected boundary corresponds to one and only one actual boundary, for the number of false alarms nf and that of misses nm we can write

n f = N − n c , n m = nb − n c .

(2)

Hence, in order to provide the equal number of false alarms and misses and, thus, “balanced” integral performance measure of the segmentation algorithm proposed, the total number of claimed boundaries N must be equal to the actual number of scene boundaries nb. Under this condition recall r and precision p defined as

r=

nc nc , p= nc + n m nc + n f

(3)

are equal to each other and integral segmentation performance measure F1 calculated as

F1 =

2rp r+ p

(4)

is near its maximal value (as it was shown by experimental evaluations). Supposing that the mean scene duration does not differ much for different videos, we estimate the number of actual boundaries nb and set it to the parameter N according to

N=

T , S

(5)

where T denotes the duration of a video to be segmented, S is the mean scene duration evaluated from a learning data set. Experimental evaluations of the proposed segmentation algorithm have shown that its performance is greatly improved if it is constrained to select scene boundaries which are temporally apart from each other at least by some threshold value Smin. This can be explained by the fact that each observable local feature vector D used in this work is in fact conditionally dependent from its context and a high value of likelihood ratio in a point corresponding to an actual scene boundary is often accompanied by high likelihood ratio values at surrounding candidate points which should be excluded from consideration. An example of likelihood ratio curve is given in Fig. 1.

282

V. Parshin, A. Paradzinets, and L. Chen

lik e lih o o d ra tio

100 10 1 0.1 0.01 34000

35000

36000

37000

38000

39000

40000

frame #

Fig. 1. Log-scale likelihood ratio versus frame number. Vertical dashed lines delimit scenes.

So, the scene segmentation algorithm is finally formulated as follows. 1. 2. 3.

Segment an input video into shots and select shot transition moments as candidate points of scene boundaries. At each candidate point calculate the likelihood ratio for the corresponding observable feature vector. Pronounce N scene boundaries at the points with maximal likelihood ratio separated from each other at least by the temporal interval Smin, where N is calculated according to expression (5).

In multimodal segmentation observable feature vector D integrates M sources of information each described by its own vector di, i.e. D = {d1 ,..., d M } . We suppose that these sources are conditionally independent given the value s. So, we can write M

p( D | s ) ≡ p(d1 ,..., d M | s ) = ∏ p(d i | s )

(6)

i =1

and, hence, likelihood ratio of the whole data D is calculated as the product of likelihood ratio values li evaluated for each i-th informational source independently:

L≡

M M p(d i | s = 1) p (d1 ,..., d M | s = 1) =∏ ≡ ∏ li . p(d1 ,..., d M | s = 0) i =1 p(d i | s = 0) i =1

(7)

Note that this expression provides extensibility of the segmentation framework since it allows us to easily add new features as they are available. In fact the proposed segmentation algorithm detects scene boundaries at local maxima of likelihood ratio curve and thus reminds conventional unimodal techniques searching for extremums of some scene consistency measure [1], [6]. From this point of view expression (7) can be considered as a way of combining several measures calculated independently for each mode into a single curve. In this work we integrate two types of evidence about the presence or absence of scene boundaries – visual coherence and audio dissimilarity measure which are described in the next section.

Multimodal Data Fusion for Video Scene Segmentation

283

3 Feature Extraction A change of physical settings and editing effects that usually accompany a scene transition entail an abrupt change of some visual and audio parameters. Hence, both the visual and audio features can be used for the purpose of scene segmentation. In this section we propose two types of such features and the corresponding likelihood ratio estimates. 3.1 Visual Coherence

In the conventional graph-based approach [9] scenes are detected as groups of visually similar shots. Shortly this approach can be described as follows. Video scenes are usually shot by a small number of cameras that are switched repeatedly. The background and often the foreground objects shot by one camera are mostly static and, hence, the corresponding shots are visually similar to each other. In the graph-based approach these shots are clustered into equivalence classes and are labeled accordingly. As a result, the shot sequence of a given video is transformed into a chain of labels identifying the cameras. Within a scene this sequence usually consists of the repetitive labels. When a transition to another scene occurs, the camera set changes. This moment is detected at a cut edge of a scene transition graph [9] built for the video. For example, a transition from a scene shot by cameras A and B to a scene produced by cameras C and D could be represented by a chain ABABCDCD, where the scene boundary would be pronounced before the first C. In this work we propose continuous generalization of the conventional clusteringbased approach which yields flexible measure of visual scene coherence providing the confidence level of the presence or absence of scene boundary at each candidate point. Let’s consider the following shot clustering technique. First, similarity matrix for the given video is built, each element Sim(i,j) of which is the value of visual similarity between shots i and j. Then the shots which are similar enough (i.e. their similarity is higher then a threshold Tcl) are merged into clusters until the whole matrix is exhausted. This is almost a conventional clustering procedure except that the radius of the clusters is not limited. In practice we consider the shots that are far apart in time and, hence, are not likely to belong to one scene as non-similar and never combine them into one cluster. So, we need to treat only the elements of the similarity matrix located near the main diagonal which makes the computational burden approximately linear with respect to the duration of the video. Let’s define for each shot i of a given video the following variable:

C v (i ) = max Sim(a, b) . a