Advanced Procrustes Analysis Models in Photogrammetric Computer Vision [1st ed.] 978-3-030-11759-7, 978-3-030-11760-3

Table of contents :
Front Matter ....Pages i-xi
Introduction (Fabio Crosilla, Alberto Beinat, Andrea Fusiello, Eleonora Maset, Domenico Visintini)....Pages 1-3
Front Matter ....Pages 5-5
Orthogonal Procrustes Analysis (Fabio Crosilla, Alberto Beinat, Andrea Fusiello, Eleonora Maset, Domenico Visintini)....Pages 7-28
Generalized Isotropic Procrustes Analysis (Fabio Crosilla, Alberto Beinat, Andrea Fusiello, Eleonora Maset, Domenico Visintini)....Pages 29-43
Anisotropic Procrustes Analysis (Fabio Crosilla, Alberto Beinat, Andrea Fusiello, Eleonora Maset, Domenico Visintini)....Pages 45-51
Procrustes Errors-In-Variables Solutions (Fabio Crosilla, Alberto Beinat, Andrea Fusiello, Eleonora Maset, Domenico Visintini)....Pages 53-64
Front Matter ....Pages 65-65
Basics of Computer Vision (Fabio Crosilla, Alberto Beinat, Andrea Fusiello, Eleonora Maset, Domenico Visintini)....Pages 67-85
Basics of Terrestrial Laser Scanning (Fabio Crosilla, Alberto Beinat, Andrea Fusiello, Eleonora Maset, Domenico Visintini)....Pages 87-97
Front Matter ....Pages 99-99
Applications of Anisotropic Procrustes Analysis (Fabio Crosilla, Alberto Beinat, Andrea Fusiello, Eleonora Maset, Domenico Visintini)....Pages 101-151
3D Model Registration by Generalized Procrustes Analysis (Fabio Crosilla, Alberto Beinat, Andrea Fusiello, Eleonora Maset, Domenico Visintini)....Pages 153-172

Citation preview

CISM International Centre for Mechanical Sciences 590 Courses and Lectures

Fabio Crosilla Alberto Beinat Andrea Fusiello Eleonora Maset Domenico Visintini

Advanced Procrustes Analysis Models in Photogrammetric Computer Vision International Centre for Mechanical Sciences

CISM International Centre for Mechanical Sciences Courses and Lectures Volume 590

Managing Editor Paolo Serafini, Udine, Italy Series editors Elisabeth Guazzelli, Marseille, France Franz G. Rammerstorfer, Vienna, Austria Wolfgang A. Wall, Munich, Germany Bernhard Schrefler, Padua, Italy

For more than 40 years the book series edited by CISM, “International Centre for Mechanical Sciences: Courses and Lectures”, has presented groundbreaking developments in mechanics and computational engineering methods. It covers such fields as solid and fluid mechanics, mechanics of materials, micro- and nanomechanics, biomechanics, and mechatronics. The papers are written by international authorities in the field. The books are at graduate level but may include some introductory material.

More information about this series at http://www.springer.com/series/76

Fabio Crosilla Alberto Beinat Andrea Fusiello Eleonora Maset Domenico Visintini •

•

•

•

Advanced Procrustes Analysis Models in Photogrammetric Computer Vision

123

Fabio Crosilla University of Udine Udine, Italy

Alberto Beinat University of Udine Udine, Italy

Andrea Fusiello University of Udine Udine, Italy

Eleonora Maset University of Udine Udine, Italy

Domenico Visintini University of Udine Udine, Italy

ISSN 0254-1971 ISSN 2309-3706 (electronic) CISM International Centre for Mechanical Sciences ISBN 978-3-030-11759-7 ISBN 978-3-030-11760-3 (eBook) https://doi.org/10.1007/978-3-030-11760-3 Library of Congress Control Number: 2019930366 © CISM International Centre for Mechanical Sciences 2019 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface

At the beginning of the 1980s, as Assistant Professor of Surveying at the Faculty of Engineering of University of Trieste, I was involved in some research problems relating to the optimal design of geodetic control networks. The optimization model that I introduced at that time foresaw the optimal rotation of the eigenvector matrix of a n-dimensional variance–covariance matrix of unknown 2D point coordinates of a control network in such a way to dispose the bidimensional point components of the original eigenvector matrix along directions as much as possible orthogonal to those of the forecast deformations. The identification of a computational tool, able to determine the optimal n-dimensional rotation matrix (with n [ 3), was not easy. Fortunately, I was guided by University of Trieste statisticians to some books of multifactorial analysis where I could discover a chapter devoted to the so-called Orthogonal Procrustes Analysis, a set of algebraic least squares models used to directly perform transformations among corresponding matrix elements, that allowed me to easily solve my optimization problem. At the end of the 1990s, on the occasion of the 60th birthday celebration of Prof. Erik Grafarend, I was asked to present at the two-day symposium “Quo vadis Geodesia?” a short introduction on Procrustes Analysis in geodetic sciences. This opportunity offered me the possibility to further deepen this topic and to glimpse the important properties of such algorithms for the direct solution of datum transformation problems in geodesy involving the determination of unknown rotations, translations, and isotropic scale factors. Subsequent studies allowed ing. Alberto Beinat and myself to apply some new isotropic Procrustes algorithms for the solution of the photogrammetric block adjustment by independent models, for the conformal updating of a cadastral map, and for the alignment of LiDAR point clouds. These last experiments took advantage also of the valuable contribution of Prof. Domenico Visintini. More recently, the development of innovative algorithms, that consider anisotropic Procrustes models, have permitted to Prof. Andrea Fusiello and myself to find an innovative solution to the image exterior orientation problem and to its extension to the photogrammetric bundle block adjustment. Finally, Dr. Eleonora Maset was capable to find the Procrustes solution of the exterior orientation problem starting from an Errors-In-Variables (EIV) model, i.e., v

vi

Preface

for a model in which both sets of observations, image and external point coordinates, are characterized by error components. Although the theme of Procrustes Analysis constitutes the argument of some volume chapters recently written on the topics of advanced algebraic computational problems in geodesy and geoinformatics (Awange and Grafarend 2005; Awange et al. 2010; Awange and Paláncz 2016), in the scientific literature it is not present, at the moment, a volume treating in a complete and thorough manner the analytical developments and the most significant applications of Procrustes Analysis in the field of photogrammetric computer vision and laser scanning. The aim of this volume is therefore to collect under a unique roof the analytical models developed and experimented by the members of the research group Geomatics and Computer Vision of the Polytechnic Department of Engineering and Architecture of University of Udine in photogrammetric computer vision and laser scanning. The first part of the volume is devoted to the theory of Procrustes Analysis, with a detailed description of the different Procrustes models proposed and applied by the authors in the last decades. Before passing to the experimental part, some information about the basics of computer vision and some aspects of laser scanning technology and operational methodology are presented. This allows to introduce classical problems in photogrammetric computer vision and laser scanning that can be effectively solved by various Procrustes models, such as image exterior orientation, the photogrammetric bundle block and independent model adjustments and the LiDAR point cloud registration. Results obtained applying Procrustes Analysis in these fields are presented in the third part of this volume, showing also the robust version of some of the studied algorithms. The illustrated examples allow to highlight the potentiality of the proposed methods for the direct solution of nonlinear systems of equations, without any linearization of the equation systems, nor the knowledge of approximate values of the unknown parameters. These days, characterized by a rapid growing of sensor system automation and miniaturization and by the availability of huge computational resources, we are witnessing a very fast methodological evolution of problem-solving tools in the field of photogrammetric computer vision and laser scanning. For this reason, the authors of this volume warmly wish the readers to enter into the fascinating field of Procrustes Analysis and to benefit of the many computational advantages offered by this technique. Finally, as Head of the research group Geomatics and Computer Vision of the Polytechnic Department of Engineering and Architecture of University of Udine, I wish to express my gratitude to the group members for the effort put for the publication of the book. In particular, I need to thank Dr. Eleonora Maset, young Ph.D., for leading in a very effective way all the phases of the book assembly. Udine, Italy

Fabio Crosilla

Preface

vii

References J.L. Awange, E.W. Grafarend, Solving Algebraic Computational Problems in Geodesy and Geoinformatics: The Answer to Modern Challenges (Springer, 2005) J.L. Awange, E.W. Grafarend, B. Paláncz, P. Zaletnyik, Algebraic Geodesy and Geoinformatics (Springer Science & Business Media, 2010) J.L. Awange, B. Paláncz, Geospatial Algebraic Computations: Theory and Applications (Springer-Verlag, 2016)

Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part I

1 3

Theory of Procrustes Analysis Models

2 Orthogonal Procrustes Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 The Origins of Procrustes Analysis . . . . . . . . . . . . . . . . . . . . 2.2 Classification of the Procrustes Models . . . . . . . . . . . . . . . . . 2.3 OPA—Orthogonal Procrustes Analysis . . . . . . . . . . . . . . . . . 2.4 EOPA—Extended Orthogonal Procrustes Analysis . . . . . . . . . 2.5 WEOPA—Weighted Extended Orthogonal Procrustes Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 OPA and EOPA with Missing Data . . . . . . . . . . . . . . . . . . . . 2.6.1 The Case of Missing Data Dimensions . . . . . . . . . . . . 2.6.2 The Case of Missing Point Coordinates . . . . . . . . . . . 2.6.3 The Case of Missing Components . . . . . . . . . . . . . . . 2.7 Procrustes Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7.1 Procrustes Statistics with Errors in One Configuration . 2.7.2 Procrustes Statistics with Errors in Two Configurations 2.7.3 Analysis of Variance for the Procrustes Statistics . . . . . 2.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . .

. . . . .

. . . . .

7 7 8 9 11

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . .

14 17 17 19 20 21 21 22 23 25 25

3 Generalized Isotropic Procrustes Analysis . . . . . 3.1 Basic Hypothesis and Layout of the Problem . 3.2 Original GPA Solution . . . . . . . . . . . . . . . . . 3.3 Alternative GPA Solution . . . . . . . . . . . . . . . 3.4 Weighted GPA with Missing Values . . . . . . . 3.5 Properties of the Centroid . . . . . . . . . . . . . . . 3.6 Closest Point Generalized Procrustes Analysis

. . . . . . .

. . . . . . .

. . . . . . .

29 29 30 31 34 37 39

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

. . . . . . .

ix

x

Contents

3.7 Generalized Procrustes Statistics and the Analysis of Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Anisotropic Procrustes Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 AEOPA—Anisotropic Extended Orthogonal Procrustes Analysis with Row Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 AGPA—Anisotropic Generalized Procrustes Analysis . . . . . . . . 4.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Procrustes Errors-In-Variables Solutions . . . . . . . . . . . . . . . . . . . 5.1 Errors-In-Variables Model and Total Least Squares Solutions . . 5.2 Total Least Squares Solutions of Procrustes Models . . . . . . . . . 5.2.1 EIV-Extended Orthogonal Procrustes Analysis (EIV-EOPA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 EIV-Weighted Extended Orthogonal Procrustes Analysis (EIV-WEOPA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.3 EIV-Anisotropic Orthogonal Procrustes Analysis (EIV-AEOPA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Part II

40 42 43

.. ..

45 45

. . . .

. . . .

46 49 50 51

.. .. ..

53 53 54

..

55

..

58

.. .. ..

61 64 64

. . . . . . . . . . . . . . . .

67 67 70 71 72 72 74 77 78 79 80 81 82 83 84 84

An Introduction to Computer Vision and Laser Scanning

6 Basics of Computer Vision . . . . . . . . . . . . . . . . . . . . . . 6.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1.1 Collinearity Equations . . . . . . . . . . . . . . . . . 6.1.2 Camera Resection by DLT . . . . . . . . . . . . . . 6.2 Stereo Processing . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Intersection (or Triangulation) . . . . . . . . . . . 6.2.2 Relative Orientation and the Essential Matrix 6.2.3 Two-step Combined Orientation . . . . . . . . . . 6.2.4 Separate Exterior Orientation . . . . . . . . . . . . 6.2.5 Combined Single Stage Orientation . . . . . . . 6.3 Block Processing . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3.1 Bundle Block Adjustment . . . . . . . . . . . . . . 6.3.2 Resection–Intersection Method . . . . . . . . . . . 6.3.3 Independent Models Block Adjustment . . . . . 6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . .

Contents

xi

7 Basics of Terrestrial Laser Scanning . . . . . . . . . . . . . . 7.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Registration and Geo-referencing of Laser Scanning Point Clouds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Laser Scanning with Moving Systems . . . . . . . . . . . 7.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

.......... ..........

87 87

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

. . . .

90 94 96 96

8 Applications of Anisotropic Procrustes Analysis . . . . . . . . . . . 8.1 Solving the Exterior Orientation Problem with AEOPA . . . 8.1.1 Background and Related Work . . . . . . . . . . . . . . . . 8.1.2 Exterior Orientation with Fiore’s Algorithm . . . . . . 8.1.3 Exterior Orientation with AEOPA . . . . . . . . . . . . . . 8.1.4 Experimental Validation . . . . . . . . . . . . . . . . . . . . . 8.1.5 EIV-AEOPA Solution for the Exterior Orientation Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1.6 Robust Procrustes Exterior Orientation . . . . . . . . . . 8.2 Solving the Non-Perspective-n-Point Problem with AEOPA 8.2.1 NPnP Formulation . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.2 Procrustes Solution of the NPnP Problem . . . . . . . . 8.2.3 Algebraic, Direct Solution . . . . . . . . . . . . . . . . . . . 8.2.4 Experimental Validation . . . . . . . . . . . . . . . . . . . . . 8.3 Solving the Bundle Block Adjustment Problem with AGPA 8.3.1 Bundle Block Adjustment with AGPA . . . . . . . . . . 8.3.2 Experimental Validation . . . . . . . . . . . . . . . . . . . . . 8.3.3 Robust Procrustes Bundle Adjustment . . . . . . . . . . . 8.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

101 102 102 103 104 107

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . .

110 112 118 120 121 123 124 125 128 131 139 147 147

Part III

. . . .

. . . .

. . . .

Applications of Procrustes Analysis Models

9 3D Model Registration by Generalized Procrustes Analysis . . . 9.1 Solving the Independent Models Block Adjustment Problem with GPA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1.1 Experimental Validation . . . . . . . . . . . . . . . . . . . . . . 9.2 Solving the Point Cloud Registration Problem with GPA . . . 9.2.1 Global Registration with Known Correspondences . . . 9.2.2 Embedding the Correspondence Estimation Inside GPA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . 153 . . . .

. . . .

. . . .

. . . .

154 157 161 164

. . . . 164 . . . . 172 . . . . 172

Chapter 1

Introduction

Photogrammetric computer vision and laser scanning are two scientific fields whose analytical models are fundamentally based on geometrical transformations of point coordinates expressed in different reference frames. From the analytical point of view, the main problem of these transformations is that they are expressed, almost always, by nonlinear models. To solve the transformation problems, it is common to resort to the linearization of the original models and to the solution of inconsistent linearized systems of equations in order to reach, through the introduction of a proper error distribution model, the best estimation of the unknown parameters and of their precision. So, to carry out the computational procedure, it is necessary to determine, by a different method, the approximate value of the unknown parameters. The advantage of using computational tools that do not require a linearization process of the equation system and the consequent knowledge of the approximate values of the unknown parameters is, therefore, clearly evident. For this reason, in this volume we propose an alternative procedure, well known in the multifactorial analysis field as Procrustes Analysis, that allows, by one of the many variants of the original model proposed in the literature or developed by the authors of this work, to directly solve via least squares nonlinear transformation problems that generally characterize the analytical theory behind photogrammetric computer vision and laser scanning. Procrustes algorithms do not require approximate numerical solutions of the unknown parameters, offer the capability to take into consideration weights of the systems involved and admit robust procedures that allow to identify and to manage the presence of outliers. The volume is divided into three main parts. Part I is focused on the theory of Procrustes Analysis; in particular the so-called Orthogonal (OPA), Extended Orthogonal (EOPA) and Weighted Extended Orthogonal (WEOPA) isotropic models are presented in Chap. 2. A theoretical introduction to the so-called Procrustes Statistics follows. By applying a perturbation theory as an error model, it is indeed possible to perform the analysis of variance (ANOVA) of the Procrustes residuals © CISM International Centre for Mechanical Sciences 2019 F. Crosilla et al., Advanced Procrustes Analysis Models in Photogrammetric Computer Vision, CISM International Centre for Mechanical Sciences 590, https://doi.org/10.1007/978-3-030-11760-3_1

1

2

1 Introduction

in order to verify the significance of the transformation parameters obtained by the analytical process. Finally, the chapter treats the Procrustes models in the cases of missing data dimensions, missing point coordinates, and missing components. Chapter 3 describes instead the theory of the Generalized Isotropic Procrustes Analysis (GPA), while Chap. 4 extends the theoretical aspects also to the anisotropic models, namely the Anisotropic Extended Orthogonal Procrustes Analysis (AEOPA) and the Anisotropic Generalized Procrustes Analysis (AGPA). The first part of the volume ends with the generalization of Procrustes Analysis to the so-called ErrorsIn-Variables model (EIV). In contrast to ordinary least squares Procrustes approach, that finds the transformation parameters between origin and destination sets of observations minimizing errors affecting only the destination one, Chap. 5 presents the total least squares solution of both isotropic and anisotropic Procrustes models that can deal with the uncertainty affecting both sets of observations. Procrustes Analysis represents the basic tool for an alternative solution of classical problems encountered in photogrammetric computer vision and laser scanning. Hence, Part II introduces some basic concepts of computer vision (Chap. 6) and laser scanning (Chap. 7), in order to help the reader to better understand the practical applications of various Procrustes models, illustrated in Part III. In particular, Chap. 8 presents the results achieved by the authors of this volume applying the anisotropic variants of Procrustes Analysis for the following photogrammetric problems: • The exterior orientation of one image (Garro et al. 2012). Some variants of this model were implemented to find a solution when errors are considered both for image and external point coordinates (Maset et al. 2017) and to solve the NonPerspective-n-Point camera pose problem, that in turn generalizes exterior orientation to non-central cameras, i.e., generalized cameras where projection rays do not meet in a single point (Fusiello et al. 2015). • The bundle block adjustment (Fusiello and Crosilla 2015) without any a priori information on the unknown parameters. The Generalized Isotropic Procrustes Analysis has been instead successfully employed to solve the following problems, which involve 3D models originating from photogrammetric computer vision or laser scanning (Chap. 9): • The matching of different 3D models deriving from pairs of images (Crosilla and Beinat 2002) and from LiDAR data (Beinat and Crosilla 2001). • The global registration of multiple and partially overlapping LiDAR point clouds, solved in the case in which point correspondences among different point clouds are unknown (Toldo et al. 2010). The robust version of some of the previously described algorithms is also presented, together with its experimental validation. This volume is the result of years of research that, however, cannot be considered concluded yet. In fact, the properties of Procrustes Analysis make it a simple and efficient tool, potentially useful for various applications. We, therefore, hope that the reader can find among the pages of this book valuable hints to employ Procrustes Analysis to new problems and new fields.

References

3

References A. Beinat, F. Crosilla, Generalized procrustes analysis for size and shape 3D object reconstruction, in Optical 3-D Measurement Techniques, (Wichmann Verlag, 2001), pp. 345–353 F. Crosilla, A. Beinat, Use of generalised procrustes analysis for the photogrammetric block adjustment by independent models. ISPRS J. Photogrammetry Remote Sens. 56(3), 195–209 (2002) A. Fusiello, F. Crosilla, Solving bundle block adjustment by generalized anisotropic procrustes analysis. ISPRS J. Photogrammetry Remote Sens. 102, 209–221, ISSN 0924-2716 (2015) A. Fusiello, F. Crosilla, F. Malapelle, Procrustean point-line registration and the npnp problem, in Proceedings of the International Conference on 3D Vision (3DV), IEEE, pp. 250–255 (2015) V. Garro, F. Crosilla, A. Fusiello, Solving the pnp problem with anisotropic orthogonal procrustes analysis, in Second Joint 3DIM/3DPVT Conference: 3D Imaging, Modeling, Processing, Visualization and Transmission, pp. 262–269 (2012) E. Maset, F. Crosilla, A. Fusiello, Errors-in-variables anisotropic extended orthogonal procrustes analysis. IEEE Geosci. Remote Sens. Lett. 14(1), 57–61 (2017) R. Toldo, A. Beinat, F. Crosilla, Global registration of multiple point clouds embedding the generalized procrustes analysis into an ICP framework, in International Symposium on 3D Data Processing, Visualization and Transmission, pp. 109–122 (2010)

Part I

Theory of Procrustes Analysis Models

Chapter 2

Orthogonal Procrustes Analysis

2.1 The Origins of Procrustes Analysis The terms Procrustes Analysis or Procrustes Techniques are referred to a set of least squares mathematical models used to perform transformations among corresponding points belonging to a generic k-dimensional space, in order to satisfy their maximum agreement. This name comes from the Greek mythology. Procrustes was a giant that tortured his victims forcing them to lie on an iron bed and fitting their height to the bed length. If the victim was shorter than the bed length, his body was lengthened and hammered, if the victim height was longer, the outgoing part was cut off. In both cases the victim died. The term was proposed by Hurley and Cattell (1962), but the origins of these techniques are older and the credit for their first definition is disputed between psychometrists and morphometrists. Goodall (1991) quotes the psychometrist Mosier (1939) as one of the first developers of the method. The morphometrist Cole (1996), instead, indicates Boas (1905) and his fellow Phelps (1932) as inventors of the Orthogonal Procrustes Analysis and of an initial form of the Generalized Procrustes Analysis, respectively. A continuous branch of research on Procrustes Analysis, that starts from the 1950s and that is still active nowadays, is the one followed by the psychometrists. To the fundamental works by Green (1952), Cliff (1966), Schönemann (1966), Schönemann and Carroll (1970), and Gower (1975) in multifactorial analysis, the studies by Lissitz et al. (1976), ten Berge (1977), ten Berge and Knol (1984), and ten Berge et al. (1993) can be added, until the most recent contributions by ten Berge (2006) and Dosse and ten Berge (2010). Parallel to this activity, Procrustes techniques have been successfully applied in shape analysis. The reference works are those from Mardia et al. (1977), Kendall (1984), Bookstein (1986, 1997), Dryden and Mardia (1998), Goodall (1991), and Kent (1994). Some books on this topic are from Chiu et al. (2013), and Small (2012). © CISM International Centre for Mechanical Sciences 2019 F. Crosilla et al., Advanced Procrustes Analysis Models in Photogrammetric Computer Vision, CISM International Centre for Mechanical Sciences 590, https://doi.org/10.1007/978-3-030-11760-3_2

7

8

2 Orthogonal Procrustes Analysis

Recent contributions in different fields are given, for instance, by Gower and Dijksterhuis (2004), Larsen (2005), Wang and Mahadevan (2008), Kenobi and Dryden (2012). The first application of the Procrustes algorithms in geodesy is due to Crosilla (1983a), Crosilla (1983b), who proposed this technique for the construction of an ideal variance–covariance matrix “criterion matrix” for control networks. Currently, Procrustes techniques are widely applied in statistics, in multifactorial analysis, in management engineering, in biometrics, for the recognition and classification of genes and of the molecular components, especially with medical aims, as well as in the typical computer applications like image matching and robotic vision. Works appeared in geodesy and photogrammetry are those by Awange (1999), Crosilla (1999), Beinat et al. (2000), Beinat (2000), Crosilla and Beinat (2002), Grafarend and Awange (2000, 2003). More recent contributions in this field are due to Beinat and Crosilla (2001, 2004), Crosilla and Beinat (2006), Awange and Grafarend (2005), Toldo et al. (2010), Garro et al. (2012), Fusiello and Crosilla (2015), and Maset et al. (2017).

2.2 Classification of the Procrustes Models A rational subdivision useful to orient the reader in the wide field of Procrustes algorithms is given by the following proposal, that mainly considers the Orthogonal Procrustes Analysis and the Generalized Procrustes Analysis. Oblique Procrustes Analysis is not taken into account in this work, and its existence is just mentioned referring to the literature (e.g. Luk 1984). By the way, the Oblique Procrustes problem aims to separately rotate each axis of the original datum in such a way to find the least squares approximation with respect to the destination one, without imposing an orthogonal constraint to the transformation matrix (Borg and Groenen 2003). Orthogonal Procrustes techniques can be considered a collection of basic mathematical models, since they operate exclusively on pairs of matrix configurations, identified as origin and destination, respectively. Such algorithms permit to determine in a direct way the transformation parameters that better adapt the first configuration to the second one, satisfying the least squares principle and the orthogonal constraint. Due to the nature of the problems encountered in photogrammetry and laser scanning, we are interested in finding rotation matrices, i.e., orthogonal matrices with positive determinant. According to the searched parameters and to the stochastic model associated to the point configurations, the various orthogonal Procrustes models can be identified as: • OPA—Orthogonal Procrustes Analysis that allows to determine the least squares rotations between two configurations; • EOPA—Extended OPA that permits to define translations, rotations, and an isotropic scale factor of a least squares similarity transformation between two geometrical configurations;

2.2 Classification of the Procrustes Models Fig. 2.1 Functional relationship among the Orthogonal Procrustes Analysis models

9

WEOPA

OPA

EOPA

AEOPA

• WEOPA—Weighted EOPA that allows to obtain the expressions for the rotations, translations, and the isotropic scale factor between two sets of points separately weighted; • AEOPA—Anisotropic EOPA that permits to define translations, rotations, and anisotropic scale factors for each component or for each measurement of two geometrical configurations. Figure 2.1 shows the relationships existing among the various models. OPA model can be considered a particular case of the EOPA model, that in turn can be considered as a simple case of the most general WEOPA model. The most recent AEOPA formulation can be seen as a further generalization, allowing a different scaling of the components or of the points belonging to the two configurations. Generalized Procrustes Analysis (GPA), instead, allows to simultaneously consider a virtually unlimited number of matrix configurations. Employing the tools of the Orthogonal Procrustes models, GPA determines the mean (least squares) configuration among the given ones and the parameters of the various similarity transformations that relate each configuration to the mean one. As will be shown in the next chapter, GPA represents a powerful and original mathematical tool that can be applied for the solution of many problems in photogrammetric computer vision and geoinformatics.

2.3 OPA—Orthogonal Procrustes Analysis This is the fundamental model from which the successive updated ones have been developed. From the chronological point of view, it was formulated by Green (1952). The model allows to match two matrices, only by rotation, satisfying the least squares principle. The following description is inspired to the papers by Schönemann (1966) and Crosilla (1983a) with appropriate integrations. Let us consider a matrix A (origin or source) and a matrix B (destination or objective), containing a set of numerical data, e.g., the coordinates of p-points in Rk , so defined: ⎡ B B ⎡ A A ⎤ ⎤ A B x11 x12 · · · x1k x11 x12 · · · x1k A A A ⎥ B B B ⎥ ⎢ x21 ⎢ x21 x22 · · · x2k x22 · · · x2k ⎢ ⎢ ⎥ ⎥ B = ⎢ .. .. . . .. ⎥ . (2.3.1) A = ⎢ .. .. . . .. ⎥ ⎣ . . ⎣ . . . . ⎦ . . ⎦ A A A x p2 · · · x pk x p1

B B B x p2 · · · x pk x p1

10

2 Orthogonal Procrustes Analysis

The method allows to directly estimate the unknown rotation matrix R, for which the square of the Frobenius (or Euclidean) norm of the transformation residual matrix is minimum, i.e.,   E2F = AR − B2F = tr E E = tr EE = min

(2.3.2)

with the condition that the k × k matrix R is orthogonal, that is R R = I. Defining a function F = F(E, R, L), where L is a diagonal k × k matrix of unknown Lagrangian multipliers, and exploiting the trace properties, the cost function F can be expressed as: 

  F = tr E E +tr L R R − I

  = tr (AR − B)  (AR − B) + L R R − I

 = tr R A AR − R A B − B AR + B B + LR R − L .

(2.3.3)

The minimum condition is solved imposing the partial derivative of F with respect to R equal to zero:   ∂F = A A + A A R − 2A B + R L + L ∂R  = 2A AR − 2A B + R L + L = 0.

(2.3.4)

Multiplying on the left all the elements by R , exploiting the orthogonal condition of matrix R, and being L + L an unknown symmetric matrix, then the condition:  2R A AR − 2R A B + R R L + L = 0

(2.3.5)

implies that R A B is symmetric, since R A AR is symmetric, and R R = I. Calling (2.3.6) S = A B and exploiting the simmetry of R S, it follows:  R S = R S  = S R.

(2.3.7)

Carrying out the singular value decomposition (SVD) of S = VD S W and substituting it in the previous relationship yields:  R VD S W = VD S W  R = WD S V R. From a comparison of the corresponding terms of the matrix products, it results in: W = V R.

2.3 OPA—Orthogonal Procrustes Analysis

11

Reminding that V and W are orthonormal, it is possible to obtain the expression of the rotation matrix R, that represents the final result computed as a product of two eigenvector matrices: (2.3.8) R = VW . This formula only guarantees that R is orthogonal. The least squares estimate of a rotation matrix (Wahba 1965) obtained by  R = V diag 1, 1, det(VW ) W .

(2.3.9)

Summarizing, from the computational point of view, given A and B, after computing the SVD of the matrix product A B, that is A B = VD S W , it is possible to directly determine the rotation matrix R as  R = V diag 1, 1, det(VW ) W .

2.4 EOPA—Extended Orthogonal Procrustes Analysis EOPA model is used to determine the least squares similarity transformation between two matrices. Its first formulation was given by Schönemann and Carroll (1970) and is reported in the following. Given matrices A and B, with the meaning assigned in the preceding section, EOPA allows to define, in addition to the rotation matrix R already retrieved with the OPA model, also a translation vector t and a global scale factor c. It corresponds ˜ defined as: to searching a matrix B B˜ = cAR + jt

(2.4.1)



with j = 11 12 · · · 1 p  , for which the following condition is satisfied: 2 E2F = B − B˜ = min F

(2.4.2)

under the orthogonality constraint: R R = RR = I. The minimum condition reported in (2.4.2) is equivalent to:  (2.4.3) tr E E = min . Defining a Lagrangian function F = F (E, t, R, c, L), where L is an unknown matrix of Lagrangian multipliers, it follows

12

2 Orthogonal Procrustes Analysis



  F = tr E E + tr L R R − I

    = tr B − cAR − jt  B − cAR − jt + L R R − I = tr[B B + c2 R A AR + tj jt − 2cR A jt − 2B jt − 2cB AR + LR R − L]

(2.4.4)

and imposing the partial derivates of F with respect to R, t, and c equal to zero, it results in:  ∂F = 2c2 A AR − 2cA B + 2cA jt + R L + L = 0 ∂R ∂F = 2j jt + 2cR A j − 2B j = 0 ∂t    ∂F = 2c tr R A AR + 2 tr R A jt − 2 tr B AR ∂c   = c tr R A AR − tr R A B − R A jt = 0.

(2.4.5) (2.4.6)

(2.4.7)

Please note that the product j j generates a scalar p, corresponding to the number of points in A and B. After Formula (2.4.5) on the left by R , observing  multiplying     that R A AR and R R L + L are symmetric, one can notice that also R A B − R A jt

(2.4.8)

must be symmetric. Substituting in Eq. (2.4.8) the expression of t that results from (2.4.6): j (2.4.9) t = (B − cAR)   jj the symmetric condition for the global term becomes1 :     

jj jj sym R A I −  B + cR A  AR . jj jj Since R A

  jj j j

AR is symmetric, it consequently happens: 

  jj sym R A I −  B . jj

(2.4.10)

jj is symmetric and idempotent. Its role is to translate j j the matrix values to which it is applied (in this case A and B) to the corresponding barycenter. Calling Please note that matrix I −

1 The

predicate sym[] is true when the argument is a symmetric matrix.

2.4 EOPA—Extended Orthogonal Procrustes Analysis

  jj S = A I −  B jj

13

(2.4.11)

as done for the OPA model, it results in:  R S = R S  = S R. Performing the SVD of S:

it follows:

(2.4.12)

S = VDs W

 R VDs W = VDs W  R = WDs V R.

Multiplying on the left by R, from the comparison of the corresponding terms of the equation: (2.4.13) VDs W = RWDs V R since RR = I, one obtains V = RW, that is: R = VW .

(2.4.14)

In order to retrieve a rotation matrix, one must resort to the following formulation:  R = V diag 1, 1, det(VW ) W .

(2.4.15)

Once R is known, it is possible to compute t from (2.4.9) and the scale factor c can be retrieved from Eq. (2.4.7). Substituting the expression for vector t, the solution becomes:           tr R A I − jjj j B tr R A I − jjj j B   =     c=  (2.4.16)   tr R A I − jjj j AR tr A I − jjj j A after having considered the orthogonality condition of R. As underlined by the same authors of the EOPA method, the residual matrix E, that measures the discrepancies of matrices A and B after having applied the similarity transformation, is independent of the translation vector t. In other words, the matching result does not depend on the relative distance between the barycenters of A and B. In fact, one can retrieve E as:   jj (2.4.17) E = B − B˜ = I −  (B−cAR) jj where t is not considered. Summarizing, given A and B, it is convenient to first ¯ and B: ¯ calculate the barycenter matrices A

14

2 Orthogonal Procrustes Analysis

    ¯ = I − jj A = I − jj A A j j p

(2.4.18)

  jj ¯ B= I− B. p

(2.4.19)

¯  B¯ has been carried out, that is Once the SVD decomposition of the matrix product A   ¯ = VDs W , it is possible to directly compute R = VW . The scale factor is then ¯ B A given by: 

¯  B¯ tr R A 

(2.4.20) c= ¯ ¯ A tr A ¯ while the translation which corresponds to the least squares solution of B¯ = cAR, is: t = (B − cAR)  j/ p. (2.4.21) Please note that multiplying by the term j/ p in the previous expression corresponds to perform the mean of the difference B − cAR.

2.5 WEOPA—Weighted Extended Orthogonal Procrustes Analysis This model allows to determine the similarity transformation between two matrices, permitting a different weighing for each point (by rows), for each component (by columns), or for both. The first formulation was proposed by Goodall (1991) and the solution is presented below, following the derivation reported in Beinat (2000). Given a p × p weight matrix P2R and a k × k weight matrix PC2 , the unknown parameters R (rotation), t (translation) and c (isotropic scale factor) are searched so to satisfy the following minimum condition:  

 tr B − cAR − jt  P2R B − cAR − jt PC2 = min

(2.5.1)

under the orthogonality constraint R R = RR = I. Please note that P2R weights the points, while PC2 weights the components. We shall allow P2R and PC2 to be any square matrix, but in most applications they will be diagonal (Gower and Dijksterhuis 2004). Let us now separately consider the different situations, according to the value assumed by the weight matrices. First case: Point weighing (P2R = I and PC2 = I). If P2R = I is real, symmetric, and positive definite, the minimum condition (2.5.1) corresponds to the minimum of the Lagrangian function:

2.5 WEOPA—Weighted Extended Orthogonal Procrustes Analysis

F = tr B P2R B + c2 R A P2R AR + tj P2R jt − 2cB P2R AR − 2B P2R jt + 

 +2cR A P2R jt + tr LR R − L .

15

(2.5.2)

Imposing the partial derivatives of F with respect to R, c, and t equal to zero, it follows:  ∂F = 2c2 A P2R AR − 2cA P2R B + 2cA P2R jt + R L + L = 0 ∂R ∂F = 2tj P2R j + 2cR A P2R j − 2B P2R j = 0 ∂t   ∂F = 2c tr R A P2R AR − 2tr R A P2R B − R A P2R jt = 0 ∂c

(2.5.3) (2.5.4) (2.5.5)

and from (2.5.4) it results in: t = (B − cAR) 

P2R j . j P2R j

(2.5.6)

From (2.5.3), after having multiplied on the left all the terms by R and observ   ing that R R L + L and R A P2R AR are symmetric matrices, it follows that also R A P2R B − R A P2R jt is symmetric. Substituting the expression of t given by (2.5.6), one gets: 

P2 jj P2 P2 jj P2 sym R A P2R B − R A R 2 R B + cR A R 2 R AR . j PR j j PR j

(2.5.7)

Furthermore, observing that the term R A

P2R jj P2R AR j P2R j

(2.5.8)

in Eq. (2.5.7) is also symmetric, it is possible to conclude:

   jj P2 P2 jj P2 sym R A P2R B − R A R 2 R B = R A P2R I −  2R B . j PR j j PR j Defining

  jj P2 S = A P2R I −  2R B j PR j

(2.5.9)

(2.5.10)

one reaches the same condition of (2.3.7) and (2.4.12). Being S = VDs W

(2.5.11)

16

2 Orthogonal Procrustes Analysis

the SVD of matrix S, as for Eqs. (2.3.9) and (2.4.15), the rotation is given by:  R = V diag 1, 1, det(VW ) W .

(2.5.12)

For what concerns the scale factor c, substituting in (2.5.5) the expression of t given by (2.5.6), it is possible to obtain:     jj P2 tr R A P2R I − j P2Rj B R   . c=  jj P2R 2   tr R A P R I − j P2 j AR

(2.5.13)

R

Summarizing, given matrices A, B, and P2R , matrix S is computed by means of (2.5.10) and the SVD of S is carried out to directly obtain R. The scale factor c is then computed by (2.5.13), and finally, t is derived by means of (2.5.6). Being P2R symmetric and positive definite, it admits the Cholesky decomposition 2 P R = P R  P R . It is then useful to observe that the WEOPA model is nothing but the EOPA model applied to the matrices P R A and P R B. This can be proven substituting matrix A with P R A, matrix B with P R B, and j with P R j in the formulas of the EOPA model, obtaining the same expressions (2.5.6), (2.5.10), and (2.5.13) derived up to now. In fact, the expression of the translation t of the EOPA model, given by (2.4.9), becomes: PR j P2 j = (B − cAR)   R2 t = (P R B − cP R AR)    j PR PR j j PR j while matrix S of the EOPA model (given by (2.4.11)), from which the components of R are obtained, results from:   P R jj P R    S = A PR I −   PR B j PR PR j     jj P R  P R jj P2R   2  B = A P R I −  2 B. = A PR PR I −   j PR PR j j PR j Finally, the expression of the scale factor c, computed from Eq. (2.4.16) of the EOPA model, is:           jj P2 tr R A P R  I − Pj PR jjR  PPRRj P R B tr R A P2R I − j P2Rj B R     . =  c=  jj P2R P R jj P R  2      tr R A P R I − j P R  P R j P R AR tr R A P R I − j P2 j AR R

Weights contained in matrix P R can be chosen according to the precision associated to each point. However, an appropriate weighing procedure can be applied also to detect outliers in matrices A and B, that could skew the Procrustes least squares estimate. Following the Iteratively Reweighted Least Squares (IRLS) method (Holland and Welsch 1977), problem (2.5.1) can be solved iteratively, estimating weights

2.5 WEOPA—Weighted Extended Orthogonal Procrustes Analysis

17

at each iteration by analyzing the residuals of the current solution. The weights are assigned by a specific weight function in such a way to penalize outliers and promote inliers. More details on this robust procedure and its application will be described in Chap. 8. Second case: Component weighing (P2R = I and PC2 = I). In this case, no algebraic direct solutions are known in the literature (Koschat and Swayne 1991; Gower and Dijksterhuis 2004). In fact, starting from Eq. (2.5.1), one can demonstrate that matrix R can be directly computed via a SVD decomposition only if ¯ =I ¯ R RPC2 R = R

(2.5.14)

PC2 = I

(2.5.15)

that is satisfied when

in contrast to the initial assumption PC2 = I. Therefore, such a problem can be faced only by adopting an iterative algorithm like the one proposed by Koschat and Swayne (1991).

2.6 OPA and EOPA with Missing Data In the preceding models, matrices A and B (origin and destination) have the same dimensions: they contain the same number of points and components. Different algorithms have been proposed (e.g. ten Berge et al. 1993) to solve the cases when entire dimensions (i.e., complete columns), entire point coordinates (i.e., complete rows), or some point components are missing in one matrix or in both. It must be stressed that such algorithms furnish enough precise results if the lack of data is limited. Experimental tests carried out by the aforementioned authors have put in evidence that, if the available data are affected by errors and the number of missing components is significant, the algorithms suffer to reproduce in a reliable manner the general structure of the complete set of data.

2.6.1 The Case of Missing Data Dimensions The problem is known in the literature as Asymmetric (ten Berge and Knol 1984) or Unbalanced Orthogonal Procrustes Problem (e.g. Park 1991). In general terms, given A ∈ R p×m and B ∈ R p×n with m > n, a semi-orthogonal matrix R ∈ Rm×n is required, such that: AR − B2F = min (2.6.1)

18

2 Orthogonal Procrustes Analysis

where R R = I. Park (1991) proposed a very efficient solution to the problem, that can be implemented for the parallel computation. Neglecting the case in which m = n, that directly leads to the OPA model, in the other situations where m > n, the condition R R = RR = I is not satisfied and therefore the considerations about the symmetry of the matrices, directly leading to the Expression (2.3.9), are no longer applicable. To overcome this issue, all the existing algorithms propose to address the unbalanced problem through an iterative solution of symmetric problems. First of all, m − n columns are added to matrix B so to obtain a matrix B¯ having the same shape as matrix A. The two successive steps are then iterated till convergence: • the EOPA model is solved between A and B¯ so to obtain     R = R1m×n R2m×(m−n) • the m − n columns of B¯ are substituted by the product AR2 . Green and Gower (1979) proposed to fill in the m − n added columns of B with zero entries or with random numbers. In (ten Berge and Knol 1984), the authors ¯ in which the supplementary suggested to start the iterative process with a matrix B, columns are given by the product A. , in this case, is a matrix whose columns contain the components of the m − n eigenvectors associated to the smallest eigenvalues of A A. This last approach is assumed to be more reliable against local minima and faster than the previous one. The critic that Park (1991) moved to both methods is the low computational efficiency, since they both require to perform the SVD of a matrix product many times. He therefore proposed a new algorithm that transforms the unbalanced problem into an equivalent symmetric problem in which a new destination matrix is only partially defined. Such algorithm benefits of the procedure proposed by Heath et al. (1986). In this way, R is computed through a sequence of simple plane rotations. The Park’s method (Park 1991) is characterized by the following steps: • Given A p×m and B p×n with m > n, matrix B¯ is built with the same dimensions as A, but defined only in correspondence of the first n columns through the corresponding elements of B. The remaining columns are never involved in the computation. In this way, the asymmetric problem is transformed into a symmetric one. ¯ m×m corresponds to the one that best adapts the first n columns • The rotation matrix R p×m ¯ m×m ¯ m×m , i.e., R = R to those of B p×n . The first n columns of of the product A  R   ¯ 2m×(m−n) . ¯ m×m = R ¯ 1m×n R ¯ 1m×n , constitute the searched solution, being R R The reader must consider that the search of R is possible thanks to the use of the cited procedure by Heath et al. (1986). In fact, it permits to iteratively define R, solving all the possible OPA models between homonymous column pairs of A and B, and between column pairs of A and the only correspondent of matrix B, repeating the entire cycle till (fast) convergence. For instance, in case of a matrix A p×4 and a matrix B p×3 , all the possible OPA models that generate a chain of rotations R2×2 occur among the columns:

2.6 OPA and EOPA with Missing Data

19

(1, 2) A ↔ (1, 2) B ; (1, 3) A ↔ (1, 3) B ; (1, 4) A ↔ (1) B ; (2, 3) A ↔ (2, 3) B ; (2, 4) A ↔ (2) B ; (3, 4) A ↔ (3) B .

2.6.2 The Case of Missing Point Coordinates Although proposed by Commandeur (1991) to solve missing row problems in the case of m data matrices in a Generalized Procrustes Analysis problem (see Chap. 3), the method here reported is easily adaptable also to the case of two matrices, A and B, each one missing one or more corresponding rows, and it can be considered a special case of WEOPA. Commandeur suggested to premultiply matrices A and B through appropriate diagonal binary matrices, setting 1 for the terms of the main diagonal corresponding to existing rows, and 0 otherwise. This allows to solve OPA or EOPA problems, substituting A with M A A∗ and B with M B B∗ . A∗ is used to indicate the enlarged matrix of A, to which some rows have been added. These contain zeros (or any random number) in the positions where the elements of A are missing, having a correspondence with B. The same holds for B. In this way, A∗ and B∗ have the same dimensions. Matrices M A and M B permit to carry out the Procrustes procedure only acting on the common points of A and B. In this perspective, it would be more logic to operate on the original matrices A and B, that is those matrices constituted only by the points of the first matrix that have a correspondence on the second one, and vice versa. But in this case, the algorithm capability of completing the unspecified entries would be hidden. The missing values of B can in fact be identified by transforming the existing values of A through the computed similarity transformation parameters. More specifically, for the generic missing row of B, it is: bˆ m = cam R + t .

(2.6.2)

Similar considerations can be done for the opposite case, where the missing values of A can be estimated by the corresponding ones of B, transformed through the inverse similarity transformation just mentioned: aˆ m =

1 bm − t R . c

(2.6.3)

All the supporting mathematical treatments can be easily traced back to the EOPA model, according to the considerations developed for the WEOPA model with weighing by rows. Considering that in this case the weight matrix is diagonal and defined by: P2R = P R  P R = M A M B

(2.6.4)

20

2 Orthogonal Procrustes Analysis

this directly implies, due to the binary character of the matrices: PR = M AMB = PR .

(2.6.5)

Formulas (2.4.9), (2.4.11), and (2.4.16) can then be applied, after having substituted A∗ with M A M B A∗ , B∗ with M A M B B∗ , and j∗ with M A M B j∗ .

2.6.3 The Case of Missing Components Gulliksson (1995) discussed some methods to adapt A over B when A contains a certain number of missing components, defining this case as a partial Procrustes problem. The algorithm here proposed is partially inspired by the method of Commandeur (1991) mentioned before, and partially by the specific procedure proposed by ten Berge et al. (1993). This last treats the case of missing components within problems in which m matrix configurations contemporary intervene. It must be stressed that in case of contemporary missing components in the configurations A and B, it is not possible to perform any kind of prediction, since the required information, necessary to identify the underlying data structure, is missing. The basic idea is that missing values can be substituted by “optimal” values, following the least squares principle. In case of two matrices A and B, such values are estimated according to the solution R for which it is satisfied: 2 F (A, B, c, R, t) = cAR + jt − B F = min

(2.6.6)

subjected to the usual orthogonality constraint R R = I and to the further condition for which the values of A and B that are not missing are considered as fixed values. The solution is reached in an iterative way: 1. The missing elements of both matrices are memorized by defining two binary matrices M A and M B , with the same dimensions of A and B, having 1 in correspondence of a missing value and 0 otherwise. With these allocations, the initial values of R, c, t are computed by satisfying the condition (2.6.6), used as a convergence criterion. 2. The Procrustes best-fit matrix B p = cAR + jt is computed. From it, the new values of the missing components are extracted. These values are inserted in B maintaining unchanged the others. In symbolic notation, this becomes: B = B  (∼ M B ) + B p  M B

(2.6.7)

where  represents the Hadamard product (element by element) and ∼ (not) is the logic function that inverts the binary value of the terms to which it is applied. 3. The same procedure is than applied to A. The best-fit inverse matrix is computed according to the just updated matrix B, from which the values attributed to the

2.6 OPA and EOPA with Missing Data

21

missing components of A are selected. These values are substituted in matrix A through the corresponding form of Expression (2.6.7). 4. With A and B so updated the transformation parameters R, c, t are recomputed, repeating steps 2 and 3 till convergence. It must be emphasized that the objective of the method is to determine the similarity transformation parameters of A over B. The values of the missing components are a secondary product. Their precision depends on the precision of the known values.

2.7 Procrustes Statistics Procrustes statistics is the residual sum of squares obtained after a Procrustes transformation between a pair of origin and destination matrix configurations (Sibson 1978). The statistics is based on the application of a perturbation theory to the point coordinates (Sibson 1978), leading to an approximate normally distributed model. By applying the analysis of variance (ANOVA), it is possible to separately investigate the significance of each of the Procrustes transformation parameters against the statistics of the final residuals. In the following, the cases where the perturbation model is applied to one matrix and to both matrices will be presented.

2.7.1 Procrustes Statistics with Errors in One Configuration Let A be, as before, an origin matrix containing a set of numerical data, e.g., the coordinates of n points in Rk . Let B be a n × k destination matrix containing the coordinates of the same points, obtained summing up to A a matrix of random errors E of zero mean and unit variance, with E premultiplied also by a common factor s, i.e., B = A + sE (2.7.1) where E ≈ N (0, 1), or equivalently vec (E)  N {0,  = Q P ⊗ Q K },2 with Q P = I, Q K = I and ⊗ representing the Kronecker product. The Procrustes distance between the two configurations is given by the sum, extended to all the n points, of the square coordinate differences, that is in matrix form:   G (A, B) = tr (B − A)  (B − A) = s 2 tr E E .

(2.7.2)

The initial quantity G (A, B), due to the normal distribution of the elements of E, follows a χ 2 distribution with n × k degrees of freedom. To match one matrix with respect to the other, for what seen before, the procedure requires the sequential application to one of the two matrices of a group of geometrical transformations: 2 The

vec operator transforms a matrix into a vector by stacking its columns.

22

2 Orthogonal Procrustes Analysis

– translation; – translation and rotation; – translation, rotation, and isotropic dilatation; in such a way to minimize the Procrustes distance between the two matrices, for each kind of transformation. This condition can be written for all the transformations as the satisfaction of the following expressions:   G t (A, B) := inf G A + jt , B

(2.7.3)

  G t,R (A, B) := inf G AR + jt , B

(2.7.4)

  G t,R,c (A, B) := inf G cAR + jt , B .

(2.7.5)

By a series expansion of the generic Procrustes distance G φ (A, A + sE) with respect to s, where φ stands in turn for translation (t), translation and rotation (t, R), translation, rotation, and dilatation (t, R, c), Sibson (1979, p. 222, Lemma 3.3) showed, and experimentally verified, that the distances G t (A, A + sE) and G t,R (A, A + sE) follow a χ 2 distribution, with [nk − k] and [nk − 1/2k(k + 1)] degrees of freedom, respectively. This holds also for great values of s, that is: 2 G t (A, A + sE)  s 2 χ[nk−k] 2 G t,R (A, A + sE)  s 2 χ[nk−1/2k(k+1)]

while the quantity G t,R,c (A, A + sE) (Sibson 1979, p. 225, Proposition 3.6) follows a χ 2 distribution with [nk − 1/2k(k + 1) − 1] degrees of freedom and remains valid just for small values of s, that is: 2 . G t,R,c (A, A + sE)  s 2 χ[nk−1/2k(k+1)−1]

Goodall (1991, p. 312) reported that the χ 2 distribution of G t , G t,R , and G t,R,c holds also for Q P = I, that is for a different weight distribution of the various points.

2.7.2 Procrustes Statistics with Errors in Two Configurations Let us now consider the case in which two matrices A and B are obtained by the same coordinate matrix Z to which two independent matrices E A and E B are summed up. E A and E B contain random errors with zero mean and unit variance, and both matrices are premultiplied by the same factor s (Langron and Collins 1985, p. 279), i.e., (2.7.6) A = Z + sE A

2.7 Procrustes Statistics

where E A ∼ N (0, 1),

23

B = Z + sE B

(2.7.7)

where E B ∼ N (0, 1), or, for both cases: {vec (E A ) , vec (E B )} ∼ N {0,  = Q P ⊗ Q K } with Q P = I and Q K = I. The original Procrustes distance between A and B is given by:  G (A, B) = tr (B − A)  (B − A)  = s 2 tr (E B − E A )  (E B − E A )  = s 2 tr W AB  W AB

(2.7.8)

where the matrix W AB = E B − E A follows a N (0, 2) distribution, due to the independence of E B and E A . Analogously to the previous case, where just one matrix configuration was modified by a random error matrix, G(A, B) follows a χ 2 distribution with n × k degrees of freedom (see Langron and Collins 1985, p. 281); that is: 2 . G (A, B)  2s 2 χ[nk]

(2.7.9)

We proceed again to the matching of one matrix with respect to the other by applying geometrical transformations of the following kinds: – translation; – translation and rotation; – translation, rotation, and dilatation. Langron and Collins (1985, pp. 279 and 280, Theorems 4.2 and 4.3) proved that the types of distributions that characterize the residual distances are the following: 2 G t (A, B)  2s 2 χ[nk]

(2.7.10)

2 G t,R (A, B)  2s 2 χ[nk−1/2k(k+1)]

(2.7.11)

2 . G t,R,c (A, B)  2s 2 χ[nk−1/2k(k+1)−1]

(2.7.12)

2.7.3 Analysis of Variance for the Procrustes Statistics The intention is to assess the significance of the parameters of each geometrical transformation, i.e., translation, rotation and dilatation, that minimize, independently, the Procrustes distance between matrices A and B. The proposed test is based on

24

2 Orthogonal Procrustes Analysis

the analysis of variance (ANOVA) of the various contributions and consists in the study of the statistical distribution characterizing the numerical values obtained as the ratio between the Procrustes distance, specific for each single transformation (translation, rotation, and dilatation), and the residual distance when a complete similarity transformation is carried out. Let us first consider the result of a translation procedure of A and B. The Procrustes distance that remains after centering both matrices is given by (see Sibson 1978, p. 235):  

 G t (A, B) = tr B − jt B − A + jt A  B − jt B − A + jt A

 = tr (B − A)  (B − A) − n (b − a)  (b − a)

(2.7.13)

where a and b are the positional vectors of the centers of gravity of the matrices A and B expressed in the original reference system. The difference: T (A, B) = G (A, B) − G t (A, B)

(2.7.14)

corresponds to the Procrustes distance due to the translation of the k-components of the two matrices. This value will be employed afterward in a statistical test in order to verify its significance. The residual distance G t (A, B) following the translation can be further partitioned into a sum of squares due to the rotation R (A, B) and in a residual distance defined by G t,R (A, B): R (A, B) = G t (A, B) − G t,R (A, B) .

(2.7.15)

Under the hypothesis previously made on matrices A and B, the Procrustes distance R (A, B), due to the rotation, follows a χ 2 distribution (Langron and Collins 1985, p. 281) 2 . (2.7.16) R (A, B)  2s 2 χ[1/2k(k−1)] The residual quantity G t,R (A, B) can be finally partitioned into a Procrustes distance due to the application of a dilatation factor c and in a residual distance G t,R,c (A, B) following a complete similarity transformation: S (A, B) = G t,R (A, B) − G t,R,c (A, B) .

(2.7.17)

The χ 2 distribution of the various distances just considered remains valid in the case of a null hypothesis, that is, when the effects of translation, rotation, and dilatation are not separable from the remaining noise. If the belonging of the various Procrustes distances to a χ 2 distribution is to be tested, it is also necessary to know exactly the value of s 2 . Due to the difficulty to know with enough precision and reliability this value, it is more reasonable to test the rate between the Procrustes distances due to translation, rotation, and dilatation and the residual distance of a complete similarity transformation. The value so obtained does not depend on s 2 and, in the case of a null hypothesis, follows an F central distribution (ratio of two χ 2 distributions). The

2.7 Procrustes Statistics

25

rejection of the null hypothesis, once a value α for a first kind error has been fixed and for the specific degrees of freedom, shows the presence of significant transformation parameters values. In the case in which the translation component is not significant with respect to the remaining white noise, then the condition given by the following null hypothesis is satisfied: T (A, B) ≤ Fk,nk−1/2k(k+1)−1 . (2.7.18) H0 : G t,R,c (A, B) In the opposite case, for significant translations, the following alternative hypothesis is satisfied: T (A, B) H1 : > Fk,nk−1/2k(k+1)−1 . (2.7.19) G t,R,c (A, B) In a similar way, we report the null hypothesis conditions of the test when it is applied for the study of the rotation component: H0 :

R (A, B) ≤ F1/2k(k−1),nk−1/2k(k+1)−1 G t,R,c (A, B)

(2.7.20)

and of the dilation component: H0 :

S (A, B) ≤ F1,nk−1/2k(k+1)−1 . G t,R,c (A, B)

(2.7.21)

2.8 Summary In this chapter, we provided a comprehensive survey on Procrustes Analysis. More in detail, we reviewed the least squares solutions of the most common orthogonal Procrustes models, namely the so-called Orthogonal (OPA), Extended Orthogonal (EOPA), and Weighted Extended Orthogonal (WEOPA) models. They are particularly appealing from the computational point of view, for they employ only matrix products and the singular value decomposition of a k × k matrix, requiring neither the linearization of equations nor the knowledge of approximate parameters values. Procrustes Analysis can therefore be applied to solve nonlinear transformation problems that generally characterize the analytical theory behind photogrammetric computer vision and laser scanning.

References J.L. Awange, Partial procrustes solution of the three dimensional orientation problem from gps/lps observations in Quo vadis geodesia? ed. by Krumm, F., Schwarze, V.S., pp. 41–51 (1999) J.L. Awange, E.W. Grafarend, in Solving Algebraic Computational Problems in Geodesy and Geoinformatics: The Answer to Modern Challenges (Springer, 2005)

26

2 Orthogonal Procrustes Analysis

A. Beinat, F. Crosilla, Generalized procrustes analysis for size and shape 3D object reconstruction, in Optical 3-D Measurement Techniques (Wichmann Verlag, 2001), pp. 345–353 A. Beinat, F. Crosilla, Procrustes statistics to test for significant ols similarity transformation parameters, in V Hotine-Marussi Symposium on Mathematical Geodesy. (Springer, 2004), pp. 113–119 A. Beinat, F. Crosilla, D. Visintini. Examples of georeferenced data transformations in GIS and digital photogrammetry by procrustes analysis techniques. Int. Arch. Photogramm. Remote. Sens., XXXII(6W8/2), 2–5 (2000) A. Beinat. Tecniche di analisi procustiana e trasformazioni di datum in topografia e fotogrammetria. Ph.D. thesis, Dipartimento di Ingegneria Idraulica Ambientale e del Rilevamento, Politecnico of Milan (2000) M. Bennani Dosse, J.M.F. ten Berge, Anisotropic orthogonal procrustes analysis. J. Cl.Ification, 27(1), 111–128 (2010) F. Boas, The horizontal plane of the skull and the general problem of the comparison of variable forms. Science 21(544), 862–863 (1905) F.L. Bookstein, Morphometric Tools for Landmark Data: Geometry and Biology (Cambridge University Press, 1997) F.L. Bookstein, Size and shape spaces for landmark data in two dimensions. Stat. Sci. 1(2), 181–222 (1986) I. Borg, P. Groenen, Modern multidimensional scaling: theory and applications. J. Educ. Meas. 40(3), 277–280 (2003) S.N. Chiu, D. Stoyan, W.S. Kendall, J. Mecke, Stochastic Geometry and its Applications (Wiley, 2013) N. Cliff, Orthogonal rotation to congruence. Psychometrika 31(1), 33–42 (1966) T.M. Cole, Historical note: early anthropological contributions to “geometric morphometrics”. Am. J. Phys. Anthropol.: Off. Publ. Am. Assoc. Phys. Anthropol. 101(2), 291–296 (1996) J.J.F. Commandeur, Matching Configurations (DSWO Press, 1991) F. Crosilla, Procrustean transformation as a tool for the construction of a criterion matrix for control networks. Manuscripta Geodetica 8, (1983b) F. Crosilla, Procrustes analysis and geodetic sciences, in Quo vadis geodesia...?, vol. 1, pp. 69–78. Department of Geodesy and GeoInformatics, University of Stuttgart (1999) F. Crosilla, A. Beinat, A forward search method for robust generalised procrustes analysis, in Data analysis, classification and the forward search (Springer, 2006) pp. 199–208 F. Crosilla, A criterion matrix for the second order design of control networks. Bulletin géodésique 57(1–4), 226–239 (1983a) F. Crosilla, A. Beinat, Use of generalised procrustes analysis for the photogrammetric block adjustment by independent models. ISPRS J. Photogramm. & Remote. Sens. 56(3), 195–209 (2002) I.L. Dryden, K.V. Mardia, Statistical Shape Analysis (Wiley, New York, 1998) A. Fusiello, F. Crosilla, Solving bundle block adjustment by generalized anisotropic procrustes analysis. ISPRS J. Photogramm. Remote. Sens. 102, 209–221 (2015). ISSN 0924-2716 V. Garro, F. Crosilla, A. Fusiello, Solving the pnp problem with anisotropic orthogonal procrustes analysis, in Second Joint 3DIM/3DPVT Conference: 3D Imaging (Modeling, Processing, Visualization and Transmission, 2012), pp. 262–269 C.R. Goodall, Procrustes methods in the statistical analysis of shape. J. R. Stat. Society. Ser. B (Methodol.) 53(2), 285–339 (1991) J.C. Gower, Generalized procrustes analysis. Psychometrika 40(1), 33–51 (1975) J.C. Gower, G.B. Dijksterhuis, Procrustes Problems (Oxford University Press, Oxford Statistical Science Series, 2004) E.W. Grafarend, J.L. Awange, Determination of the vertical deflection by gps/lps measurements. Zeitschrift für Vermessungswesen 125(8), 279–288 (2000) E.W. Grafarend, J.L. Awange, Nonlinear analysis of the three-dimensional datum transformation [conformal group C7 (3)]. J. Geod. 77(1–2), 66–76 (2003) B.F. Green, J.C. Gower, A Problem with Congruence, in Annual meeting of the psychometric society (Monterey, California, 1979)

References

27

B.F. Green, The orthogonal approximation of an oblique structure in factor analysis. Psychometrika 17(4), 429–440 (1952) M. Gulliksson. The partial procrustes problem: a first look. Technical Report UMINF 95.11., (Department of Computing Science, Umeå University, Sweden, 1995) M.T. Heath, A.J. Laub, C.C. Paige, R.C. Ward, Computing the singular value decomposition of a product of two matrices. SIAM J. Sci. Stat. Comput. 7(4), 1147–1159 (1986) P.W. Holland, R.E. Welsch, Robust regression using iteratively reweighted least-squares. Commun. Stat.-Theory Methods 6(9), 813–827 (1977) J.R. Hurley, R.B. Cattell, The procrustes program: producing direct rotation to test a hypothesized factor structure. Behav. Sci. 7(2), 258–262 (1962) D.G. Kendall, Shape manifolds, procrustean metrics, and complex projective spaces. Bull. Lond. Math. Soc. 16(2), 81–121 (1984) K. Kenobi, I.L. Dryden, Bayesian matching of unlabeled point sets using procrustes and configuration models. Bayesian Anal. 7(3), 547–566 (2012) J.T. Kent, The complex bingham distribution and shape analysis. J. R. Stat. Society. Ser. B (Methodol.) 56(2), 285–299 (1994) M.A. Koschat, D.F. Swayne, A weighted procrustes criterion. Psychometrika 56(2), 229–239 (1991) S.P. Langron, A.J. Collins, Perturbation theory for generalized procrustes analysis. J. R. Stat. Society. Ser. B (Methodol.), pp. 277–284 (1985) R. Larsen, Functional 2d procrustes shape analysis, in Scandinavian Conference on Image Analysis (Springer, 2005), pp. 205–213 R.W. Lissitz, P.H. Schönemann, J.C. Lingoes, A solution to the weighted procrustes problem in which the transformation is in agreement with the loss function. Psychometrika 41(4), 547–550 (1976) F.T. Luk, Oblique procrustes rotations in factor analysis. SIAM J. Sci. Stat. Comput. 5(4), 764–770 (1984) K.V. Mardia, R. Edwards, M.L. Puri, Analysis of central place theory. Bull. Int. Stat. Inst. 47(2), 93–110 (1977) E. Maset, F. Crosilla, A. Fusiello, Errors-in-variables anisotropic extended orthogonal procrustes analysis. IEEE Geosci. Remote. Sens. Lett. 14(1), 57–61 (2017) C.I. Mosier, Determining a simple structure when loadings for certain tests are known. Psychometrika 4(2), 149–162 (1939) H. Park, A parallel algorithm for the unbalanced orthogonal procrustes problem. Parallel Comput. 17(8), 913–923 (1991) E.M. Phelps, A critique of the principle of the horizontal plane of the skull. Am. J. Phys. Anthropol. 17(1), 71–98 (1932) P. Schönemann, A generalized solution of the orthogonal procrustes problem. Psychometrika 31(1), 1–10 (1966) P. Schönemann, R. Carroll, Fitting one matrix to another under choice of a central dilation and a rigid motion. Psychometrika 35(2), 245–255 (1970) R. Sibson, Studies in the robustness of multidimensional scaling: perturbational analysis of classical scaling. J. R. Stat. Society. Ser. B (Methodol.), 217–229 (1979) R. Sibson, Studies in the robustness of multidimensional scaling: procrustes statistics. J. R. Stat. Society. Ser. B (Methodol.), 234–238 (1978) C.G. Small, The Statistical Theory of Shape. Springer Science & Business Media (2012) J.M.F. ten Berge, Orthogonal procrustes rotation for two or more matrices. Psychometrika 42(2), 267–276 (1977) J.M.F. ten Berge, The rigid orthogonal procrustes rotation problem. Psychometrika 71(1), 201–205 (2006) J.M.F. ten Berge, H.A.L. Kiers, J.J.F. Commandeur, Orthogonal procrustes rotation for matrices with missing values. Br. J. Math. Stat. Psychol. 46(1), 119–134 (1993) J.M.F. ten Berge, D.L. Knol, Orthogonal rotations to maximal agreement for two or more matrices of different column orders. Psychometrika 49(1), 49–55 (1984)

28

2 Orthogonal Procrustes Analysis

R. Toldo, A. Beinat, F. Crosilla, Global registration of multiple point clouds embedding the generalized procrustes analysis into an ICP framework, in International Symposium on 3D Data Processing, Visualization and Transmission, pp. 109–122 (2010) G. Wahba, A Least Squares Estimate of Satellite Attitude. SIAM Review 7(3), (1965) C. Wang, S. Mahadevan, Manifold alignment using procrustes analysis, in Proceedings of the 25th International Conference on Machine Learning (ACM, 2008), pp. 1120–1127

Chapter 3

Generalized Isotropic Procrustes Analysis

3.1 Basic Hypothesis and Layout of the Problem The main peculiarity of the problem treated in this chapter is given by the fact that m > 2 matrix configurations are simultaneously considered. All the matrices are independently and simultaneously rotated, scaled, and translated so to satisfy a prefixed objective function. To better describe the model, we can take advantage from the work of Goodall (1991), providing a geometric interpretation of the results. Let A1 , . . . , Am be m matrices, each one containing the coordinates of the same p points in Rk , defined in m different data. As represented in Fig. 3.1, each matrix Ai can be thought as obtained by a similarity transformation applied to an unknown matrix Z, containing the true coordinates of the p points defined into a common mean reference system, apart from a random error matrix Ei , i.e.,   Ai = αi Z − jti  + Ei Ti

(3.1.1)

  assuming that vec (Ei )  N 0,  = σ 2 (Q P ⊗ Q K ) . In this expression,  is the variance–covariance matrix, Q P is the cofactor matrix of the p points, Q K is the cofactor matrix of the k components, ⊗ is the Kronecker product, and σ 2 is the variance factor. In Eq. (3.1.1), also a rotation matrix Ti , a scale factor αi , and a translation vector ti appear together with the usual terms. If this relationship is valid, we can write also its inverse and suppose that the unknown matrix Z can be defined through an inverse similarity transformation applied to the known Ai matrices, except for an error matrix Ei : Z + Ei = Ai = ci Ai Ri + jti  p

(3.1.2)

where ci = αi−1 and Ri = Ti−1 . Let us initially assume an isotropic general stochastic model  = σ 2 I. The objective of the Generalized Procrustes Analysis (GPA) is to

© CISM International Centre for Mechanical Sciences 2019 F. Crosilla et al., Advanced Procrustes Analysis Models in Photogrammetric Computer Vision, CISM International Centre for Mechanical Sciences 590, https://doi.org/10.1007/978-3-030-11760-3_3

29

30

3 Generalized Isotropic Procrustes Analysis

Fig. 3.1 The generalized procrustes analysis concept

{R1, t1, c1} A1

A2

A1p A2p

Z

{R2, t2, c2} Amp Am

{Rm, tm, cm}

determine the least squares estimate of the set of unknown parameters ci , ti and, Ri (i = 1 . . . m) that minimizes the following objective function: S=

m     p p Ai − A j  i< j

= tr

m          ci Ai Ri + jti − c j A j R j + jt j  ci Ai Ri + jti − c j A j R j + jt j i< j

(3.1.3) p where Ai = ci Ai Ri + jti . The solution can be retrieved by an iterative process, applying two different alternative procedures, described in the following sections.

3.2 Original GPA Solution The original formulation and the first solution proposal of the GPA are due to Gower (1975), following a previous iterative scheme described by Kristof and Wingersky (1971). Further improvements are due to ten Berge (1977). According to this approach, the minimum condition expressed in (3.1.3) is reached by an iterative algorithm: (a) Solve for all the translations by computing the barycenter of the matrices Ai (i = 1 . . . m), multiplying them by I − jj / p; (b) Solve for the rotations applying the following iterative method, based on consecp utive OPA solutions (Sect. 2.3). At each iteration t, rotate every Ai Ri(t−1) matrix with respect to the sum of the remaining ones: Step 1: Matrix A1 R1(t−1) is rotated toward p

m

j=2

A j R(t−1) , so to obtain A1 R1(t) ; j p

Step 2: Matrix A2 R2(t−1) is rotated toward A1 R1(t) + p

A2 R2(t) ; p

p

p

m

j=3

A j R(t−1) , so to obtain j p

3.2 Original GPA Solution

31

Step 3: Matrix Am Rm(t−1) is rotated toward p

m−1

j=2

(t) A j R(t) j , so to obtain Am Rm ; p

p

Step 4: Iterate from step 1 until internal convergence. (c) Solve the scale factors, computing the correlation matrix of the vector  for p vec Ai , with f the eigenvector corresponding to the largest eigenvalue. The scale factor ci is given by: ⎧⎛ ⎫1 ⎞ m  ⎨  2 ⎬2    p  ⎠  p 2 fi ci = ⎝ A j  / Ai ⎩ ⎭

(3.2.1)

j=1

where f i is the ith element of eigenvector f. (d) Iterate from step (b) until convergence. p

Please note that, at each step of the procedure, Ai is updated, until the final value reached at convergence. At the end, it is possible to determine the so-called centroid p of the matrices, given by the arithmetic mean of all the sets Ai : m 1  p A . C= m i=1 i

(3.2.2)

The properties of the centroid will be analyzed in the following. Figure 3.2 offers a graphical description of the GPA solution and of the meaning of the centroid. In this example, there are three sets Ai , each one represented by a quadrilateral. Figure 3.2a shows the initial conditions, while Fig. 3.2b illustrates the centering effect: the sets are overlapped in correspondence of their barycenter. Figure 3.2c emphasizes a step of the rotation and scaling sequence maintaining unchanged the common barycenter until the satisfaction of the prefixed minimum condition. Figure 3.2d shows the centroid, that is the mean shape of the final transformed sets.

3.3 Alternative GPA Solution Commandeur (1991) described a different approach to retrieve the GPA solution, based on the properties of the centroid previously introduced by Formula (3.2.2). Instead of searching the solutions ci , ti , and Ri that minimize Eq. (3.1.3), he directly considers the expression of the geometric centroid in such a way to define an optimal method capable to satisfy the just mentioned minimum condition. The procedure proposed by Commandeur (1991) is based on the following proposition: Proposition 1 Given m points in a k-dimensional space, the sum of the squared distances among the m points is always equal to m times the sum of the squared distances among the m points and their centroid. That is:

32

3 Generalized Isotropic Procrustes Analysis

(b) Centering

(a) Initial configuration

(c) Rotation and scaling

(d) Centroid

Fig. 3.2 Graphical sequence of the GPA carried out with the original method

m m       xi − x j  xi − x j = m (c − xi )  (c − xi ) i< j

(3.3.1)

i

where: xi , x j represent the generic column vectors containing the coordinates of the m 1

xi is the centroid. m points and c = m i=1 Proof The first term of Formula (3.3.1) can be expressed also as: m m          xi − x j  xi − x j = xi xi + x j  x j − 2xi  x j i< j

i< j m m      = xi  x j xi xi + x j  x j − 2 i< j

= (m − 1)

i< j m  i

since:



xi xi − 2

m  i< j

xi  x j

(3.3.2)

3.3 Alternative GPA Solution

33

m m      xi xi + x j  x j = (m − 1) xi  xi i< j

(3.3.3)

i

as can be verified by induction compiling a table containing the terms corresponding to m = 1, m = 2, m = 3 . . . m = m. Developing the second term of Formula (3.3.1), it is possible to obtain: m

m 

(c − xi )  (c − xi ) = m

i

=m

m   i m 

c c + xi  xi − 2c xi m 

c c + m

i



xi  xi − 2mc

i

= m 2 c c + m

m 

m 

xj

m 

j

xi  xi − 2mc

m 

i

m 

xi  xi − 2

i m 

= (m − 1)

xi  xi + (2 − 4)

xi  xi − 2

i

m  i

i m 

xi

i

xi + m

= (1 + m − 2)

xi

i

i

=

m 

m 

xi 

m 

xi

i

xi  x j

i< j m 

xi  x j

i< j



that completes the proof.

This proposition can be extended to the case of m configurations, each one containing p points in Rk . The formulation is equivalent when applied to matrices instead of vectors. Alternatively, under a different perspective, we can imagine to apply this proposition only to the ith row of the m matrix configurations, where each row represents the same single point, expressed in different data. Repeating this operation for the remaining p − 1 points, the extendibility of the proposition is verified. According to this, there exists an extension of Proposition 1, for which: m  m 2     p  p p A − C2 Ai − A j  = m i i< j p

p

(3.3.4)

i

where Ai , A j represent the generic matrices containing the same p points and m 1

p A is the geometric centroid of the group of matrices. Formula (3.3.4) C= m i=1 i allows to consider equivalent the following measures:

34

3 Generalized Isotropic Procrustes Analysis m  m 2        p p p p p p tr Ai − A j  Ai − A j Ai − A j  = i< j

m

(3.3.5)

i< j

m m     p  p   p  A − C2 = m tr Ai − C  Ai − C . i i

(3.3.6)

i

Replacing Formula (3.3.5) with the Expression (3.3.6) inside the objective function S given by (3.1.3), it is possible to define an alternative solving criterion to the GPA problem. In this case, it is necessary to iteratively compute the centroid C till the satisfaction of the minimum condition given by (3.3.6). To summarize, the various computation phases require: (a) The iterative solution of independent similarity transformations for each matrix with respect to the centroid C, by using formulas of the EOPA process (Sect. 2.4); (b) The iterative updating of the centroid C by a simple arithmetic mean following p the sequential updating of matrices Ai ; (c) The iteration of the process till convergence, that is a stabilization of the geometric configuration of the centroid C. This strategy leads to great advantages in terms of simplicity and efficiency.

3.4 Weighted GPA with Missing Values The comparison and the multiple and simultaneous adaptation of m point configurations can be afforded also in the case when one or more points are missing in some configurations. The problem formulation was completely solved by Commandeur (1991). Starting from this approach, in the following, it is shown how this method can be extended in such a way to manage more general situations of weighted matrix configurations by rows, where the missing data represent a peculiarity. The new tool is based on the following theorem: p

Proposition 2 The generalization of Proposition 1 for m configurations Ai premultiplied by a diagonal matrix Di is given by:  m −1 m m        p   p   p p  p p Ai − A j tr Ai − C Di Ai − C = tr Ai − A j Di D j Dk i=1

i< j

k=1

(3.4.1) where: C=

 m  k=1

−1 Dk

m 

p

Di Ai .

i=1

Proof Let us first expand the right term of Eq. (3.4.1):

(3.4.2)

3.4 Weighted GPA with Missing Values

35

 m −1 m       p p  p p Ai − A j tr Ai − A j Di D j Dk i< j

=

m 

p tr(Ai ) Di D j

i< j

−

m 

 m 

k=1

−1

Dk

k=1

p tr(A j ) Di D j

 m 

i< j

p Ai

+

 m 

p tr(A j ) Di D j

i< j

−1 Dk

m 

p Ai

−

k=1

m 

−1

k=1

 m 

p tr(Ai ) Di D j

i< j

p

Dk

Aj

−1 p

Aj .

Dk

k=1

Exploiting the trace properties for which: tr(AB) = tr(BA) and tr(A ) = tr(A) and remembering that: m 

tr D j Dk +

m 

j 2. The matches that correspond to already reconstructed points are used to infer correspondences between 2D and 3D. Based on these, the projection matrix Pi is computed with exterior orientation. The structure is updated using intersection (triangulation) by (i) refining the position of 3D points that have been observed in the current image and by (ii) initializing a new 3D point when point matches become available that were not related to an existing point. Frequent bundle adjustment is needed in practice to contain error accumulation.

6.3.3 Independent Models Block Adjustment This is a classical method in photogrammetry (Kraus 2007). • First stereomodels are created, by relative orientation. Each of these models is described in a local, arbitrary reference frame. 1 We

assume that even if images are unordered they can be suitably sequenced.

84

6 Basics of Computer Vision

• In the course of the block adjustment, the individual models will be amalgamated into a single one and simultaneously transformed into the ground coordinate system. We will describe in Chap. 9 a Procrustes approach, which by its nature produces a “free” solution (i.e., expressed in an arbitrary reference frame), but can be modified to include GCPs. A similar Procrustes approach will be also introduced in Chap. 8 to compute bundle block adjustment.

6.4 Summary This chapter has introduced a number of basic geometric concepts of multi-view geometry in computer vision and photogrammetry. The key actor in two-view geometry is the essential matrix. It captures the epipolar geometry algebraically, and constraints the position of corresponding points to known lines. Reconstruction of 3D structure and motion from multiple overlapping images is instead possible through the use of various block processing approaches, some of which can be solved also via Procrustes Analysis, as will be shown in the following chapters.

References A. Ansar, K. Daniilidis, Linear pose estimation from points or lines. IEEE Trans. Pattern Anal. Mach. Intell. 25(5), 578–589 (2003) M. Arie-Nachimson, S.Z. Kovalsky, I. Kemelmacher-Shlizerman, A. Singer, R. Basri, Global motion estimation from point matches, in Proceedings of the Joint 3DIM/3DPVT Conference: 3D Imaging, Modeling, Processing, Visualization and Transmission (2012) P. Beardsley, A. Zisserman, D. Murray, Sequential update of projective and affine structure from motion. Int. J. Comput. Vis. 23(3), 235–259 (1997) N. Börlin, P. Grussenmeyer, Bundle adjustment with and without damping. Photogram. Rec. 28(144), 396–415 (2013) M. Brown, D.G. Lowe, Unsupervised 3D object recognition and reconstruction in unordered datasets, in Proceedings of the International Conference on 3-D Digital Imaging and Modeling (June 2005) F. Crosilla, A. Beinat, Use of generalised procrustes analysis for the photogrammetric block adjustment by independent models. ISPRS J. Photogrammetry Remote Sens. 56(3), 195–209 (2002) O. Enqvist, F. Kahl, C. Olsson, Non-sequential structure from motion, in Eleventh Workshop on Omnidirectional Vision, Camera Networks and Non-classical Camera (2011) O. Faugeras, Three-Dimensional Computer Vision: A Geometric Viewpoint (The MIT Press, Cambridge, 1993) O. Faugeras, Q.-T. Luong, The geometry of multiple images (MIT Press, 2001) P.D. Fiore, Efficient linear solution of exterior orientation. IEEE Trans. Pattern Anal. Mach. Intell. 23(2), 140–148 (2001) M.A. Fischler, R.C. Bolles, Random Sample Consensus: a paradigm model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

References

85

X.-S. Gao, X.-R. Hou, J. Tang, H.-F. Cheng, Complete solution classification for the perspectivethree-point problem. IEEE Trans. Pattern Anal. Mach. Intell. 25, 930–943 (2003) R. Gherardi, M. Farenzena, A. Fusiello, Improving the efficiency of hierarchical structure-andmotion, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 13–18 June 2010, pp. 1594–1600 V.M. Govindu, Combining two-view constraints for motion estimation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2001) R.I. Hartley, P. Sturm, Triangulation. Comput. Vis. Image Underst. 68(2), 146–157 (1997) R.I. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision, 2nd edn. (Cambridge University Press, Cambridge, 2003) J.A. Hesch, S.I. Roumeliotis, A direct least-squares (dls) solution for PnP, in Proceedings of the International Conference on Computer Vision (2011) T.S. Huang, O.D. Faugeras, Some properties of the E matrix in two-view motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 11(12), 1310–1312 (1989) A. Irschara, C. Zach, H. Bischof, Towards wiki-based dense city modeling, in Proceedings of the 11th International Conference on Computer Vision (2007), pp. 1–8 F. Kahl, R.I. Hartley, Multiple-view geometry under the l∞ -norm. IEEE Trans. Pattern Anal. Mach. Intell. 30(9), 1603–1617 (2008) K. Kanatani, H. Niitsuma, Optimal computation of 3-d similarity: Gauss Newton vs. Gauss Helmert. Comput. Stat. Data Anal. 56(12), 4470–4483 (2012) K. Kraus, Photogrammetry: Advanced Methods and Applications, vol. 2 (Dümmler, 1997). ISBN 9783427786948 K. Kraus, Photogrammetry—Geometry from Images and Laser Scans, 2nd edn. (Walter de Gruyter, 2007). ISBN 978-3-11-019007-6 V. Lepetit, F. Moreno-Noguer, P. Fua, EPnP: an accurate O(n) solution to the PnP problem. Int. J. Comput. Vis. 81(2), 155–166 (2009) D.G. Lowe, Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004). ISSN 0920-5691 J.R. Magnus, H. Neudecker. Matrix Differential Calculus with Applications in Statistics and Econometrics, revised edition (Wiley, 1999) D. Martinec, T. Pajdla, Robust rotation and translation estimation in multiview reconstruction, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2007) P. Moulon, P. Monasse, R. Marlet, Global fusion of relative motions for robust, accurate and scalable structure from motion, in Proceedings of the International Conference on Computer Vision (2013), pp. 3248–3255 K. Ni, F. Dellaert, Hypersfm, in Proceedings of the Joint 3DIM/3DPVT Conference: 3D Imaging, Modeling, Processing, Visualization and Transmission (2012), pp. 144–151 N. Snavely, S.M. Seitz, R. Szeliski, Photo tourism: exploring photo collections in 3D, in SIGGRAPH: International Conference on Computer Graphics and Interactive Techniques (2006), pp. 835–846 B. Triggs, P.F. McLauchlan, R.I. Hartley, A.W. Fitzgibbon, Bundle adjustment—a modern synthesis, in Proceedings of the International Workshop on Vision Algorithms (Springer, Berlin, 2000), pp. 298–372. ISBN 3-540-67973-1 M. Vergauwen, L. Van Gool, Web-based 3D reconstruction service. Mach. Vis. Appl. 17(6), 411– 426 (2006)

Chapter 7

Basics of Terrestrial Laser Scanning

This chapter introduces the basic concepts of the automatic 3D surveying by means of laser scanning techniques, a methodology developed less than 20 years ago (see, e.g., Boehler et al. 2002) thanks to the technological improvement of the topographical instrumentations, in particular in the movement and in the electronical components. The main topic here explained concerns the geometrical measurement principle of a 3D scene by means of different automatic scans, so performing different 3D point clouds, and the analytical steps to refer such scans to the one chosen coordinate system. Laser scanning technology described in this chapter considers terrestrial static systems, anyway in the last part, some hints are reported about scans acquired by moving systems, as those mounted on aerial vehicles or on portable devices.

7.1 Background First of all, just to introduce this technique of surveying, let us specify that the similar definitions Terrestrial Laser Scanner and Terrestrial Laser Scanning, both corresponding to the TLS acronym, have a different meaning: the scanner is the instrument allowing the surveying by scanning, although roughly speaking the concepts are analogous. A first classification dealing with TLS technique is the belonging of these systems to the active sensors, namely instrumentations emitting radiations in a certain direction, obviously toward the investigated object, and detecting the radiation that is reflected or backscattered from it. Passive sensors, as the CCD/CMOS cameras acquiring images, detect instead (generally) natural radiation reflected by the object being observed: of course, the reflected sunlight is the most common radiation measured by passive sensors. First prototypes of TLS instrumentations have been applied for industrial and civil engineering investigations, while nowadays they are largely used also for the surveying of architectural, archeological and cultural heritage sites, namely working © CISM International Centre for Mechanical Sciences 2019 F. Crosilla et al., Advanced Procrustes Analysis Models in Photogrammetric Computer Vision, CISM International Centre for Mechanical Sciences 590, https://doi.org/10.1007/978-3-030-11760-3_7

87

88

7 Basics of Terrestrial Laser Scanning

situations with noteworthy geometrical irregularity, where such systems warrant a high level of accuracy and detail (e.g., Gonizzi Barsanti et al. 2013; Balletti and Guerra 2015; Tucci and Bonora 2015). One of the most complete book on TLS but also on Aerial/Airborne Laser Scanning (ALS) was edited by Vosselman and Maas (2010). Referring to the measurement principle, TLS systems can be divided into two main families: light transit time estimation (or laser range finding) and triangulation based. In this chapter, the focus is on the former kind of scanners, also called distance meter systems, since they are more used in cases where images and computer vision techniques are also applied. Briefly, triangulation-based active sensors project a moving laser line or a fixed pattern onto the surveyed object: the angular direction of any single line or of all the grid are then known. An imaging sensor, located at a calibrated distance (base of the triangle) records the reflected signals: from the two image coordinates, the corresponding angles adjacent to the base are derived and, by simple trigonometric relations, the 3D position of the projected points is so computed. Considering a distance meter TLS, as well as for any electronic distance meter (EDM) topographic system, the speed and the form of the traveling light waves are known: by considering the time of the light travel source—reflective surface— source, it is possible then to evaluate the distance 2d. In literature, these systems are known as Time-of-Flight (TOF) or Light Detection and Ranging (LiDAR) systems. Anyway, the distance d is computed using the two classical methods applied in topographical surveying: the Time-of-Flight of the transmitted–reflected signal or the phase difference of such signal. In the former case, the systems are classified as Pulsed Wave (PW) scanners, while in the latter as Continuous Wave (CW) scanners; more details can be found in Pfeifer and Briese (2007). TLS systems can vary a lot in terms of shape, weight, field, and direction of scan, acquisition speed and, obviously, measure range and surveying accuracy. Of course, the continuous hardware development pushes toward always smaller, lighter, faster, and more accurate systems, anyway maximum range and accuracy are yet opposite properties. Figure 7.1 (left) shows the geometry of the scanning process: the laser scanner is fixed over a tripod in a certain “scan position,” again as a topographic total station. In this case, the direction of the Z S -axis is appositely turned to scan almost horizontal surfaces as an archeological excavation area is. The adopted coordinate reference frame X S, Y S, Z S has origin in the TLS system center and, for this reason, is called Scanner’s Own Coordinate System (SOCS). Such system is thus a local and arbitrary reference frame and, aware that many scans are needed to adequately survey whatever site, this will require the computations later described in Sect. 7.2. From the instrumental point of view, the fundamental aspect is the capability, fulfilled by all TLS systems, to direct a laser beam in whatever surrounding direction, generally thanks to a mirror quickly rotated by a servomotor.

7.1 Background

89

Fig. 7.1 Left: a laser scanner surveying of an archeological area with the X S, Y S, Z S (SOCS) reference frame centered on the scanner. Right: the obtained 3D point cloud

Consider such laser beam in a generic spatial direction: the coordinates of the hit point are given by the following equation: ⎞ ⎛ ⎞ sin ϕ sin ϑ XS r S = ⎝ Y S ⎠ = d ⎝sin ϕ cos ϑ⎠ cos ϕ ZS ⎛

(7.1.1)

where: • d is the distance from the TLS instrument center to the hit point, computed as explained before; • ϕ is the “vertical” angle of the ray direction, known since imposed by the rotation of the mirror around the“horizontal” rotating axis; • ϑ is the “horizontal” angle of the ray direction, also this known since imposed by the rotation of the mirror around the“vertical” Z S -axis. From the mathematical point of view, Eq. (7.1.1) expresses the (bi-univocal) transformation from spherical coordinates d, ϕ, ϑ (nearly the same measures acquired by a topographic total station), to Cartesian coordinates X S, Y S, Z S . Coming back to the TLS surveying, the scanning effect is achieved quickly reflecting (nowadays up to one million times per second!) the emitted laser beam in different directions by changing the mirror deflection angle of small increments. For each point, the value of d is measured, while those of ϕ and ϑ are imposed: by continuously applying Eq. (7.1.1), the corresponding X S, Y S, Z S coordinates are then automatically computed in quasi-real-time. High accuracy in recording the angular values is truly important, since it settles on the precision of the obtained X S, Y S, Z S coordinate point: in general, the error in the 3D point positioning is restricted to a few millimeters.

90

7 Basics of Terrestrial Laser Scanning

The strong similarity with terrestrial topographic instruments allows to consider the TLS technology as the “evolution toward automation” of the terrestrial topographic total stations, whose motorized models (standard nowadays) already make it possible an automatic collimation along an angular grid, although extremely slow. TLS data acquisition is hence fully automatic: once fixed the angular increments for ϕ and ϑ, and possibly the minimum and maximum values of ϕ and ϑ within the instrumental operative field of scan, the acquisition of millions of points from each position is automatically carried out in few minutes, also for panoramic scans, i.e., when ϑ ranges from 0◦ to 360◦ . Furthermore, a fourth value for each beam is acquired: the intensity I of the returned laser signal. In this way, it is possible to map the materials of the scanned surface that can be represented, e.g., by means of values in a gray scale (as in Fig. 7.1 (right)). Some TLS prototypes can also measure the returned intensity in the spectral band of the RGB colors, but most TLS systems are integrated instead with an internal/external photogrammetric camera (see, the camera mounted over the TLS head in Fig. 7.1 (left)). The only manual operation to carry out is a uniform positioning of suitable targets, for instance, cylindrical or adhesive flat reflecting targets, allowing to simplify the fundamental step of scans registration described in Sect. 7.2. Exploiting reflecting targets, these are automatically detected among the point cloud already in situ thanks to their very high reflectivity, as visible in Fig. 7.1 (right) for the cylinder onto the bricks structure. For each target, a fine scan of tens of thousands beams is thus quickly performed, so to accurately estimate its barycenter position. Regarding the image acquisition, this is achieved straight after the scans, by rotating the photogrammetric camera around the Z S -axis by a suitable step of ϑ, so to obtain a “horizontal” field of view covering the “horizontal” field of scan. The “vertical” overlap of the images onto the scan is about 90% by using an external camera with wide-angle objectives, less with normal focal length lens. For each image acquired, the position and attitude of the camera reference frame with respect to the SOCS one can be considered known (after a suitable calibration), since depending only from the value of the imposed angle ϑ. In any case, CCD images of TLS systems are exploited only for the RGB coloring of the point clouds and not to photogrammetrically measure the depicted objects.

7.2 Registration and Geo-referencing of Laser Scanning Point Clouds As already mentioned, acquiring TLS data from different points of scanning, i.e., different SOCSs, originates the problem, known as registration of the scans, to merge such independent clouds of points. Sometimes, this process is called alignment. Considering Fig. 7.2, the problem is that, applying Eq. (7.1.1) for the same point P, the coordinates X S1, Y S1, Z S1 with respect to SOCS1 are clearly different from

7.2 Registration and Geo-referencing of Laser Scanning Point Clouds

91

Fig. 7.2 Registration problem: the same point P has different SOCS coordinates

those X S2, Y S2, Z S2 with respect to SOCS2 . Obviously, this fact is much more boring dealing with any other further ith scan and relating SOCSi . In other words, the point scans/clouds have to be suitably referred to a unique common reference frame, called Project Coordinate System (PCS). In general, the origin of this frame is in the center of the surveyed area and with the Z P -axis exactly vertical. Sometimes, a specific vertical scan position, the more central one, can be chosen as PCS. Referring again to Fig. 7.2 for sake of simplicity, considering SOCS1 as the final absolute reference, i.e., the PCS, and SOCS2 as the initial local reference, i.e., any SOCSi , the analytical problem is described by the following relationship: ⎛

⎞ ⎛ P⎞ ⎛ Si ⎞ XP X Si X r P = ⎝ Y P ⎠ = ⎝ Y SiP ⎠ + R (, , K ) ⎝ Y Si ⎠ P ZP Z Si Z Si

(7.2.1)

where: • X P, Y P, Z P are the coordinates of point P with respect to PCS; P P , Y SiP , Z Si are the unknown SOCSi origin coordinates with respect to PCS; • X Si • R is the 3×3 matrix of the unknown rotation from SOCSi to PCS, defined by three rotation angles , , K around the three principal directions; • X Si, Y Si, Z Si are the coordinates of point P with respect to SOCSi . Relationship (7.2.1) states that, after the registration transformation, the coordinates with respect to SOCSi (finally) correspond to the coordinates with respect to PCS, apart from the observation errors. From the geometrical point of view, the registration problem is very simple, involving a similarity transformation (rotation and translation) but without scale variation (namely a rigid roto-translation). As usual in geomatics, the six unknowns are computed by solving a system with a redundant number of equations (7.2.1) on the same unknowns and by performing a least squares estimation. From the computational point of view, the registration requires the “double” 3D coordinates of at least two points in the two reference frames PCS and SOCSi , for this reason called double points or tie-points. Please note that this problem can be solved with the Extended Orthogonal Procrustes Analysis (EOPA), extensively described in Sect. 2.4.

92

7 Basics of Terrestrial Laser Scanning

In principle, in the overlapped zone between two point clouds, millions of quasicommon points are available, with the word “quasi” remembering us that the scanned point is never exactly the same in the different scans. Hence, theoretically millions of double points can be exploited to analytically solve the registration, but with the titanic problem of the correct pairing of the “same” point among the scans. All these troubles can be completely skipped by exploiting the reflecting targets as tiepoints, considering their barycentric SOCS coordinates, and automatically finding the correct point pairing by choosing, among all the possible combinations, the one producing the minimum error (sum of the squares of the residuals). Some details about this point pairing can be found in Beinat et al. (2007). The accuracy of the obtained registration can be easily evaluated by considering the residuals on the tie-points. Following the same approach for each couple of scans and/or considering the common targets among more than two scans, three translation and three rotation parameters per scan are therefore computed. By applying these transformations to each point of each cloud, they “move” from their SOCS frame to a common PCS frame: as a consequence, a unique cloud of millions of points is finally obtained. A possible procedure to solve the registration problem is the well-known Iterative Closest Point (ICP) algorithm (Chen and Medioni 1992; Besl and McKay 1992), already described in Chap. 3. Just to recall the basic idea, the estimation of the six transformation parameters is iteratively carried out by means of a least squares minimization of a “metric error”, usually the distance, between the overlapping surfaces. Hence, it is required to compute the surfaces generated from two or more point clouds (in truth, randomly considering a subsample of such point clouds). In general, wide rotations/translations are first computed by model (7.2.1) and then refined rotations/translations are later estimated via the ICP algorithm. In the former case, the residuals are minimized onto few tie-points, in the latter for all the overlapping surfaces. Once all the scans have been registered, another topographic problem arises: the just obtained unique PCS point cloud should be referred to a Global Coordinate System (GCS), e.g., a cartographic reference frame, thus requiring the so-called georeferencing of the laser scanning data. The problem to transform X P, Y P, Z P coordinates into X G, Y G, Z G ones can be analytically solved by means of a direct or indirect method of geo-referencing (see, e.g., Schuhmacher and Böhm 2005). In the first case, in truth not much applied, the TLS system is integrated with a Global Navigation Satellite System (GNSS) receiver and/or topographic devices allowing to directly measure the transformation parameters from SOCSi to GCS frame, hence bypassing the registration step. Following instead the indirect method, some scanned point (e.g., the same reflecting targets) constitutes a set of control points surveyed by means of a topographic total station from the vertices of a topographic network defined into a geodetic datum. Nowadays, the novelty in this approach is the fixing of the network geodetic datum by GNSS measurements on some vertices.

7.2 Registration and Geo-referencing of Laser Scanning Point Clouds

93

From the analytical point of view, the indirect geo-referencing problem is described by a relationship very similar to the previous (7.2.1): ⎞ ⎛ G⎞ ⎛ P⎞ XP X XG r G = ⎝ Y G ⎠ = ⎝ Y PG ⎠ + R () ⎝ Y P ⎠ ZG ZP Z GP ⎛

(7.2.2)

where: • X G, Y G, Z G are the coordinates of point P with respect to GCS; • X GP , Y PG , Z GP are the unknown PCS origin coordinates with respect to GCS; • R () is the 3×3 matrix of the unknown rotation from PCS to GCS, defined by only one rotation angle  since Z P -axis and Z G -axis are both vertical. Also the least squares solution of relationship (7.2.2) follows the same strategy of the previous registration step (7.2.1), where now the double points are given from the control points, with topographically measured X G, Y G, Z G coordinates and registered X P, Y P, Z P ones. From a geometrical point of view, the availability of points of known position, realistically with a good accuracy and a uniform positioning, makes it possible to locally constrain the various clouds on “true 3D positions”. In this sense, by means of the registration step, tie-points are only joined together while, through the geo-referencing step, such tie-points are constrained to coincide with the topographically measured positions. As a result, the total point cloud is locally deformed, avoiding the effects of errors propagation of the registration and, consequently, improving the surveying accuracy. Summarizing, scans registration and geo-referencing both involve the estimation of translation and rotation parameters and, for such reason, can be simultaneously solved. It is important to underline how these transformation problems, involving nonlinear estimations and sets of coordinates of double points in two data, can be very efficiently solved thanks to the algorithms from Procrustes Analysis reported throughout this book. As a last consideration, these TLS steps are analogous to the photogrammetric problems (see Chap. 6) of the relative orientation between two images with the construction of a stereoscopic model (registration) and of the absolute orientation of the so-created stereoscopic model (geo-referencing). Once the whole point cloud has been geo-referenced, the laser scanning data processing can be considered accomplished: the subsequent steps of surface modeling by mesh, surface optimization, and image texturing are similar to those applied to 3D point clouds obtained from images via the Structure from Motion approach (Hartley and Zisserman 2003).

94

7 Basics of Terrestrial Laser Scanning

7.3 Laser Scanning with Moving Systems Positioning (7.1.1), registration (7.2.1) and geo-referencing (7.2.2) models are based on the assumption that the terrestrial laser scanner is alternatively static on various scan positions and well-distributed tie-points are available. More complex scanning procedures can be carried out by using scanners mounted on moving vehicles such as aircrafts and helicopters, but nowadays also on Unmanned Aerial Vehicles (UAVs), vans, as well as very recent “portable by man” devices. All these kinds of systems belong to the Mobile Mapping Systems (MMS) category, a multisensors technology introduced more than 20 years ago (ASPRS 1995; Schwarz and El-Sheimy 1996) to survey roads, mainly exploiting, at that time, photogrammetric imaging sensors. Nowadays, systems are instead basically equipped with a laser scanner for measuring, while as navigation sensors a GNSS receiver and an Inertial Navigation System (INS) are employed. In few words, GNSS/INS navigation sensors enable the estimation of the position and the attitude of the moving vehicle, and then of the TLS, at every instant. Considering Fig. 7.3, now the position r G of a generic point with respect to the GCS (PCS is no more exploited) is given by the more complex equation below:  ins tls  G G ins + Rins Rtls r + atls r G = rgnss/ins

(7.3.1)

where: G • rgnss/ins is the position of the GNSS/INS sensors, sometimes embedded, with respect to GCS, suitably computed;

Fig. 7.3 Model of geo-referencing of a laser scanning moving systems

7.3 Laser Scanning with Moving Systems

95

G • Rins is the 3 × 3 matrix of the rotation from the INS frame to the GCS frame, with the instantaneous rotation angles measured by the three INS gyroscopes; ins is the 3 × 3 matrix of the rotation from the TLS frame to the INS frame, with • Rtls fixed angles suitably estimated in the system calibration procedure; • rtls is the position of the surveyed point with respect to the TLS frame (analogous to r S of Eq. (7.1.1)); ins is the position of the TLS frame with respect to the INS frame, with fixed • atls components generally of few centimeters, nevertheless also this to be suitably measured/estimated in the system calibration step.

The kinematic positioning of every scanned point by means of Eq. (7.3.1) can be reinterpreted as the static positioning rtls by (7.1.1) from a moving TLS system whose G G G ins = rgnss/ins + Rins atls , while the instantaneous rotation instantaneous position is rtls G G ins is Rtls = Rins Rtls . In other words, the six roto-translation registration parameters (in principle for each laser beam since TLS is moving!) are directly estimated by means of GNSS/INS measurements and not indirectly by using tie-points as in (7.2.1). The most common application of model (7.3.1) is in the Aerial Laser Scanning (ALS) (see, e.g., Baltsavias 1999), widely exploited for the territorial surveying: although rtls could be kilometric and the aircraft/helicopter speed tens of km/h, the accuracy of the positioning r G is surprising, generally limiting the errors under 10 cm. Concluding this chapter, some words are devoted to the last (for now!) frontier of the TLS technology represented by the portable systems, developed very recently and variously called as human, handheld, backpack, trolley systems. From the technological point of view, also these multisensor systems have one or more scanning (measuring) sensors and one or more navigation sensors, although with generally lower performances. To underline the terrestrial but moving features of this technology, the term Indoor Mobile Mapping Systems encloses all such instrumentations (Thomson et al. 2013; Nocerino et al. 2017; Lehtola et al. 2017; Tucci et al. 2018), where the word indoor remember us that GNSS data could be not available for a part or all the surveying area. By applying model (7.3.1), with some systems already in real time, the positioning r G is clearly affected by the propagation of scanning and navigation errors. Anyway, this “first-obtained” point cloud is exploited to re-compute the TLS positioning and orientation. The analytical process making possible such improvement is very complex and comes from algorithms developed in robotics (Nüchter et al. 2007). Summarizing, the fundamental step is a suitable subdivision of the whole point cloud into a lot of n “virtual single static scans”, each one considering a certain TLS localization. The obtained clouds have geometrical discrepancies, due to errors: nevertheless, these are minimized by analytical procedures generally based again on ICP algorithm, i.e., by suitably reducing the distance among the surfaces of these n clouds.

96

7 Basics of Terrestrial Laser Scanning

7.4 Summary From the geometrical point of view, the TLS measurement principle and the consequent generation of 3D point cloud are very easy to understand, since they simply involve a transformation of topographically observed spherical coordinates into Cartesian coordinates referred to the TLS center. In spite of this simplification, since the TLS system has to be moved into a second, third and so on scan position, any new point cloud is referred to a moving and local reference frame: hence, the registration of all the scans to a single chosen project coordinate system is a fundamental step. More complete results are obtained if the chosen coordinate system is a global/geographic reference frame and, for this aim, coordinates acquired by EDM topographic total stations and GNSS receivers are mandatory to solve the geo-referencing step. Involving rigid rotations and translations on point sets, registration, and geo-referencing transformations can be conveniently solved by means of Procrustes Analysis, as it will be shown in Part III of this book. To conclude this chapter, some aspects of the new trends in laser scanning technology were briefly described, just to underline the potentialities of the nowadays new moving systems, exploiting also navigations sensors to solve the geo-referencing problem.

References ASPRS. 1995 mobile mapping symposium, in Proceedings of the First International Workshop on Mobile Mapping Technology, Columbus, OH, USA (May 1995) C. Balletti, F. Guerra, The survey of cultural heritage: a long story. Rend. Lincei 26(1), 115–125 (2015) E.P. Baltsavias, Airborne laser scanning: basic relations and formulas. ISPRS J. Photogramm. Remote Sens. 54(2), 199–214 (1999) A. Beinat, F. Crosilla, D. Visintini, F. Sepic, Automatic non parametric procedures for terrestrial laser point clouds processing. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 36(3), W49B (2007) P. Besl, N. McKay, A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992) W. Boehler, G. Heinz, A. Marbs, The potential of non-contact close range laser scanners for cultural heritage recording. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 34(5/C7), 430–436 (2002) Y. Chen, G. Medioni, Object modeling by registration of multiple range images. Image Vis. Comput. 10(3), 145–155 (1992) S. Gonizzi Barsanti, F. Remondino, D. Visintini, 3d surveying and modelling of archaeological sites—some critical issues. ISPRS Photogramm. Remote Sens. Spat. Inf. Sci. 2(5/W1):145–150 (2013) R.I. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision, 2nd edn. (Cambridge University Press, Cambridge, 2003) V.V. Lehtola, H. Kaartinen, A. Nüchter, R. Kaijaluoto, A. Kukko, P. Litkey, E. Honkavaara, T. Rosnell, M.T. Vaaja, J.-P. Virtanen, M. Kurkelaand, A. El-Issaoui, L. Zhu, A. Jaakkola, J.

References

97

Hyyppa, Comparison of the selected state-of-the-art 3d indoor scanning and point cloud generation methods. Remote Sens. 9(8), 796 (2017) E. Nocerino, F. Menna, F. Remondino, I. Toschi, P. Rodríguez-Gonzálvez, Investigation of indoor and outdoor performance of two portable mobile mapping systems, in Videometrics, Range Imaging, and Applications XIV, vol. 1033 (International Society for Optics and Photonics, 2017), p. 103320I A. Nüchter, K. Lingemann, J. Hertzberg, H. Surmann, 6d slam—3d mapping outdoor environments. J. Field Robot. 24(8–9), 699–722 (2007) N. Pfeifer, C. Briese, Laser scanning–principles and applications, in GeoSiberia 2007— International Exhibition and Scientific Congress (Citeseer, 2007) S. Schuhmacher, J. Böhm, Georeferencing of terrestrial laserscanner data for applications in architectural modeling, in International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, Proceedings of the ISPRS Working Group V/4 Workshop 3D-ARCH 2005, vol. 36 (2005) K.P. Schwarz, N. El-Sheimy, Kinematic multi-sensor systems for close range digital imaging. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 31, 774–785 (1996) C. Thomson, G. Apostolopoulos, D. Backes, J. Boehm, Mobile laser scanning for indoor modelling. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2(5/W2), 289–293 (2013) G. Tucci, V. Bonora, Geomatics and management of at-risk cultural heritage. Rend. Lincei 26(1), 105–114 (2015) G. Tucci, D. Visintini, V. Bonora, E.I. Parisi, Examination of indoor mobile mapping systems in a diversified internal/external test field. Appl. Sci. 8(3), 401–430 (2018) G. Vosselman, H.-G. Maas, Airborne and Terrestrial Laser Scanning (Whittles Publishing, 2010)

Part III

Applications of Procrustes Analysis Models

Chapter 8

Applications of Anisotropic Procrustes Analysis

As extensively shown in the previous chapters, Procrustes Analysis allows to easily perform transformations among corresponding point coordinates belonging to a generic k-dimensional space and it is therefore suited to solve problems encountered in geodesy, photogrammetric computer vision and laser scanning. In particular, the anisotropic row-scaling variant of the Procrustes Analysis, described in Chap. 4, represents the basic tool for an alternative solution of classical photogrammetric problems. The analytical solution of the Anisotropic Extended Orthogonal Procrustes Analysis (AEOPA) with row scaling was first derived by Gower and Dijksterhuis (2004) and subsequently applied by Garro et al. (2012) to perform the exterior orientation of one image without any a priori information. Fusiello et al. (2013) proposed a robust version of this algorithm, based on Least Median of Squares to detect a minimal outliers-free sample, and a Forward Search procedure, used to augment the inliers set one sample at a time. In (Fusiello et al. 2015) the point-line registration problem, which generalizes absolute orientation to point-line matching, was formulated as an instance of the AEOPA model, and its solution was derived. The same formulation solves the Non-Perspective-n-Point camera pose problem, that in turn generalizes exterior orientation to non-central cameras, i.e., generalized cameras where projection rays do not meet in a single point. The generalized version of AEOPA, known as AGPA and first derived by Commandeur (1991), leads instead to the Procrustes solution of the classical bundle block adjustment (Fusiello and Crosilla 2015), also in its robust version (Fusiello and Crosilla 2016). In this chapter, we will review the aforementioned Procrustes solutions, together with their experimental validation. Experiments on both synthetic and real datasets show that Procrustes algorithms lead to results that are comparable in terms of accuracy with those given by the state-of-the-art methods. Co-authored with Valeria Garro, University of Verona (Italy) and Francesco Malapelle, University of Udine (Italy). © CISM International Centre for Mechanical Sciences 2019 F. Crosilla et al., Advanced Procrustes Analysis Models in Photogrammetric Computer Vision, CISM International Centre for Mechanical Sciences 590, https://doi.org/10.1007/978-3-030-11760-3_8

101

102

8 Applications of Anisotropic Procrustes Analysis

8.1 Solving the Exterior Orientation Problem with AEOPA In Sect. 6.2.4, we introduced the problem of estimating the position and attitude of a perspective camera given its interior orientation and a set of n world-to-image correspondences, known as the exterior orientation problem in photogrammetry or Perspective-n-Point camera pose problem (PnP) in computer vision. In both communities, it has received much attention, being one of the building blocks for many applications like structure from motion, object/camera localization, robotics, just to mention a few. An extensive literature exists on this topic, which will be summarized in Sect. 8.1.1. The trade-off that all the methods face is between speed and accuracy. Direct methods are usually faster but less accurate, as they do not minimize a significant cost function, whereas iterative methods, that explicitly minimize a meaningful geometric error, are more accurate but slower. On the theoretical side, the contribution of the following sections, adapted from (Garro et al. 2012), lies in casting the PnP problem as an instance of the Anisotropic Extended Orthogonal Procrustes (AEOPA) with row scaling, extensively described in Sect. 4.2. We will show that this method reaches a good trade-off between speed and accuracy and is extremely easy to implement.

8.1.1 Background and Related Work In the literature, several solutions for the PnP problem have been presented. They can be classified along several dimensions, the most relevant being: iterative/closed form, minimal (n = 3 or 4)/general (∀n ≥ 3), and linear/nonlinear solution. Iterative approaches (Lowe 1991; Haralick et al. 1989; Kumar and Hanson 1994; Horaud et al. 1997) generally formulate the pose estimation as a nonlinear least squares problem minimizing a cost function related to a geometric (e.g., re-projection residual) or algebraic error (Hartley 1998). They usually achieve better accuracy compared to the closed-form methods at a price of a significant computational load. They generally need a good initial guess in order to converge properly to the correct solution. In (Lu et al. 2000), the authors minimize an error based on collinearity in 3D space; they successively improve an estimate of the rotation matrix enforcing the orthogonal constraint and then compute the translation vector. The method proposed in (DeMenthon and Davis 1995) (POSIT) applies iteratively a linear closed-form solver for a scaled orthographic projection camera. It alternates between computing the (orthographic) pose and computing correction factors that are directly connected to the points depth. The final perspective pose is obtained from a sequence of scaled orthographic approximations. More recently, Schweighofer and Pinz (2008) presented an algorithm that transforms in linear time the minimization problem into a semi-definite positive program (SDP) and than solves it using SeDuMi (Sturm 2001), an iterative solver that yields a global optimal solution.

8.1 Solving the Exterior Orientation Problem with AEOPA

103

Conversely, closed-form algorithms directly compute the camera pose formulating the problem into a system of equations. The first closed-form methods (Grunert 1841; Finsterwalder and Scheufele 1903; Fischler and Bolles 1981) dealt with the so-called P3P problem, which aims at finding the exterior orientation for the minimal case n = 3. The P3P problem implies the computation of multiple solutions, up to four as first explained in (Finsterwalder and Scheufele 1903; Grunert 1841). An extensive review of these works published before 1994 can be found in Haralick et al. (1994). More recently, Gao et al. (2003) presented an algebraic approach combined with a set of analytical criteria to determine the number of real positive solutions. They also provided a more intuitive geometric approach that involves linear inequalities. In Kneip et al. (2011), the authors presented a novel parameterization of the P3P problem which computes in one stage the orientation and the position of the camera in the world reference frame. Several solutions (Fischler and Bolles 1981; Quan and Lan 1999; Fiore 2001; Ansar and Daniilidis 2003; Lepetit et al. 2009) have been proposed for the general case n ≥ 3. In Lepetit et al. (2009), the main idea is to represent the n points in the object space as a weighted sum of only four control points and consequently to solve the problem in terms of the camera reference coordinates of these four virtual points. This transformation can be done in O(n). Hesch and Roumeliotis (2011) recently presented a closed-form method that minimizes a nonlinear least squares cost function whose size is not dependent on the number of points. In Fiore (2001), the author recognizes that, if the depths of the 3D points are known, the exterior orientation problem can be reduced to an absolute orientation problem. Hence, he first finds the depth with an handcrafted method and then recovers the absolute orientation by solving an orthogonal Procrustes problem. As the method based on AEOPA presented in this chapter is closely related to (Fiore 2001), we will briefly summarize it, following the formulation found in Fusiello (2007).

8.1.2 Exterior Orientation with Fiore’s Algorithm Given a number of 2D-3D point correspondences xi ↔ Xi (in homogeneous coordinates) and the intrinsic camera parameters K, we are required to find a rotation matrix R and a translation vector t (which specify attitude and position of the camera) according to Eq. (6.2.15), here recalled: ˜ i for all i ζi x˜ i = K[R|t]X

(8.1.1)

where ζi denotes the depth1 of Xi , and the ∼ denotes homogeneous coordinates (with a trailing “1”). The core of Fiore’s algorithm is the recovery of the unknown depth ζ i . Let us write Eq. (8.1.1) in matrix form: 1 The

depth of a point is its distance from the focal plane of the camera.

104

8 Applications of Anisotropic Procrustes Analysis

˜ 1, X ˜ 2, . . . X ˜ n ]. [q˜ 1 , q˜ 2 , . . . q˜ n ] diag (ζ ) = [R|t][X       M

U

where q˜ i = K−1 x˜ i and diag(·) returns a diagonal matrix that has the same diagonal elements of its argument. Let r = rank(M). Take its singular value decomposition: M = VDW and let W2 be a matrix composed by the last n − r columns of W, which span the null space of M. Then, MW2 = 03×(n−r ) , and also Udiag (ζ ) W2 = 03×(n−r ) .

(8.1.2)

From the last equation, the depths ζ can be recovered (up to a scale factor) by solving the homogeneous system of equations:    W2 ◦ U ζ = 0.

(8.1.3)

where ◦ denotes the Khatri-Rao product. The size of the coefficients matrix is 3(n − r ) × n; hence, its null space has dimension one as soon as 3(n − r ) ≥ n − 1. Therefore, at least n ≥ (3r − 1)/2 points are needed: six points in general position, or four points on a plane, are sufficient. Now that the left side of Eq. (8.1.1) is known, up to a scale factor, we are left with an absolute orientation (with scale) problem, which can be solved linearly as an Extended Orthogonal Procrustes Analysis (EOPA) problem, described in Sect. 2.4. As a result, the estimated rotation matrix is orthonormal by construction.

8.1.3 Exterior Orientation with AEOPA Whereas Fiore’s solution is partially procrustean, for it employs EOPA in the second step, in this section we will cast the exterior orientation (or PnP) problem as an instance of Anisotropic Extended Orthogonal Procrustes Analysis (AEOPA). After some rewriting, (8.1.1) becomes: ⎡

0 ... .. . . . . 0 0 ...   ζ1 ⎢ .. ⎣.

Z

⎤⎡  ⎤ ⎡ ⎤ ⎡  ⎤ q˜ 1 O X1 0 .. ⎥⎢ .. ⎥R + ⎢ .. ⎥ = ⎢ .. ⎥. ⎣.⎦ ⎣ . ⎦ . ⎦⎣ . ⎦

ζn



q˜ n    Q



O    jO

(8.1.4)



Xn    S

where q˜ i = K−1 x˜ i , O = −R t, and j is the unit vector. In matrix form: S = ZQR + jO

(8.1.5)

8.1 Solving the Exterior Orientation Problem with AEOPA

105

where • Q is the matrix by rows of (homogeneous) image coordinates defined in the camera frame. • S is the matrix by rows of point coordinates defined in the external system. • Z is the diagonal (positive) depth matrix. • O is the coordinate vector of the projection center. • R is the orthogonal rotation matrix. One can recognize an instance of the classical AEOPA model (4.2.1), whose solving equations are reported in Sect. 4.2. Therefore, the Procrustes solution of the PnP problem can be summarized as follows: Algorithm 1 Procrustes PnP (PPnP) Input: a set of 2D-3D correspondences (Q, S) Output: position O and attitude R of the camera 1. Start with Z = I (or any  guess on thedepth); 2. Compute R = V diag 1,1,det(VW ) W   with VDW = Q Z I − j j /n S; 3. Compute O = (S − ZQR)  j/n; 4. Compute Z = (QQ  I)−1 (QR(S −Oj )  I); where  denotes the Hadamard product 5. Iterate over steps 2, 3, 4 until convergence.

The nonnegativity constraint on Z must be enforced a posteriori in step 4 by clipping to zero negative values, if any. Please note that the solution of the AEOPA problem finds Z, R, and O in such a way to minimize the sum of squares of the residual  (see Fig. 8.1), where  is the difference vector between a reference 3D point (S) and the back-projected 2D point (Q) based on its estimated depths (Z) and the estimated camera attitude and position (R, O). In order to gain some understanding about the error being minimized—which is S − ZQR − jO 2F —let us rewrite:   S − ZQR − jO = Z Z−1 (S − jO )R − Q .   

(8.1.6)

B

The term (S − jO )R in the expression of the matrix B represents the 3D coordinates of control points in the camera reference system; let [x j , y j , z j ] be the jth row of such matrix, and let [u j , v j , −c] be the jth row of Q. Then, we have:

106

8 Applications of Anisotropic Procrustes Analysis

Fig. 8.1 Position and attitude of the camera plus the depth of the points are estimated in such a way to minimize the length of the s for all the points, in a least squares sense. The estimated depth defines a 3D point along the optical ray of the image point q. The segment (perpendicular to the optical ray) joining this point and the corresponding reference 3D point X is 

2

1

X2

X1

X3 3

q2 q1

q3

O B 2F = Z−1 (S − jO )R − Q 2F = n  1 [x j , y j , z j ] − [u j , v j , −c] 2 = zj j=1  n   x j yj 2 − [u j , v j ] 2 . n(1 − c) + , z z j j j=i

(8.1.7)

The first term is constant, while the second term is the sum of the squared distances between the measured image points [u j , v j ] and the projection of the corresponding reference points. Hence, minimizing B 2F is equivalent to minimizing the sum of squared image coordinate residuals of the collinearity equations (i.e., the classical bundle adjustment error function). The PPnP method instead minimizes ZB 2F , which is related to B 2F by the following inequality: min(Z2 ) B 2F ≤ ZB 2F ≤ max(Z2 ) B 2F .

(8.1.8)

Since the entries of Z are bounded by physical constraints, i.e., z min < z i < z max with suitable constants z min and z max , when ZB 2F approaches zero so does B 2F and vice versa. The insight to be gained from this application is that whereas long-established Orthogonal Procrustes Analysis seeks transformations in 3D space, the anisotropic extension allows to deal with problems involving a projection to 2D space by introducing an auxiliary unknown, the depth of the points, which allows to back-project 2D points into 3D space, thereby restoring the 3D problem.

8.1 Solving the Exterior Orientation Problem with AEOPA

107

8.1.4 Experimental Validation The Procrustes PnP algorithm (PPnP) was compared with state of the art (algorithms whose implementation is available on the Web), namely: • • • •

DLS: the direct least squares method of Hesch and Roumeliotis (2011); EPnP: the non-iterative linear solution of Lepetit et al. (2009); gOP: the semi-definite program approach of Schweighofer and Pinz (2008); Fiore: the SVD-based method by Fiore (2001).

In all the experiments, the initial depths were set to one, as specified in step 1 of Algorithm 1, and the method always converged to the correct solution. The algorithm terminates when the difference between the values of the residual matrix of two consecutive iterations is smaller than a threshold (set to 10−5 in the experiments). Both synthetic and real experiments were run (Garro et al. 2012). In the synthetic case, n = {6, . . . , 50} 3D points were randomly distributed at a distance between 0.5 and 1.5 meters from the camera. The focal length of the camera was set to 600 pixel, and the image resolution is 800 × 600. The image coordinates obtained from the projection of the 3D points were perturbed with different values of noise σ = {1, . . . , 10} [pixel]. For each value of n, the test was run 100 times and the average error norm was computed. Results are reported in Figs. 8.2, 8.3, and 8.4. As a figure of merit, the rotation ˆ F, error is shown, i.e., the angle of the residual rotation, computed as log(R R) ˆ where R is the ground truth, R is the actual rotation, and · F is the Frobenius norm. Rotation error vs noise 3

mean rotation error [deg]

2.5

DLS EPnP gOp Fiore PPnP

2 1.5 1 0.5 0 1

2

3

4

7 6 5 noise std dev [pixel]

8

9

10

Fig. 8.2 Rotation error versus noise using 30 correspondences. The average rotation error is plotted against the standard deviation of the noise added to image coordinates. PPnP, gOP, and DLS are practically superimposed with the lowest error

108

8 Applications of Anisotropic Procrustes Analysis Rotation error vs #points 4

DLS EPnP gOp Fiore PPnP

mean rotation error [deg]

3.5 3 2.5 2 1.5 1 0.5 0 5

10

15

20

35 30 25 number of points

40

45

50

Fig. 8.3 Rotation error versus number of points. The average rotation error is plotted against the number of points that were used Time vs #points 0.1

mean time [sec]

0.08

DLS EPnP gOp Fiore PPnP

0.06

0.04

0.02

0

10

15

20

25 30 35 number of points

40

45

50

Fig. 8.4 Execution time versus number of points. The average execution time is plotted against the number of points involved. gOP is by far the slowest algorithm

In real experiments, shown in Fig. 8.5, real data coming from structure-and-motion reconstructions were considered, where cameras and points were bundle adjusted. In the lack of a proper ground truth, this was taken as the reference. Basing on given 2D3D correspondences, each camera was oriented with respect to the structure and the errors were computed. n = {20, . . . , 50} 2D-3D pairs were randomly chosen from the set of correspondences of each camera. For these experiments, the minimum value of n was set to 20 instead of 6 in order to deal with the presence of noise. For

8.1 Solving the Exterior Orientation Problem with AEOPA

109

Fig. 8.5 “RectStone” dataset. The figure depicts points and cameras obtained by a structure-andmotion pipeline 0.09

DLS EPnP Fiore PPnP

mean orientation error [deg]

0.08 0.07 0.06 0.05 0.04 0.03 0.02 0.01 0 0

2

4

6

8 10 camera

12

14

16

18

Fig. 8.6 Mean rotation error for each of the 17 cameras of the “RectStone” dataset

each value of n, the test was run 100 times and the orientation average error norm was computed. In this case, the gOP algorithm was not tested because the code for the planar case was not longer available on the Web. The first dataset is shown in Fig. 8.5, while results are reported in Fig. 8.6. The second reported experiment was run using the standard dataset “Herz-JesuP8” available on the Web.2 The related results are shown in Fig. 8.7. Real experiments confirm the algorithms’ performance in the synthetic case: PPnP achieves always the best accuracy; instead, DLS has a slightly worse error, with an unexplainable glitch for camera n. 4 of the “RectStone” dataset. 2 http://cvlab.epfl.ch/~strecha/multiview/denseMVS.html.

110

8 Applications of Anisotropic Procrustes Analysis 0.045 DLS EPnP Fiore PPnP

mean orientation error [deg]

0.04 0.035 0.03 0.025 0.02 0.015 0.01 0.005 0

1

2

3

4 5 camera

6

7

8

Fig. 8.7 Mean rotation error for each of the 8 cameras of the “Herz-Jesu-P8” dataset

8.1.5 EIV-AEOPA Solution for the Exterior Orientation Problem The least squares Procrustes solution presented in the previous sections assumes that all random errors are confined to matrix S, that contains the 3D point coordinates defined in the external system, whereas the image coordinates contained in Q are assumed noise-free. However, this assumption is often unrealistic, since both Q and S derive from measurements and can be corrupted by errors. It is therefore appropriate to resort to the EIV-AEOPA solution, presented in Sect. 5.2.3, that considers the Errors-In-Variables (EIV) model, a more general model wherein both matrices Q and S are assumed to be contaminated by errors. Following the solving equations reported in Sect. 5.2.3, the total least squares Procrustes solution of the PnP problem (Maset et al. 2017), that we call EIV-PPnP, can be summarized as in Algorithm 2. It is particularly significant in this photogrammetric application the capability of the EIV-PPnP algorithm to take into account the different variances of Q and S, since image coordinate and 3D point coordinates are measured with different accuracy. Experiments were performed (Maset et al. 2017) to compare the EIV-PPnP solution against the LS formulation PPnP (Sect. 8.1.3). To carry out the simulation, n = {6, . . . , 30} 3D points were randomly distributed in a sphere of unit radius centered on the origin and perturbed with random noise with standard deviation σ S = {0.10, 0.27, 0.71, 1.88, 5.00} [mm] (corresponding to five logarithmically spaced values from 0.10 to 5.00). The camera was positioned at distances of 5 and 10 m from the origin, and the focal length was chosen so as to yield a view angle of 60◦ with an image size of 1000 × 1000 pixels. Different values of noise σ P = {1, . . . , 5}

8.1 Solving the Exterior Orientation Problem with AEOPA

111

Algorithm 2 EIV Procrustes PnP (EIV- PPnP) Input: a set of 2D-3D correspondences (Q, S) and an estimate of the coordinate standard deviation σ Q and σ S Output: position O and attitude R of the camera 1. Start with Z = I (or any  guess on thedepth); 2. Compute R = V diag 1,1,det(VW ) W with   VDW = Q Z (σ S I + σ Q Z2 )−1 I − jj /ε(σ S I + σ Q Z2 )−1 S and   ε = tr (σ S I + σ Q Z2 )−1 .  2 −1 3. Compute O = (S −  ZQR) (σ S I + σ Q Z ) j/ε;  4. Compute Z from σ Q Z Z + σ S Z − σ Q  Z − σ S   I = 0, with  = QR and  = S − jO ; and  denoting the Hadamard product 5. Iterate over steps 2, 3, 4 until convergence.

[pixel] were added to the image coordinates obtained from the projection of the noise-free 3D points. For each setting, the test was run 500 times and the mean and median error norms were computed. In all the experiments, the initial depths were set to one. Results are reported in Figs. 8.8 and 8.9. Even in this case, as a figure of merit the rotation error is shown.

Rotation error vs noise 3d points coordinate noise [mm] 0.00 0.40

0.27

0.71

1.88

5.00

2

3

4

5

median EIV-PPnP median PPnP mean EIV-PPnP mean PPnP

0.35

rotation error [deg]

0.10

0.30 0.25 0.20 0.15 0.10 0.05 0.00 0

1

image coordinate noise [pixel] Fig. 8.8 Rotation error versus noise using 10 correspondences and a distance of the camera from the origin equals to 10 m. The mean and the median rotation errors are plotted against the standard deviation of the noise added to image coordinates and 3D point coordinates

112

8 Applications of Anisotropic Procrustes Analysis

Rotation error vs #points 0.8 median EIV-PPnP median PPnP mean EIV-PPnP mean PPnP

0.7

rotation error [deg]

0.6 0.5 0.4 0.3 0.2 0.1 0 5

10

15

20

25

30

number of points Fig. 8.9 Rotation error versus number of points. The mean and the median rotation errors are plotted against the number of points that have been used. The distance of the camera from the origin is equal to 10 m, and the standard deviation of the noise added is 3 pixel for the image coordinate and 0.71 mm for the 3D point coordinates

Comparing the median error, one can notice that EIV-PPnP and PPnP lead almost to the same accuracy. On the other hand, the mean is different when the number of correspondences in the image is small (n ≤ 15) (this is clear in Fig. 8.8 where 10 points are considered). This is due to the fact that PPnP in a few cases converges to wrong results, that are not sufficient to skew the median, but affects the mean error. The rundown of this simulation is that the total least squares solution (EIV-PPnP) works better than the ordinary least squares algorithm (PPnP) when the number of reference points (whose coordinates are known in both the camera and the external reference frame) is small, which is a common situation in photogrammetry.

8.1.6 Robust Procrustes Exterior Orientation The methods illustrated in Sects. 8.1.3 and 8.1.5 are based on (ordinary or total) least squares estimation, so they suffer from the presence of outliers. To overcome this problem, we present in the following a robust version of the PPnP algorithm, first described in (Fusiello et al. 2013), that can tolerate a minority of arbitrarily large outliers. At the beginning, least median of squares (LMedS) (Rousseeuw 1984) is used in order to detect an initial minimal subset of data, free from outliers. The initial subset is then augmented one sample at a time with Forward Search (FS) (Hadi and

8.1 Solving the Exterior Orientation Problem with AEOPA

113

Simonoff 1993; Atkinson and Riani 2000), an iterative scheme that proved to be effective in dealing with the well-known masking and swamping problems that can occur when multiple outliers are present in the dataset. An empirical comparison of FS against the customary classification based on the median absolute deviation (MAD) scale estimation was carried out. As it will be explained in the following sections, up to 20% of outliers MAD and FS produce comparable results, but when the percentage of outliers increases, FS becomes more reliable for outlier detection.

8.1.6.1

Robust Estimation Background

Almost inescapably image measurements contain gross errors (outliers) that may be arbitrarily large and cannot be “averaged out,” as is typically done with small-scale noise. They lead to incorrect results in parameter estimation; therefore, robust estimation techniques have been developed over the last years in both the photogrammetric computer vision and the statistics literature (Ben-Gal 2005; Stewart 1999). The least median of squares is a robust regression method that achieves the maximum theoretical breakdown point of 0.5; i.e., it can tolerate up to 50% percent of outliers in the data without being affected. LMedS estimates the parameters of the model by minimizing the median of the absolute residuals. Let Y = {yi } be a set of data points and let a be a k-dimensional parameter vector to be estimated, while ri,a is the Euclidean distance between a point yi and the model determined by a. The LMedS estimate is: 2 . (8.1.9) aˆ = argmina medyi ∈Y ri,a The value of a is determined using a random sampling technique. A number of kpoints subsets of the n data points is randomly selected, and a parameter vector a is fitted to the points in each subset . Each a is tested by calculating the squared 2 of each of the n − k points in Y −  and finding the median. residual distance ri,a  The a corresponding to the smallest median is chosen   as the estimate aˆ . If n is the number of points in the initial set of data, there are nk different subsets of k points that should be considered. In order to speed up the process, the number of subsets to test can be reduced to log(1 − p) , (8.1.10) M= log(1 − (1 − )k ) where p is the probability of calculating the correct estimate and  is the percentage of outliers. Since LMedS is known to have a low statistical efficiency, it is important to refine the fit on a “clean” (i.e., outliers-free) set of data point. This is customarily done by computing a robust scale estimate σ ∗ based on the median absolute deviation (MAD):   5 ∗ −1 (8.1.11) medyi ∈Y−ˆ ri,ˆa2 σ = 1/φ (0.75) 1 + n−k

114

8 Applications of Anisotropic Procrustes Analysis

ˆ is the subset used to calculate aˆ , φ −1 is the inverse of the cumulative normal where  distribution, and 1 + 5/(n − k) is a correction factor. Then, aˆ is refined with a least squares estimate using only the data points such that: 2 ∗ 2 ri,ˆ a < (θ σ )

(8.1.12)

where θ is a constant that depends on the confidence level chosen for the test (typically around 2 or 3). According to the taxonomy set forth in (Davies and Gather 1993), this is a univariate single-step outlier detection procedure, that will be referred to as MAD in the following. Sequential procedures, on the contrary, perform successive elimination or addition of points. In particular, inward testing starts with the whole set of data points and iteratively tests the point with the highest residual for being an outlier; it stops when the first inlier is encountered. On the contrary, outward testing starts with an outlier-free set of points and iteratively tests the remaining points for inclusion in the clean subset, based on statistics computed on the clean subset; it stops when all the points have been processed. Forward Search (FS) (Hadi and Simonoff 1993; Atkinson and Riani 2000) is an outlier detection method of the latter category. The algorithm starts from an initial subset of observations of size m < n that is known to be outlier-free. In our case, ˆ from LMedS, and m = k. Then, the iterative process starts and at this is the set  every ith iteration the s samples (s = m + i − 1) with the lowest residual are used to estimate the parameters of the model as and the s + 1 residual rs+1,as is monitored to detect the point when the iteration starts including outliers. Several monitoring heuristics have been proposed in the literature. We consider here one of the simplest, that dates back to (Hadi and Simonoff 1993); the Forward Search stops when: 2 ≥ (t(1−α/2(s+1),s−k) σs )2 (8.1.13) rs+1,a s where t(1−α/2(s+1),s−k) is the 1 − α/2(s + 1) quantile of the t-distribution with s − k degrees of freedom, and σs is the standard deviation of the least s residuals (the residual divided by σs is also called “studentized” residual). Figure 8.10 plots the (s + 1)th residual at each iteration: Please note how the threshold adapts to the variance of the clean subset and how the residual of the first outlier pops out. Same analysis can be made on Fig. 8.11, which plots the ordered residuals of the whole point set: The outliers are clearly separated and well above the threshold. The ith iteration of the algorithm can be summarized as follows: 1. Estimate the parameters of the model as using the s = m + i − 1 points in the clean subset. 2. Calculate the residuals rs+1,as of all the n points and arrange them in ascending order.

8.1 Solving the Exterior Orientation Problem with AEOPA

115

−1

Residual

10

−2

10

20

15

10

5

0

35

30

25

Iteration

Fig. 8.10 (s + 1)th residual (diamonds) and the value of the threshold (t(1−α/2(s+1),s−k) σs ) (red line) at each iteration of FS 0

10

−1

Residual

10

−2

10

−3

10

−4

10

0

5

10

15

20

25

30

35

40

45

50

Point

Fig. 8.11 Residuals at the last iteration of FS arranged in ascending order. The red horizontal line is the threshold that has been exceeded

3. Test the (s + 1)th residual: 2 If rs+1,a ≥ (t(1−α/2(s+1),s−k) σs )2 then declare all the remaining n − s obsers vations as outliers and stop. Otherwise increment i (include the (s + 1)th point in the inliers).

4. If m + i − 1 = n, then stop, otherwise go to step 1.

8.1.6.2

Robust Procrustes Solution

In order to work toward the robust version of Algorithm 1 (Sect. 8.1.3), we start noting that: • step 2 can be seen as the solution of a simple OPA problem (Sect. 2.3), namely   finding R such that ZQ = R I − j j /n S;

116

8 Applications of Anisotropic Procrustes Analysis

• step 3 is simply the mean along the columns of the matrix S − ZQR. Therefore, one can immediately figure out that the PPnP algorithm can be made robust against outliers by substituting: • step 2. with a robust solution (e.g., LMedS) to the problem of finding the rotation  between the two point sets ZQ and I − j j /n S; • step 3. with the median instead of the mean along the columns of S − ZQR. This works pretty straightforward, and being an iterative process, convergence can be improved by re-using the estimate of the inliers set from one iteration to the next. As customary, we used a threshold based on MAD (described in Sect. 8.1.6.1) to reject outliers at each iteration. However this simple strategy cannot recover the inlier set with high accuracy for every condition. There are cases where FS, albeit more computationally expensive, should be used because of its superior accuracy, as experiments will show. Therefore, the clean subset produced by the LMedS procedure is refined with FS; the resulting robust PPnP procedure is summarized in Algorithm 3.

Algorithm 3 Robust PPnP Input: a set of 2D-3D correspondences (Q, S) Output: position O and attitude R of the camera • Find a clean subset of m points:3 1. 2. 3. 4. 5. 6.

Start with Z = I (or any guess  on thedepth); Compute R s. t. Q Z = R I − j j /n S using LMedS (on current inliers); Compute O = meaninliers (S − ZQR)  ; Compute Z = (QQ  I)−1 (QR(S − Oj )  I); where  denotes the Hadamard product Declare as inliers points that pass the test of Eq. (8.1.12); Iterate over steps 2, 3, 4, 5 until convergence;

• Refine inliers estimate using FS; • Compute R, O, Z (Algorithm 1) considering the set of points determined in the previous step.

8.1.6.3

Experimental Validation

Synthetic experiments were run using n = {20, 40, 60, 80, 100} 3D points (Fusiello et al. 2013). The image coordinates obtained from the projection of the points were perturbed with gaussian noise with standard deviation σ = 2.0 and different percentages of outliers {10, 20, 30, 40, 50%} were added to the data. Outliers are generated by adding a random value with uniform distribution in [0, 100] to both coordinates 3 Exterior

orientation requires m = 3, but a larger set can be used to improve speed.

8.1 Solving the Exterior Orientation Problem with AEOPA

117

of a point. The focal length of the camera was set to 500 pixel, and the image size is ≈600 × 600 (Fig. 8.12 shows an example of synthetic image). For each value of n and percentage of outliers, the test was run 100 times. Initially, the first part of the algorithm was tested (i.e., the one responsible for extracting a clean subset, based on LMedS) and, as expected, it provides a clean subset (with up to 50% of outliers) in most of the cases (76%). The percentage reaches 92% if one considers only a maximum of 40% of outliers. Figure 8.13 reports the false negative rate (see below) for the first part of the algorithm. Then, the robust PPnP method (with FS) was empirically compared against the baseline method with the MAD-based test only, in order to assess whether FS brings some advantage in detecting outliers. The outlier detection stage post-LMedS can be considered as a classification task, where each point has to be labeled as inlier or outlier (i.e., the null hypothesis that a point is an outlier is tested). θ and α were set to 2.0 and 0.0001, respectively, so as

Fig. 8.12 An example of the synthetic image data. The green squares are the inliers, the red stars are the outliers

600

500

400

300

200

100

0 0

Fig. 8.13 False negative rate versus percentage of outliers and number of points for the first part of the algorithm

100

200

300

400

500

600

LMedS

False negative rate

0.4 0.3 0.2 0.1 0 50 40

20

30

40

20

60 10

% outliers

80 100

# points

118

8 Applications of Anisotropic Procrustes Analysis

to yield a similar rejection threshold for a chosen reference case, corresponding to 100 points, 10% outliers, σ = 2.0. Let c be the number of false negatives, i.e., actual outliers that are erroneously deemed as inlier (type I error), a be the number of true positives, i.e., outliers that are correctly detected, and d be the number of true negatives, i.e., inliers that are correctly detected. In order to compare the two algorithms, two figures of merit were monitored: – the false negative rate defined as c/(a + c), and – the accuracy, defined as (a + d)/n. Please note that false negatives (type I error) are way more dangerous than false positives, as they may skew the final least squares estimate, whereas false positives (type II error) can only impact on the statistical efficiency, as they reduce the number of “good” measurements that are considered in the final least squares estimate. For this reason, the main indicator is the false negative rate, which should be as small as possible. The accuracy indeed is monitored in order to check that the method is not achieving a low false negative rate by being too “fussy,” i.e, rejecting too many samples as outliers. Results are reported in Fig. 8.14. The bar corresponding to 20 points and 50% outliers can be considered as an extreme case and should probably left out from general considerations. The opposite case, corresponding to 100 points and 10% outliers, is the “easy” one. The rundown of the experiments is that with a fairly large set of point (>60) and moderate outliers contamination ( 4. Let us consider its SVD decomposition:

124

8 Applications of Anisotropic Procrustes Analysis

[S, j] = VDW

(8.2.13)

and in particular let Wr contain the last n − r columns of W corresponding to zero singular values. Hence [S, j] Wr = 0 and Wr  [S, j] = 0 and also Wr  (ZQ + G) = 0

(8.2.14)

from which Z can be recovered by solving the linear system of equations: (Q ◦ Wr  ) diag−1 (Z) = − vec(Wr  G).

(8.2.15)

The coefficient matrix has dimension 3(n − r ) × n and there are n unknowns. If points are in general position r = 4, hence at least 6 points are needed. If n ≥ 6 the least squares solution in closed form is: diag−1 (Z) = (QQ  Wr Wr  )−1 diag−1 (Wr Wr  GQ ).

(8.2.16)

Once Z is known the parameters of the similarity transformation R, O and α are computed with EOPA (see Sect. 2.4) as in Sect. 8.2.2. The analytical procedure, henceforth dubbed gFiore, is summarized in Algorithm 5. Algorithm 5 gFiore Input: 3D points S and lines (G, Q) (origin, direction) Output: parameters R, O and α of the similarity transformation that aligns points and lines 1. Obtain Wr from the SVD of [S, j]; 2. Compute Z with diag−1 (Z)=(QQ Wr Wr  )−1 diag−1 (Wr Wr  GQ ); where  denotes the Hadamard product   3. Compute R = V diag 1,1,det(VW ) W   with VDW = (ZQ + G) I − j j /n S; 4. Compute α with (8.2.7); 5. Compute O = (αS − (ZQ + G)R) j/n.

8.2.4 Experimental Validation Experiments on simulated data were run as follows (Fusiello et al. 2015). Line origins were generated randomly in the unit cube centered on zero, 3D points were randomly placed on a sphere of unit radius, whereas line directions were computed as unit vectors from camera origins to 3D points. A random similarity transformation was applied to 3D points, where rotations is obtained from three random Euler angles, translation is a random vector with norm in [0.5, 10] and scale is a random number

8.2 Solving the Non-Perspective-n-Point Problem with AEOPA

125

between 0.1 and 10. Noise was added by perturbing line versors with a Gaussian random vector with increasing standard deviation σ from 0 to 0.1 units. Every point of the graph is the average of 100 trials. As a figure of merit, the rotation error in degrees was evaluated as a function of noise level and number of points. The two algorithms previously described, namely gFiore (Sect. 8.2.3) and gPPnP (Sect. 8.2.2) were compared with gDLS,4 the only recent non-minimal solver for NPnP that deals with scale. Results are reported in Fig. 8.16. The three algorithms have substantially equivalent performances, and the rotation error grows almost linearly with the noise standard deviation, and decreases with the number of points. However, when the noise level increases the differences becomes more noticeable and gPPnP achieves the lowest error, followed by gFiore.

8.3 Solving the Bundle Block Adjustment Problem with AGPA Bundle block adjustment is the most classical analytical problem in photogrammetry. Great effort was given to the problem solution since the middle of the last century (see, e.g., Baetsle 1956; Brown 1958; Ackermann 1962; Cunietti 1968). An exhaustive synthesis of the analytical developments carried out in this fundamental field of photogrammetry can be found, for instance, in (Triggs et al. 2000). As already shown in Sect. 6.3.1, the standard formulation requires the linearization of the collinearity equations and the satisfaction of the least squares principle for the equation residuals. Some additional unknown terms can be considered for each equation in order to calibrate the camera for some systematic error terms or for simultaneously estimating the image interior orientation parameters. Robust least squares solutions have been also proposed in the literature in order to reduce the influence of outliers (e.g., Zhang et al. 2006). According to Triggs et al. (2000), the most significant bundle block adjustment paradigmatic enhancements, in chronological sequence, are: 1. recursive partitioning by Gyer (1967) and Brown (1968) that led to the modern sparse matrix techniques; 2. S-transformations and criterion matrices by Baarda (1973), that allowed the correct estimate of the network degrees of freedom and the uncertainty modeling in the adjustment process; 3. photogrammetric precision and reliability overparametrization and model choice by Gruen (1980) and Förstner (1985), that opened the way to modern robust statistics and model selection in photogrammetry;

4 The

C++ code is available on the Web as part of the Theia library (Sweeney 2015).

126

8 Applications of Anisotropic Procrustes Analysis 2.5 gPPNP gDLS gFiore

Rotation error [deg]

2

1.5

1

0.5

0 0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

noise std dev [units] 2.5 gPPNP gDLS gFiore

Rotation error [deg]

2

1.5

1

0.5

0 10

16

26

40

64

100

158

252

398

630

1000

n. of points

Fig. 8.16 Rotation error in degrees is plotted as a function of the noise level σ (top) with 64 points and the number of points (bottom) with σ = 0.04

8.3 Solving the Bundle Block Adjustment Problem with AGPA

127

4. “geometrically constrained multiphotograph and globally enforced least squares matching” by Gruen and Baltsavias (1986), that introduced the so-called imagebased matching technique procedures. In spite of these fundamental steps in the methodological development of bundle adjustment, the common underlying scheme, based on the least squares solution of a large nonlinear system of equations, has been the same since its origins. In the following sections we describe instead an alternative analytical bundle block solution method (Fusiello and Crosilla 2015, 2016) rooted in the framework of Procrustes Analysis, in particular focusing on its generalized anisotropic variant, illustrated in Sect. 4.3. As demonstrated also for the PnP and NPnP problems, the main advantage of the Procrustes solution is that—upon convergence—it furnishes a least squares solution without any linearization of the original equations, and without any approximate value of the unknown parameters and of the tie-points 3D coordinates. More in detail, the method extends the same relaxation procedure that characterized AEOPA, used to solve the PnP problem (Sect. 8.1.3), by considering also the presence of unknown tie-points imaged to some or all exposures. Their role is to constraint the different images one to each other in order to iteratively update their approximate exterior orientation parameters until a global convergence of the entire block. The method presented below falls in the wider structure-from-motion family. Its closest neighbors are the iterative factorization methods and the global motion-first methods. Iterative factorization methods (Sturm and Triggs 1996; Heyden 1997; Oliensis 1999; Oliensis and Hartley 2007) yield a projective model from multiple images by a two step iteration (a block relaxation, in fact), where in one step a measurement matrix, containing image points coordinates, is factorized with SVD, and in the subsequent step the depths of the points are computed, assuming all the other parameters fixed. This bears some resemblance of the scheme described in the following, which however, deals with calibrated cameras (i.e., known interior orientation) and outputs a Euclidean model instead of a projective one (a projective/Euclidean model differs from the true one by a projectivity/similarity transformation). Moreover, unlike these algorithms, the Procrustes solution of bundle block adjustment does not require all points to be visible in all views. The issue of visibility in matrix factorization methods can be side-stepped by matrix completion techniques, exploiting the low rank of the measurement matrices (Brand 2002; Kennedy et al. 2013; Hartley and Schaffalitzky 2003), or by providing additional information. Indeed, Kaucic et al. (2001) and Rother and Carlsson (2002) describe algorithms based on SVD for the projective modeling from multiple perspective views, based on having four points on a reference plane visible in all views. Unlike iterative factorization ones, these algorithms does not require all points to be visible in all views and are also direct. If three orthogonal vanishing points are specified in addition, the model can be upgraded to Euclidean. The scheme proposed in (Hartley and Schaffalitzky 2003), in particular, can be contrasted with the Procrustes method, since its iteration can be interpreted as linearly solving alternately

128

8 Applications of Anisotropic Procrustes Analysis

for camera matrices and 3D points until their product converges to the measurement matrix. Global motion-first methods share a common scheme: they start from known interior orientation, compute epipolar or trifocal geometry which results in relative rotations and relative translations (up to a scale). Then solve the rotation registration or rotation averaging (Hartley et al. 2013) problem that gives the rotational component of the cameras orientation; this problem, if one ignores outliers, can be solved directly by eigen-decomposition of a matrix (Martinec and Pajdla 2007; Arie-Nachimson et al. 2012). Finally, camera location is solved (a.k.a. translation registration) by a variety of direct/iterative methods, including solving a linear system of equations (Sect. 4.1 Kraus 1997), (Arie-Nachimson et al. 2012; Jiang et al. 2013), eigen-decomposition (Brand et al. 2004), linear programming (Moulon et al. 2013), Second Order Cone programming (Kahl and Hartley 2008; Martinec and Pajdla 2007), nonlinear least squares (Wilson and Snavely 2014). Since some of these methods are direct and the Procrustes scheme is iterative, they can be considered superior from this point of view. However, they minimize algebraic residuals, which in certain cases are based only on the orientations (e.g., Brand et al. 2004), ignoring the 3D points until the final intersection. On the other hand, the described solution, based on AGPA, minimizes a geometric residual, similarly to photogrammetric bundle block adjustment. As a matter of fact, the experiments reported show that the Procrustes method achieves RMS error values (w.r.t. ground control points) practically equal to those obtained by photogrammetric bundle block adjustment.

8.3.1 Bundle Block Adjustment with AGPA Let us now consider m images depicting the same n 3D tie-points X1 . . . Xn . In the bundle block adjustment problem it is required to simultaneously find the image exterior orientation parameters and the tie-points 3D coordinates that minimize a geometric error, without introducing intermediate models, as opposed to the block adjustment by independent models. As done in (Fusiello and Crosilla 2015), in order to derive the Procrustes solution to the bundle block adjustment problem, we can start recalling Eq. (8.1.5), introduced for the PnP problem in Sect. 8.1.3, which is an equivalent formulation of the collinearity equations. We can write for each image i: S = Zi Qi Ri + jOi  .

(8.3.1)

In this formula the image coordinates Qi are known, but all the other quantities are unknown, including the 3D points S and their depths Zi . If Zi were known, the problem would reduce to a GPA with a rigid transformation (see Chap. 3), where the point sets to be aligned are the Zi Qi for i = 1 . . . m.

8.3 Solving the Bundle Block Adjustment Problem with AGPA

129

If we rewrite Eq. (8.3.1) as Qi Ri = Z−1 (S − jOi  ) the model becomes the same as the STIMIDIO—modulo the inversion of the weight matrix—described at p. 129 of (Commandeur 1991), which in turn derives from the PINDIS (Lingoes and Borg 1978) model. The iterative solution proposed in (Fusiello and Crosilla 2015), that we call Procrustes Bundle Block Adjustment (PBBA), is sightly different, though. In particular, it is based on the first AGPA scheme reported in Sect. 4.3 and consists of two nested cycles, with a GPA as the inner one, as outlined in Algorithm 6. Algorithm 6 PBBA Input: 2D-2D correspondences Qi over m images Output: position Oi and attitude Ri of the images, 3D points S 1. 2. 3. 4. 5.

∀i : Initialize Zi > 0 Normalize the Zi to unit average; Let Ai = Zi Qi ; solve a GPA problem (Chap. 3) obtaining (Ri , Oi ) and centroid S ∀i : Compute Zi =(Qi Qi  I)−1 (Qi Ri (S −Oi j )I) where  denotes the Hadamard product Iterate from step 2. until convergence.

Since the global scale of the solution is discretionary, only the ratios of the Zi are relevant. Therefore, at each iteration the Zi are normalized to unit average, in order to better condition the convergence. If not all points are visible in all the images there are missing data, which are handled inherently by GPA in step 2., while in step 3., for each image i, only the points visible in that image are considered. With respect to the classical photogrammetric bundle block adjustment, the error being minimized is different: as for the PnP problem, Procrustes bundle block adjustment minimizes a geometric error in the 3D space, whereas photogrammetric bundle block adjustment residuals are measured in the image plane. In particular, it is easy to see (Commandeur 1991) that this procedure minimizes the following objective function: m  S − Zi Qi Ri − jOi  2F (8.3.2) i=1

where each term of the sum represents the difference vector between 3D tie-points (S) and the back-projected 2D points (Qi ) based on their estimated depths (Zi ) and the estimated image attitude and position (Ri , Oi ) of image i. Let us call i j each individual difference vector relative to exposure i = 1 . . . m and tie-point j = 1 . . . n (see Fig. 8.17). Overall, what is being minimized is the length of the residuals i j for each exposure and each tie-point, in a least squares sense. The relationship with the classical photogrammetric bundle block adjustment cost function can be enlightened by following the same path asfor the PnP case (Eqs. 8.1.6–8.1.8). First we write the cost function Eq. (8.3.2) as i Zi Bi 2F where Bi = Zi−1 (S − jOi  )Ri  − Qi , and then prove that:

130

8 Applications of Anisotropic Procrustes Analysis

Fig. 8.17 Consider the rays emanating from each point as “sticks” in space. Procrustes bundle block adjustment optimizes camera position and attitude and the length of each stick so as to bring the sticks’ endpoints as close as possible to each other, i.e., it minimizes the length of the i j

ij

Xj

Image i

Oi

(i)



2 i Bi F

is equivalent to the photogrammetric bundle block adjustment residual up to an additive constant (ii) Procrustes bundle block adjustment error is pinched between two functions proportional to the bundle adjustment error: m min min(Zi2 ) i

m  i=1

Bi 2F ≤

m 

Zi Bi 2F

i=1

m m   Zi Bi 2F ≤ m max max(Zi2 ) Bi 2F . i=1

i

(8.3.3)

i=1

The Procrustes bundle block adjustment solution corresponds to a free adjustment, as it does not involve any object space constraint. Points and images are expressed in an arbitrary reference system (inherent in the GPA), which can be brought into alignment with the desired coordinate frame via a similarity transformation, solving an absolute orientation problem with given ground control points. The solution here described inherits from GPA its slow rate of convergence, which has been already pointed out in (Akca 2003). Also Commandeur (1991, p. 134) comments on the slowness of his iterative solution of the STIMIDIO model. This trait has been partially cured in the implementation proposed in (Fusiello and Crosilla 2015) by introducing a damping factor greater than one in the updating of Zi : the algorithm applies a correction to Zi that is greater than the value computed according to step 4. of a factor ranging from 1 to 2, depending on the convergence rate. In order to further speed up the PBBA algorithm, Fusiello and Crosilla (2016) proposed an alternative iterative scheme without inner cycles, that corresponds to the second AGPA formulation reported in Sect. 4.3. The method, henceforth dubbed

8.3 Solving the Bundle Block Adjustment Problem with AGPA

131

Fast PBBA, is described in Algorithm 7. As shown in (Fusiello and Crosilla 2016), it has the same accuracy and convergence properties of the one described by Algorithm 6, but it is considerably faster. Algorithm 7 Fast PBBA Input: 2D-2D correspondences Qi over m images Output: position Oi and attitude Ri of the images, 3D points S p

1. ∀i: Initialize Zi = I and Qi = Qi 1 m p 2. Compute centroid S = Q m i=1 i 3. Register each Qi to centroid S computing Zi as  −1   Qi Ri (S − Oi j )  I where  denotes the Hadamard product Zi = Qi Qi   I p 4. Update Qi = Zi Qi Ri + jOi  5. Iterate from step 2. until convergence.

8.3.2 Experimental Validation First of all, we report the experimental validation of PBBA carried out on repeated trials with simulated data (Fusiello and Crosilla 2015). For each trial, n 3D tie-points were randomly generated in a sphere of unit radius centered on the origin. Sixteen random cameras (m = 16) looking toward the origin were positioned in a 60◦ sector of the sphere, at an average distance of d units5 from the origin. The focal length was chosen so as to yield a view angle of {60◦ , 120◦ } with an image size of 1000 × 1000 pixels. Random noise was added to the image coordinates. Missing points have been simulated by zeroing random elements in the visibility matrix. A sample simulated scene is shown in Fig. 8.18. Figure 8.19 reports the convergence of error (as defined in Eq. 8.3.2) and depth values for the first image. Please note that the initial value of Z needs not to have any relationship with the actual average depth of the 3D points, for the global scale of the solution is meaningless. Figure 8.20 shows the average and standard deviation of the residuals i j for each point, over the number m of exposures. These residuals are, for each tie-point j imaged in exposure i, the distance from the back-projected point (according to its estimated depth) to its centroid, which represents the estimate of the actual 3D point (see Fig. 8.17). In the experiment aimed at assessing (empirically) the accuracy of the method, the parameters were set to: n = 96, d = 10, p = 36 ( p is the number of tiepoints visible in each image). Increasing random noise with standard deviation σ = {0, 0.5, 1, 1.5, 2, 2.5, 3, 3.5} was added to image points coordinates and 25 trials for each noise level were averaged. In each trial first PBBA was run and then 5 In

these experiments on simulated data, measures are in arbitrary “units.”

132

8 Applications of Anisotropic Procrustes Analysis

Fig. 8.18 Top: A depiction of a simulated scene (camera stations and points) used in the experiment (n = 192, d = 10, p = 36); Bottom: the simulated visibility matrix, showing which point (x-axis) is visible in which image (y-axis). Ray multiplicity is 3

0 10 0

20

40

60

80

100

120

140

160

180

rays: 3

photogrammetric bundle block adjustment, starting from the output of the former. The output of both methods was compared after a least squares alignment with the ground-truth 3D points (Fig. 8.21). The RMS error for PBBA is only slightly higher than the error achieved by photogrammetric bundle block adjustment; it can be considered equal for all practical purposes. Note how the RMS error of 3D points position grows linearly as the standard deviation of the noise increases. A further experiment was designed for characterizing the failure rate, i.e., counting how many times PBBA failed to reach the correct solution in 100 random trials, starting with the usual uninformed initialization (Z = I). A random noise with 1 pixel standard deviation was added to image coordinates. The parameters considered in this simulation are the distance d of the camera from the origin and the number p of tie-points visible in each image. For each value of d = {2, 10, 20} 3D tie-points were stretched in the X Y plane so as to fit in the view frustum, while the Z range is kept constant (to the original two units), so as to give rise to increasing values for the distance to Z -range ratio, namely {1, 5, 10} respectively. The number of visible tie-points per image was set to p = {18, 36, 54}. When the number of images and the number of visible tie-points per image are fixed, total number of points and the ray multiplicity become dependent. In one case the total number of points was kept fixed to n = 96 and the ray multiplicity took the values

8.3 Solving the Bundle Block Adjustment Problem with AGPA

133

Convergence of the internal error

−1

(log) internal error

10

−2

10

−3

10

−4

10

0

10

1

2

10

10

3

10

(log) iterations Convergence of Z−values in image 1 1.5 1.4 1.3

depth (Z)

1.2 1.1 1 0.9 0.8 0.7 0.6 0.5 0 10

1

2

10

10

3

10

(log) iterations

Fig. 8.19 Convergence of error (top) and depth values (bottom) for the simulated scene of Fig. 8.18. Please note that some axes are logarithmic

134

8 Applications of Anisotropic Procrustes Analysis Average and std dev of 3D residuals

−3

3.5

x 10

3

[units]

2.5

2

1.5

1

0.5

0

0

20

40

60

80

100

120

140

160

180

Fig. 8.20 Average and standard deviation of the i j (defined in the text) for the simulated scene of Fig. 8.18 0.01

RMS 3D alignment error [units]

0.009

Bundle Block Adjustment PBBA

0.008 0.007 0.006 0.005 0.004 0.003 0.002 0.001 0 0

0.5

1

1.5

2

2.5

3

3.5

noise std dev [pix]

Fig. 8.21 RMS error on 3D point position versus standard deviation of random noise added to image tie-points. Dark green is the error obtained by photogrammetric bundle block adjustment, while yellow is the error achieved by PBBA. The regression line is also plotted (in red)

8.3 Solving the Bundle Block Adjustment Problem with AGPA

135

{3, 6, 9}. In another case ray multiplicity was fixed to 3 and the total number of points took values n = {96, 192, 288}. Figure 8.22 reports the results. When the ray multiplicity is greater than 3 (top row, p = {36, 54}) the convergence rate is 100%. The case of ray multiplicity= 3 is analyzed in the bottom row. The case p = 18, view angle of 60◦ (with d = 2, in particular) is the worst; convergence rate is at least 95% in all the other cases. Figure 8.23 reports the median RMS error on 3D points position for this experiment. The RMS error is in percentage with respect to the radius of the point cloud (i.e., 1% is 0.01 units on 3D points in a sphere with radius 1). This error is always below 2% and in most cases is below 1%. As expected, the error increases with the distance and decreases with the number of tie-points visible in each image.

Failure rate (96−120)

25

25

20

20

failure rate [%]

failure rate [%]

Failure rate (96−60)

15 10 5

15 10 5

0

0 20

18 10

camera distance

20

36 2

54

18 10

camera distance

points per image

Failure rate (3−60)

2

54

points per image

Failure rate (3−120)

25

25

20

20

failure rate [%]

failure rate [%]

36

15 10

15 10 5

5

0

0 20

18 10

camera distance

36 2

54

points per image

20

18 10

camera distance

36 2

54

points per image

Fig. 8.22 Failure rate of PBBA as a function of camera distance d and number of tie-points visible in each image. Top row: Corresponding ray multiplicity values are [3, 6, 9], where n = 96. Bottom row: Corresponding total number of points are [96, 192, 288], where the ray multiplicity is fixed to 3. Left/right column correspond to 60◦ /120◦ field of view, respectively

136

8 Applications of Anisotropic Procrustes Analysis median error (96−120)

1.5

median error [%]

median error [%]

median error (96−60)

1

0.5

1.5

1

0.5

0

0 20

20

18 10

camera distance

2

54

18 10

36

points per image

camera distance

2

54

points per image

median error (3−120)

median error (3−60)

1.5

median error [%]

median error [%]

36

1

0.5

0

1.5

1

0.5

0 20

18 10

camera distance

36 2

54

points per image

20

18 10

camera distance

36 2

54

points per image

Fig. 8.23 Median RMS error of PBBA as a function of camera distance d and number of tie-points visible in each image p. The error is in percentage with reference to points distributed on a unit sphere. Top row: Corresponding ray multiplicity values are [3, 6, 9], where n = 96. Bottom row: Corresponding total number of points are [96, 192, 288], where the ray multiplicity is fixed to 3. Left/right column correspond to 60◦ /120◦ field of view, respectively

It also worsen when switching from a field of view of 60◦ (left column) to 120◦ (right column). A final but important remark: although the convergence of PBBA is slow near the optimum, only few iterations are needed to dramatically reduce the error and obtain approximately correct depths (see Fig. 8.19 which is a paradigmatic example). This might suggest a combined use of PBBA and photogrammetric bundle block adjustment: PBBA is run for some iterations (≈20) and then photogrammetric bundle block adjustment takes on and quickly converges to the minimum. As for the computational cost, one iteration of photogrammetric bundle block adjustment requires approximately the same time spent in 7.4 iterations of PBBA,

8.3 Solving the Bundle Block Adjustment Problem with AGPA

137

Fig. 8.24 Running time versus number n of tie-points (m = 16, d = 10, ray multiplicity = 6) for PBBA and Fast PBBA

considering a MATLAB implementation of both running on the simulated data of Fig. 8.18. As shown in (Fusiello and Crosilla 2016), the same results in terms of accuracy and convergence rate can be obtained substituting the PBBA algorithm with the Fast PBBA one. It is interesting to report here the experiment aimed at assessing the speedup of Fast PBBA with respect to PBBA (both methods are implemented in MATLAB). In this case, the parameters were set to: m = 16, d = 10 and the ray multiplicity was set to 6. The number of points n was increased from 32 to 144, and the average running time over 20 independent trials was recorded. The results are illustrated in Fig. 8.24, where the reader can appreciate that the acceleration is at least of one order of magnitude. Experiments were carried out also to better evaluate PBBA in real applications. In particular, one close-range experiment and one aerial (UAV) application (Fig. 8.25) were selected. Table 8.1 gives the main features of these datasets. In both cases image tie-points were automatically computed using the structure-from-motion pipeline proposed in (Farenzena et al. 2009; Gherardi et al. 2010).6 The MATLAB implementation of bundle block adjustment was taken from the Vision Lab Geometry Library.7

6 Code 7 Code

available from http://samantha.3dflow.net. available from http://vision.ucla.edu/vlg/.

138

8 Applications of Anisotropic Procrustes Analysis

Fig. 8.25 Left: one sample image of the Herz-Jesu-P25 set; Right: one sample image of the nearvertical Hessigheim set Table 8.1 Datasets features. Image size and cardinality, average depth, ground sampling distance (GSD) size avg. Z (m) GSD (mm) Herz-Jesu-P25 Hessigheim

3072 × 2048 × 25 4288 × 2848 × 15

3.4 242

5.2 61

The close-range data is described in (Strecha et al. 2008). In particular the HerzJesu-P25 set was used, composed by 25 images, which had been carefully oriented in the referenced work, using a laser scan as the ground truth. For the Herz-Jesu-P25 image-set (Fig. 8.26) the final re-projection RMS error is 0.54 pixels, whereas the PBBA error (average distance of back-projected points to their centroid) is 2.0 m. The RMS error with respect to the reference camera stations is 9.2 mm; Fig. 8.27 reports the error with respect to ground control points, both as distance in space and as coordinates differences. After running photogrammetric bundle block adjustment, this error improves only slightly, reaching 8.6 mm; this is not relevant though, for both measures are well within the uncertainty affecting the measured camera stations (reported in Strecha et al. 2008). Figure 8.28 shows the evolution of the error and depth values of points in the first image as the algorithm iterates. The Hessigheim data8 is described in (Cramer 2013). In particular, only a small subset of those data, corresponding to 15 images, was used to evaluate PBBA. Interior orientation was obtained by auto-calibration, as part of the structure-from-motion pipeline which provided the image tie-points. For the Hessigheim image-set (Fig. 8.29) the final re-projection RMS error is 0.56 pixels, whereas the PBBA error (average distance of back-projected points to their centroid) is 29 mm. The RMS error with respect to ground control points is 15.5 cm; Fig. 8.30 reports the errors for each image and each dimension. After running photogrammetric bundle block adjustment, this error reaches 15.2 cm. This improvement 8 Courtesy

of N. Haala.

8.3 Solving the Bundle Block Adjustment Problem with AGPA

139

0 10 20 0

50

100

150

200

rays: [15, 23]

Fig. 8.26 Top: A depiction of points and camera stations of the Herz-Jesu-P25 image-set; Bottom: the visibility matrix, showing which point (x-axis) is visible in which image (y-axis). The text below this matrix reports the minimum and maximum ray multiplicity

however has little significance, if compared with the scale of the problem. Convergence can be appreciated in Fig. 8.31. Please note that, in accordance with the remark on the global scale already made before, in both cases PBBA starts with Z = 1 even if actual average depths are completely different (3.4 and 242 m respectively). The rundown of these experiments on real data is that the qualitative behavior of PBBA on simulated data is confirmed in all regards.

8.3.3 Robust Procrustes Bundle Adjustment In real-world applications one must consider that, when tie-points have been obtained automatically—a very common trend (e.g., Hartmann et al. 2015) borrowed from the structure-from-motion literature—some of them are rogue points that would skew the Procrustes least square estimate, if proper countermeasures are not taken. This is particularly dangerous since the Procrustes procedure is fully automatic and does not require any a priori information about the unknowns. Resistance to outliers can be obtained, following (Verboon and Heiser 1992), by substituting the least squares cost function of the classical Procrustes method with specific robust cost functions, and solving the resulting minimization problem with Iteratively Reweighted Least Squares (IRLS) (Holland and Welsch 1977).

140

8 Applications of Anisotropic Procrustes Analysis Distance to true camera positions 0.06

[m]

0.04 0.02 0 0

5

10

15

20

25

Difference with true camera positions on each dimension 0.02

x [m]

0.015 0.01 0.005 0 0

5

10

15

20

25

5

10

15

20

25

5

10

15

20

25

y [m]

0.03 0.02 0.01 0 0

z [m]

0.06 0.04 0.02 0 0

Fig. 8.27 Top: Distance of computed to reference camera stations for the Herz-Jesu-P25 image-set. Bottom: Differences on each dimension (X-Y-Z)

8.3 Solving the Bundle Block Adjustment Problem with AGPA

141

Convergence of the internal error

−1

(log) internal error

10

−2

10

−3

10

−4

10

0

10

2

1

10

10

3

10

(log) iterations

Convergence of Z−values in image 1 1.6 1.5 1.4

depth (Z)

1.3 1.2 1.1 1 0.9 0.8 0.7 0 10

1

2

10

10

3

10

(log) iterations

Fig. 8.28 Convergence of error (left) and depth values (right) for the Herz-Jesu-P25 image-set. Please note that some axes are logarithmic

142

8 Applications of Anisotropic Procrustes Analysis

0 10 0

40

20

60 rays: [6, 14]

80

100

Fig. 8.29 Top: A depiction of points and camera stations of the Hessigheim image-set; Bottom: the visibility matrix, showing which point (x-axis) is visible in which image (y-axis). The text below this matrix reports the minimum and maximum ray multiplicity

This technique iteratively solves weighted least squares problems where the weights are estimated at each iteration by analyzing the residuals of the current solution. The weights are assigned by a specific weighting function in such a way to penalize outliers and promote inliers. For the bundle block adjustment, the weighted cost function can be written as: m   p    p tr Qi − S  PW Mi Qi − S i=1

where: • Qi = Zi Qi Ri + jOi  ; • S is the centroid defined as p

S=

 m  i=1

−1 Mi

m  i=1

p

Mi Qi ;

(8.3.4)

8.3 Solving the Bundle Block Adjustment Problem with AGPA

143

Distance to ground control points 0.8

[m]

0.6 0.4 0.2 0

1

2

3

4

5

6

Difference with ground control points on each dimension 0.2

x [m]

0.15 0.1 0.05 0

1

2

3

4

5

6

1

2

3

4

5

6

1

2

3

4

5

6

0.2 y [m]

0.15 0.1 0.05 0

0.4

z [m]

0.3 0.2 0.1 0

Fig. 8.30 Top: Distance of computed 3D points to ground control points for the Hessigheim imageset. Bottom: Differences on each dimension (X-Y-Z)

144

8 Applications of Anisotropic Procrustes Analysis Convergence of the internal error

−2

(log) internal error

10

−3

10

−4

10

0

10

1

10

2

10 (log) iterations

3

10

4

10

Convergence of Z−values in image 1 1.25 1.2 1.15

depth (Z)

1.1 1.05 1 0.95 0.9 0.85 0.8 0 10

1

10

2

10 (log) iterations

3

10

4

10

Fig. 8.31 Convergence of error (top) and depth values (bottom) for the Hessigheim image-set. Please note that some axes are logarithmic

8.3 Solving the Bundle Block Adjustment Problem with AGPA

145

• PW is a diagonal matrix containing the positive weights PW ( j, j) for each point; • Mi is a diagonal matrix, defined for each matrix Qi , containing unit values along the main diagonal in case the corresponding row of Qi is specified, and zero on the contrary (in order to take into account that not all points are visible in all the images). The weighted version of the problem can be solved (Fusiello and Crosilla 2016) adopting the same iterative procedure of the Fast PBBA previously described, based on AGPA, with the following substitutions: 1/2

Qi ← Ni Qi S ← Ni S j ← Ni j with Ni = Mi PW . Please note that the matrix PW is not defined for each exposure, meaning that outlyingness is conservatively attributed to tie-points, as opposed to their image projections. Several weighing functions have been proposed in the literature; the ones offering greatest resistance to outliers are the so-called hard redescenders, that assign zero weight to points with residual higher than a threshold. Among them, one can choose the popular bisquare (a.k.a. biweight) function (Mosteller and Tukey 1977):

PW ( j, j) =

⎧ ⎪ ⎨ ⎪ ⎩0

1−

 r 2 2 j

k

if |r j /k| ≤ 1

(8.3.5)

otherwise

where the tuning constant k is assigned, as customary, so as to yield a reasonably high efficiency in the normal case, and still offers protection against outliers. In particular, k = 4.685σ produces 95-percent efficiency when the errors are normal with standard deviation σ . The latter is estimated robustly from the median absolute deviation (MAD) of the residuals as σ = MAD/0.6845. In the bundle block adjustment case the residuals are defined as: rj =

m 

p

Mi ( j, j) S( j, :) − Qi ( j, :) 22 j = 1 . . . n

(8.3.6)

i=1 p

p

where S( j, :) (resp. Qi ( j, :)) denotes the jth row of S (resp. Qi ). Please note that: p

(i) the term S( j, :) − Qi ( j, :) correspond to i j as defined in Fig. 8.17; n m  p  p     rj = tr Qi − S  Mi Qi − S . (ii) j=1

i=1

The Robust (or Resistant) PBBA (RPBBA) is summarized in Algorithm 8. It is worth noting that the computation of the centroid S does not involve the IRLS weights PW but only Mi , and that Zi does not depend on the weights at all, as all computations, although in matrix form, are essentially point-wise, so any unspecified value in the jth row of Qi only reflects the jth diagonal value of Zi that should be

146

8 Applications of Anisotropic Procrustes Analysis

Algorithm 8 RPBBA Input: a set of models Qi , Mi i = 1 . . . m Output: translation Oi , attitude Ri and scale Zi of each model p

1. Initialize PW = I and Zi = I, Qi = Qi ∀i 2. Solve AGPA: m −1 m   p a. Compute centroid S = Mi Mi Qi i=1

i=1

b. Register each model Qi to S: 1/2

i. Let Ni = Mi PW and ni = diag-1 (Ni )   ii. Compute Ri = V diag 1,1,det(VW ) W    with VDW = Qi Ni Zi (I−(ni ni )/(ni  ni ))Ni S iii. Compute Oi = (Ni S − Zi Ni Qi Ri )  ni /(ni  ni ) iv. Compute Zi =(Qi Qi  I)−1 (Qi Ri (S −Oi j )I) where  denotes the Hadamard product p v. Update Qi = Zi Qi Ri + jOi  vi. Compute residuals m  p rj = Mi ( j, j) S( j, :) − Qi ( j, :) 22 j = 1 . . . n i=1

c. Iterate from step (a) until convergence of residual

n 

rj

j=1

  2 2 3. Compute weights PW ( j, j) = (|r j /k| ≤ 1) · 1 − r j /k 4. Iterate from step 2. until convergence of weights.

ignored. For the same reason it does not make sense to down-weight outliers in the computation of Zi .

8.3.3.1

Experimental Validation

RPBBA was tested on repeated trials with simulated data, in order to assess its resistance to outliers (Fusiello and Crosilla 2016). The same validation protocol as in Sect. 8.3.2 was followed. For each trial, n = 96 3D tie-points were randomly generated in a sphere of unit radius centered on the origin. Sixteen random cameras (m = 16) looking toward the origin were positioned in a 60◦ sector of the sphere, at an average distance of d = 10 units from the origin. The focal length was chosen so as to yield a view angle of {60◦ , 120◦ } with an image size of 1000 × 1000 pixels. Random noise with unit variance was added to the image coordinates. Missing points were simulated by zeroing random elements in the visibility matrix. Outliers were generated with uniform random coordinates in [−500, 500]. An increasing number of outliers was added and 100 trials for each contamination level were run. The results, reported in Fig. 8.32, indicate a breakdown point at 10%.

8.4 Summary

147

0.7 Mean Median

0.6

RMS error

0.5

0.4

0.3

0.2

0.1

0 0

2

4

6

8

10

12

14

16

18

% of outliers

Fig. 8.32 Mean and median RMS error versus percentage of outliers (m = 16, n = 96, d = 10, p = 36). The RMS error is in percentage with reference to points distributed on a unit sphere

8.4 Summary In this chapter, we described the Procrustes solution of classical photogrammetric computer vision problems, namely Perspective-n-Point (or exterior orientation), Non-Perspective-n-Point and bundle block adjustment. Whereas classical Orthogonal Procrustes Analysis seeks transformations in 3D space, the anisotropic extension allows to deal with problems involving a projection to 2D space by introducing an auxiliary unknown, the depth of the points, which makes it possible to back-project 2D points into 3D space, thereby restoring the 3D problem. Results obtained with Procrustes algorithms in photogrammetric computer vision are comparable in terms of accuracy with those given by state-of-the-art methods.

References E.F. Ackermann, Some results of an investigation into the theoretical precision of planimetric block adjustment. Photogrammetria, 19(8) (1962) D. Akca, Generalized procrustes analysis and its applications in photogrammetry. Internal Technical Report at the Institute of Geodesy and Photogrammetry - ETH, Zurich (2003) A. Ansar, K. Daniilidis, Linear pose estimation from points or lines. IEEE Trans. Pattern Anal. Mach. Intell. 25(5), 578–589 (2003)

148

8 Applications of Anisotropic Procrustes Analysis

M. Arie-Nachimson, S.Z. Kovalsky, I. Kemelmacher-Shlizerman, A. Singer, R. Basri. Global motion estimation from point matches. Proceedings of the Joint 3DIM/3DPVT Conference: 3D Imaging, Modeling, Processing, Visualization and Transmission (2012) A.C. Atkinson and M. Riani. Robust Diagnostic Regression Analysis. Springer Series in Statistics. Springer, 2000. ISBN 9780387950174 W. Baarda, S-transformations and criterion matrices. Netherlands Geodetic Commission, Publications on Geodesy, New series, 5(1) (1973) P.L. Baetsle. Compensation des blocs photogrammetriques en altimetrie par relaxation. Bull. Soc. Belg. Photogramm. 4 (1956) I. Ben-Gal. Outlier detection, in Data Mining and Knowledge Discovery Handbook, chapter 7, pages 131–146–146. Springer US, 2005 M. Brand, Incremental singular value decomposition of uncertain data with missing values, in Proceedings of the European Conference on Computer Vision (Springer, 2002), pp. 707–720 M. Brand, M. Antone, S. Teller. Spectral solution of large-scale extrinsic camera calibration as a graph embedding problem, in Proceedings of the European Conference on Computer Vision (2004) D. Brown, On first and second order partitioned regression. Internal working paper, DBA Systems Inc (1968) D. Brown. A solution to the general problem of multiple station analytical stereotriangulation. Technical Report AFMTC TR 58-8, Patrick Airforce Base, Florida (1958) C.-S. Chen, W.-Y. Chang, On pose recovery for generalized visual sensors. IEEE Trans. Pattern Anal. Mach. Intell. 26(7), 848–861 (2004) J.J.F. Commandeur. Matching Configurations (DSWO Press, 1991) M. Cramer. The UAVLGL BW project - a NMCA case study, in Photogrammetric Week ’13, pp. 165–179. Wichmann, Berlin/Offenbach (2013) M. Cunietti. Use of strips connected to blocks for large scale mapping. Results of Experimental Research Organized by Commission B of the OEEPE from 1959 through 1966 - Frankfurt a.M. (1968) L. Davies, U. Gather, The identification of multiple outliers. J. Am. Stat. Assoc., 88(423), 782–792 (1993). ISSN 01621459 J. de Leeuw. Block-relaxation algorithms in statistics, in Information Systems and Data Analysis (Springer-Verlag, 1994) pp. 308–325 D.F. DeMenthon, L.S. Davis, Model-based object pose in 25 lines of code. Int. J. Comput. Vis. 15, 123–141 (1995) M. Farenzena, A. Fusiello, R. Gherardi, Structure-and-motion pipeline on a hierarchical cluster tree, in IEEE International Workshop on 3-D Digital Imaging and Modeling, pp. 1489–1496 (October 3-4 2009) S. Finsterwalder, W. Scheufele, Das Rückwärtseinschneiden im Raum (Verlag der K, Akademie, 1903) P.D. Fiore, Efficient linear solution of exterior orientation. IEEE Trans. Pattern Anal. Mach. Intell. 23(2), 140–148 (2001) M.A. Fischler, R.C. Bolles, Random Sample Consensus: a paradigm model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981) W. Förstner, The reliability of block triangulations. Photogramm. Eng. Remote. Sens. 51(8), 1137– 1149 (1985) A. Fusiello, F. Crosilla, Fast and resistant procrustean bundle adjustment. ISPRS Ann. Photogramm., Remote. Sens. Spat. Inf. Sci., III-3, 35–41 (2016) A. Fusiello, F. Crosilla, Solving bundle block adjustment by generalized anisotropic procrustes analysis. ISPRS J. Photogramm. Remote. Sens., 102, 209–221 (2015). ISSN 0924-2716 A. Fusiello, F. Crosilla, F. Malapelle, Procrustean point-line registration and the npnp problem, in Proceedings of the International Conference on 3D Vision (3DV), pp. 250–255. IEEE (2015) A. Fusiello, E. Maset, F. Crosilla, Reliable exterior orientation by a robust anisotropic orthogonal procrustes algorithm, in Proceedings of the ISPRS Workshop 3D-ARCH 2013: 3D Virtual Reconstruction and Visualization of Complex Architectures, volume XL-5/W1 of ISPRS Archives, pp. 81–86 (2013)

References

149

A. Fusiello, A matter of notation: several uses of the kronecker product in 3d computer vision. Pattern Recognit. Lett. 28(15), 2127–2132 (2007) X.-S. Gao, X.-R. Hou, J. Tang, H.-F. Cheng, Complete solution classification for the perspectivethree-point problem. IEEE Trans. Pattern Anal. Mach. Intell. 25, 930–943 (2003) V. Garro, F. Crosilla, A. Fusiello, Solving the pnp problem with anisotropic orthogonal procrustes analysis, in Second Joint 3DIM/3DPVT Conference: 3D Imaging (Modeling, Processing, Visualization and Transmission, 2012), pp. 262–269 R. Gherardi, M. Farenzena, A. Fusiello, Improving the efficiency of hierarchical structure-andmotion, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1594 – 1600 (13-18 June 2010) J.C. Gower, G.B. Dijksterhuis, Procrustes Problems (Oxford University Press, Oxford Statistical Science Series, 2004) A.W. Gruen, E.P. Baltsavias, Adaptive least squares correlation with geometrical constraints, in Computer Vision for Robots, vol. 0595 of Proceedings of SPIE, pp. 72–82 (1986) A.W. Gruen, Precision and reliability aspects in close range photogrammetry. Int. Arch. Photogrammetry 11(23B), 378–391 (1980) G.A. Grunert, Das pothenotische problem in erweiterter gestalt nebst über seine anwendungen in geodäsie. Archiv für Mathematik und Physik (1841) M.S. Gyer, The inversion of the normal equations of analytical aerotriangulation by the method of recursive partitioning. Technical Report RADC-TR-67-69, Rome Air Development Center, NY (1967) A.S. Hadi, J.S. Simonoff, Procedures for the identification of multiple outliers in linear models. J. Am. Stat. Assoc., 88(424), 1264–1272 (1993). ISSN 01621459 R.M. Haralick, H. Joo, C. Lee, X. Zhuang, V.G. Vaidya, M.B. Kim, Pose estimation from corresponding point data. IEEE Trans. Syst., Man Cybern. 19(6), 1426–1446 (1989) R.M. Haralick, C.-N. Lee, K. Ottenberg, M. Nölle, Review and analysis of solutions of the three point perspective pose estimation problem. Int. J. Comput. Vis. 13, 331–356 (1994) R.I. Hartley, F.k Schaffalitzky, PowerFactorization: 3D reconstruction with missing or uncertain data, in Australia-Japan advanced workshop on computer vision, vol. 74, pp. 76–85 (2003) R.I. Hartley. Minimizing algebraic error, in Proceedings of the IEEE International Conference on Computer Vision (1998) R.I. Hartley, J. Trumpf, Y. Dai, H. Li, Rotation averaging. Int. J. Comput, Vis. (2013) W. Hartmann, M. Havlena, K. Schindler, Recent developments in large-scale tie-point matching. ISPRS J. Photogramm. Remote. Sens. 115, 47–62 (2015). ISSN 0924-2716 J.A. Hesch, S.I. Roumeliotis, A direct least-squares (dls) solution for PnP, in Proceedings of the International Conference on Computer Vision (2011) A. Heyden, Projective structure and motion from image sequences using subspace methods, in Scandinavian Conference on Image Analysis, pp. 963–968 (1997) P.W. Holland, R.E. Welsch, Robust regression using iteratively reweighted least-squares. Commun. Stat.-Theory Methods 6(9), 813–827 (1977) R. Horaud, F. Dornaika, B. Lamiroy, Object pose: the link between weak perspective, paraperspective, and full perspective. Int. J. Comput. Vis. 22, 173–189 (1997) N. Jiang, Z. Cui, P. Tan, A global linear method for camera pose registration, in Proceedings of the International Conference on Computer Vision (2013) F. Kahl, R.I. Hartley, Multiple-view geometry under the l∞ -norm. IEEE Trans. Pattern Anal. Mach. Intell. 30(9), 1603–1617 (2008) R. Kaucic, R. I. Hartley, N.Y. Dano, Plane-based projective reconstruction, in Proceedings of the International Conference on Computer Vision, pp. 420–427 (2001) R. Kennedy, L. Balzano, S.J. Wright, and C.J. Taylor. Online algorithms for factorization-based structure from motion. CoRR, abs/1309.6964 (2013) L. Kneip, P. Furgale, R. Siegwart. Using multi-camera systems in robotics: Efficient solutions to the NPnP problem, inProceedings of the IEEE International Conference on Robotics and Automation (2013)

150

8 Applications of Anisotropic Procrustes Analysis

L. Kneip, H. Li, and Y. Seo. UPnP: An optimal O(n) solution to the absolute pose problem with universal applicability, in 13th European Conference on Computer Vision, pp. 127–142 (2014) L. Kneip, D. Scaramuzza, R. Siegwart. A novel parametrization of the perspective-three-point problem for a direct computation of absolute camera position and orientation, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2011) K. Kraus, Photogrammetry: Advanced methods and applications, volume 2. Dümmler (1997). ISBN 9783427786948 R. Kumar, A.R. Hanson, Robust methods for estimating pose and a sensitivity analysis. CVGIP: Image Understanding, 60, 313–342 (1994) V. Lepetit, F. Moreno-Noguer, P. Fua, Epnp: an accurate O(n) solution to the pnp problem. Int. J. Comput. Vis. 81(2), 155–166 (2009) J. Lingoes, I. Borg, A direct approach to individual differences scaling using increasingly complex transformations. Psychometrika 43(4), 491–519 (1978) D.G. Lowe, Fitting parameterized three-dimensional models to images. IEEE Trans. Pattern Anal. Mach. Intell. 13(5), 441–450 (1991) C. Lu, Gregory, D. Hager, E. Mjolsness, Fast and globally convergent pose estimation from video images. IEEE Trans. Pattern Anal. Mach. Intell., 22, 610–622 (2000) D. Martinec, T. Pajdla, Robust rotation and translation estimation in multiview reconstruction, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2007) E. Maset, F. Crosilla, A. Fusiello, Errors-in-variables anisotropic extended orthogonal procrustes analysis. IEEE Geosci. Remote. Sens. Lett. 14(1), 57–61 (2017) F. Mosteller, J.W. Tukey, Data analysis and regression: a second course in statistics. AddisonWesley Series in Behavioral Science: Quantitative Methods (1977) P. Moulon, P. Monasse, R. Marlet, Global fusion of relative motions for robust, accurate and scalable structure from motion, in Proceedings of the International Conference on Computer Vision, pp. 3248–3255 (2013) D. Nistér, An efficient solution to the five-point relative pose problem. IEEE Trans. Pattern Anal. Mach. Intell. 26(6), 756–777 (2004) J. Oliensis and R.I. Hartley, Iterative extensions of the sturm/triggs algorithm: Convergence and nonconvergence. IEEE Trans. Pattern Anal. Mach. Intell., 29(12), 2217–2233 (2007). ISSN 01628828 J. Oliensis, Fast and accurate self-calibration, in Proceedings of the International Conference on Computer Vision (1999) L. Quan, Z. Lan, Linear n-point camera pose determination. IEEE Trans. Pattern Anal. Mach. Intell. 21(8), 774–780 (1999) C. Rother, S. Carlsson, Linear multi view reconstruction and camera recovery using a reference plane. Int. J. Comput. Vis. 49(2–3), 117–141 (2002) P.J. Rousseeuw, Least median of squares regression. J. Am. Stat. Assoc., 79(388), 871–880 (1984). ISSN 01621459 G. Schweighofer, A. Pinz, Globally optimal o(n) solution to the pnp problem for general camera models, in Proceedings of the British Machine Vision Conference, pp. 55.1–55.10 (2008) C.V. Stewart, Robust parameter estimation in computer vision. SIAM Rev. 41(3), 513–537 (1999) C. Strecha, W. Von Hansen, L. Van Gool, P. Fua, U. Thoennessen, On benchmarking camera calibration and multi-view stereo for high resolution imagery, in IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008) J. F. Sturm, Using SEDUMI 1.02, a MATLAB∗ toolbox for optimization over symmetric cones, 2001. Downloadable from http://sedumi.mcmaster.ca P. Sturm, B. Triggs, A factorization based algorithm for multi-image projective structure and motion, in Proceedings of the European Conference on Computer Vision, pp. 709–720 (1996) P. Sturm, S. Ramalingam, J.-P. Tardif, S. Gasparini, J. Barreto, Camera models and fundamental concepts used in geometric computer vision. Found. Trends Comput. Graph. Vis. 6, 1–183 (2011) C. Sweeney, V. Fragoso, T. Höllerer, M. Turk, gDLS: A scalable solution to the generalized pose and scale problem, in 13th European Conference on Computer Vision, pp. 16–31 (2014)

References

151

C. Sweeney. Theia Multiview Geometry Library: Tutorial & Reference. University of California Santa Barbara (2015). http://www.theia-sfm.org/ B. Triggs, P. F. McLauchlan, R. I. Hartley, and A. W. Fitzgibbon. Bundle adjustment - a modern synthesis, in Proceedings of the International Workshop on Vision Algorithms (Springer-Verlag, 2000), pp. 298–372. ISBN 3-540-67973-1 J. Ventura, C. Arth, G. Reitmayr, D. Schmalstieg. A minimal solution to the generalized pose-andscale problem, in Computer Vision and Pattern Recognition, Conference on (2014) P. Verboon, W.J. Heiser. Resistant orthogonal procrustes analysis. J. Cl.Ification, 9(2), 237–256 (1992). ISSN 0176-4268 K. Wilson, N. Snavely, Robust global translations with 1DSfM, in Proceedings of the European Conference on Computer Vision, pp. 61–75 (2014) J. Zhang, M. Boutin, D.G. Aliaga, Robust bundle adjustment for structure from motion, in Proceedings of the International Conference on Image Processing, pp. 2185–2188 (2006)

Chapter 9

3D Model Registration by Generalized Procrustes Analysis

Photogrammetric computer vision techniques and laser scanning systems can directly provide 3D models of real objects by automatically or selectively sampling the positions of a set of representative surface points. Depending on the dimension and on the shape complexity of the geometric entity under study, its complete survey often requires a multiple view approach that leads to the creation of a set of partial and independent 3D models of the same object. These parts must be then joined together to reconstruct the complete object model. As already discussed in Chap. 7, this problem is known as registration and consists of transforming the point coordinates of the different models, expressed in their own reference system, into a unique common frame. If pairs of adjacent point clouds overlap, pairwise registration can be performed by means of different methods that can be divided into two main categories: techniques based on the matching of single points (or features) extracted from models, and those that directly align portions of corresponding surfaces. The two approaches are not equivalent, and they both have advantages and disadvantages, as well as application restrictions. The first techniques, largely employed in photogrammetry (e.g., to perform block adjustment by independent models) and terrestrial laser scanning (for point cloud registration), require the selection of a set of corresponding points (tie and control points) on adjacent models. Being based on a limited number of points, the registration is extremely sensitive to matching errors. The latter instead, among which we can mention the Iterative Closest Point (ICP) algorithm (Chen and Medioni 1991) and its variants (e.g., Salvi et al. 2007), are often implemented in computer vision. Although high accuracy can be reached through these methods, surface alignment still requires an initial manual superimposition. Furthermore, the alignment cannot be carried out on regular (e.g., flat) surfaces. Another disadvantage is that ICP algorithm cannot estimate scale factors and conseCo-authored with Roberto Toldo, University of Verona (Italy). © CISM International Centre for Mechanical Sciences 2019 F. Crosilla et al., Advanced Procrustes Analysis Models in Photogrammetric Computer Vision, CISM International Centre for Mechanical Sciences 590, https://doi.org/10.1007/978-3-030-11760-3_9

153

154

9 3D Model Registration by Generalized Procrustes Analysis

quent model resizing. To address this limitation, a specific ICP extension has been proposed (Du et al. 2007). This chapter highlights the potentiality of Generalized Procrustes Analysis, described in Chap. 3, to directly perform the global registration of multiple 3D models when points on adjacent views are manually selected and matched (Beinat and Crosilla 2001; Crosilla and Beinat 2002). Moreover, we will show that the Closest Point variant of the GPA, presented in Sect. 3.6, allows to register multiple point clouds when correspondences are missing, embedding the matching step within the GPA iterative process (Toldo et al. 2010).

9.1 Solving the Independent Models Block Adjustment Problem with GPA As already introduced in Sect. 6.3.3, the Block Adjustment by Independent Models is a common method in photogrammetry. It consists in the creation (by relative orientation) of individual stereomodels described in a local reference frame that are subsequently merged into a single one and simultaneously transformed into the ground coordinate system. It is easy to see that this problem can be solved with the Generalized Procrustes Analysis (GPA) procedure (Crosilla and Beinat 2002), and in particular resorting to the solution presented in Sect. 3.4 that can cope also with missing data, namely points that are not seen in all the views. More in detail, the GPA algorithm is applied taking into account the following considerations: • the number of Ai matrices involved in the computation is equal to the number of models composing the block adjustment. The matrices contain the coordinates of the available points (perspective centers, control and tie-points) for each model; • matrices Ai have the same dimensions, equal to the global number of the block adjustment observation times the coordinate space dimension (usually 3); • the generic matrix Ai has nonzero elements only in the rows that correspond to the points belonging to the ith model (see Fig. 9.1). According to the theoretical model of the GPA presented in Chap. 3, Crosilla and Beinat (2002) implemented a procedure to perform the block adjustment by independent models, articulated in the following phases. Input Data The following data, related to the various models, are required as input to the solving procedure: 1. the univocal IDs for each tie and perspective center point composing the model; 2. their Cartesian coordinates defined in the corresponding model frame; 3. the weight values (if different from the unitary ones). Similar data are also provided for the list of control points.

9.1 Solving the Independent Models Block Adjustment Problem with GPA

155

Fig. 9.1 Matrices used in the GPA process to solve an independent model block adjustment problem

Fig. 9.2 Starting situation for the bundle adjustment scheme by independent models

Chaining of the models The starting situation is represented in Fig. 9.2, that considers, for simplicity, four adjacent models. Since the various models are formed in the coordinate system of the plotter, when they are altogether represented, they appear superimposed. A chaining process is required to reconstruct the reciprocal 3D location of the various models. Assuming one of the models as fixed, e.g., A1 , all the others are incrementally linked and oriented with respect to it by the WEOPA algorithm (Sect. 2.5). Instead of sequentially registering pairs of single models, every model (when possible) is oriented considering the topological union of all the previously oriented ones (Fig. 9.3). Schematically, the registration sequence is:

156

9 3D Model Registration by Generalized Procrustes Analysis

Fig. 9.3 Chaining of the models

A2 ⇒ [A1 ]fixed   A3 ⇒ A2P ∪ A1   A4 ⇒ A3P ∪ A2P ∪ A1 . . . etc. The approximated shape of the whole object obtained in this way provides an initial value of the centroid represented by the assembly of the (weighted) average positions of the corresponding tie-points. Approximate coordinate transformation The next step is to carry out a global similarity transformation of all the point coordinates using the available control points (see Fig. 9.4). In order to consider also control points whose only available component is the elevation, the global transformation is carried out by applying EOPA with missing values (these last corresponding to the missing planimetric components), described in Sect. 2.6. This process leads to points having coordinate values expressed in the absolute system, near to the final values. The aim of the chaining procedure and of the approximate coordinate transformation is to correctly define the geometric configuration of the centroid. Adjustment At every iteration, all the models of the block are independently rotated, translated and scaled, in such a way to locally adapt them to the temporary centroid. When a model is reallocated and the point coordinates are updated, such variation immediately conditions the definition of the centroid for the next model. In fact, the centroid is constituted by variable components given by all the tie-points, and by fixed components, constituted by the control points. Their presence, possibly weighted, is equivalent to the constraining role assumed by the pseudo-observation equations of the control point coordinates within the conventional algebraic scheme of the block adjustment. Figure 9.5 shows the relationship existing between the tie-points configuration of the different models (nodes belonging to the quadrilaterals) and the corresponding vertices of the centroid (nodes joined by the dotted lines). Control points are not evidenced.

9.1 Solving the Independent Models Block Adjustment Problem with GPA

157

Fig. 9.4 Approximate coordinates transformation according to the control points Fig. 9.5 Geometric relationship between the centroid and the corresponding tie-points

Output At the end of the adjustment the output results are: 1. 2. 3. 4.

adjusted values of all the observations in the absolute reference system; mean coordinates of the tie-points in the absolute reference system; residuals of the adjusted observations with respect to the mean value; absolute orientation parameters for each model, estimated by EOPA, between the initial and the final centroid coordinates of the points.

9.1.1 Experimental Validation The example reported in this section is related to a simple photogrammetric block adjustment by independent models, consisting of 11 models distributed on 2 strips (Crosilla and Beinat 2002). Figure 9.6 shows on the left the overlapped models with their initial point coordinates expressed in the reference frame of the plotter. The

158

9 3D Model Registration by Generalized Procrustes Analysis

Fig. 9.6 Initial configuration of the point models (on the left), and after the chaining procedure (on the right). The vertices coordinates are expressed in microns

same figure reports on the right the result of the chaining procedure. The reference model is also highlighted. The topology that relates the various models is automatically solved by assigning univocal names to the various object points. The so-joined models are transformed into the absolute system by a similarity. The transformation parameters are obtained by EOPA applied to the points of the control network whose coordinates are defined in the origin and destination systems (see Fig. 9.7). Figure 9.8 graphically shows the result of the adjustment, with the vertices of the triangles corresponding to the tie-points. This example was first carried out attributing equal weights to all the object points, and then changing the weights value for perspective centers, control and tie-points, respectively. For both kinds of experiments, a comparison with the corresponding solutions obtained by an ordinary least squares program is reported. Starting from the solutions with homogeneous weights, the first evaluation of the results was done on the computed coordinates of the tie-points. In Fig. 9.9, the residual values for the East (a), North (b), and Height (c) components obtained with the ordinary least squares solution and the GPA are shown. Tie-points are ordered according to their ID. A statistical control carried out for the residual values, that is for vec(E), where E is the difference matrix between the two solutions, shows a

9.1 Solving the Independent Models Block Adjustment Problem with GPA

159

CONTROL NETWORK

6

[m x 10 ]

5.064

5.0635

5.063

5.0625

5.062

1.5455

1.546

1.5465

1.547

1.5475

1.548

1.5485 6

[m x 10 ]

Fig. 9.7 Overlapping between the control point network and the corresponding net derived by the chaining of the models. Coordinates are expressed in meters in the map coordinate system

Fig. 9.8 Final configuration obtained after the adjustment. Coordinates are expressed in meter (in the map coordinate system)

160

9 3D Model Registration by Generalized Procrustes Analysis

[m]

0.050 0.040

Delta E

0.030 0.020 0.010 0.000 -0.010 -0.020 -0.030 -0.040

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61

-0.050

(a) Residuals for the East component [m]

0.050 0.040

Delta N

0.030 0.020 0.010 0.000 -0.010 -0.020 -0.030 -0.040

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61

-0.050

(b) Residuals for the North component [m]

0.050 0.040 0.030 0.020 0.010 0.000 -0.010 -0.020 -0.030 -0.040

Delta H

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61

-0.050

(c) Residuals for the Height component

Fig. 9.9 Residuals (in meters) between the ordinary LS and the Procrustes solutions with homogeneous weights for the East (a), North (b), and Height (c) coordinates of the tie-points

9.1 Solving the Independent Models Block Adjustment Problem with GPA

161

Table 9.1 Standard deviations of the residuals between the ordinary LS and the Procrustes solutions Solution σEst (m) σNord (m) σHeight (m) Ordinary least squares ±0.033 GPA ±0.036

±0.054 ±0.054

±0.047 ±0.051

normal distribution behavior with zero mean value and standard deviation equal to ±1.1 cm. The similarity transformation computed between the two solutions confirms the correspondence of the two results. In fact, one obtains: – Scale factor: 0.9999975 – Rotation matrix: ⎡ ⎤ 0.99999999997 −0.00000027651 −0.00000633473 ⎣ 0.00000027649 1.00000000000 −0.00000373583 ⎦ 0.00000633474 0.00000373582 0.99999999997 For what concerns the control points, the coordinate residuals with respect to their official value are practically the same for the two solutions, up to a few millimeters. Finally, as a global index of internal dispersion, the standard deviation values for the East, North and Height components, obtained by using the ordinary LS and the GPA algorithms are reported in Table 9.1. Analogous comparisons were carried out for the weighted solutions. The graphs of Fig. 9.10 show a behavior similar to the one illustrated in Fig. 9.9. Also in this case, the complete population of the residuals has a zero mean and a standard deviation equal to ±1.5 cm. The aim of the weighing is to offer a better capacity of attraction for the control points, in such a way to force the model composition to respect the shape of the control point network. At the same time, there exists the need to reduce the effect of the perspective center points, that usually have coordinates defined with less precision than those of the other points. The weight effect in the Procrustes block adjustment is similar to what expected, as evidenced by Fig. 9.11. This figure reports the residuals for the homogeneous solution and for the one obtained with different weights, with respect to the official coordinates of the control points. By applying the different weights, the standard deviation of the residuals is reduced from ±9 cm to ±6 cm.

9.2 Solving the Point Cloud Registration Problem with GPA In Chap. 7, we extensively described the problem of registering multiple point clouds, very common in photogrammetric computer vision and laser scanning. In order to

162

9 3D Model Registration by Generalized Procrustes Analysis

[m]

0.050 0.040 0.030 0.020 0.010 0.000 -0.010 -0.020 -0.030 -0.040

Delta E 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61

-0.050

(a) Residuals for the East component [m]

0.050 0.040

Delta N

0.030 0.020 0.010 0.000 -0.010 -0.020 -0.030 -0.040

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61

-0.050

(b) Residuals for the North component [m]

0.050 0.040 0.030 0.020 0.010 0.000 -0.010 -0.020 -0.030 -0.040

Delta H

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61

-0.050

(c) Residuals for the Height component

Fig. 9.10 Residuals (in meters) between the ordinary LS and the Procrustes solutions with heterogeneous weights for the East (a), North (b), and Height (c) coordinates of the tie-points

[m]

9.2 Solving the Point Cloud Registration Problem with GPA

163

0.250 0.200 0.150 0.100 0.050 14

13

12

11

10

9

8

7

6

5

4

3

2

1

0.000 -0.050 -0.100 -0.150 -0.200 Delta Est

-0.250

Delta Nord

Delta Quota

[m]

(a) GCP residuals with homogeneous weights 0.200 0.150 0.100 0.050

14

13

12

11

10

9

8

7

6

5

4

3

2

1

0.000 -0.050 -0.100 -0.150 Delta Est

Delta Nord

Delta Quota

-0.200

(b) GCP residuals with heterogeneous weights

Fig. 9.11 Residuals (in meters) of the control points coordinates from the Procrustes solution with homogeneous weights (a) and with heterogeneous weights (b)

apply GPA for its solution, one can resort to an approach, used, e.g., in Beinat and Crosilla (2001), that requires to manually select corresponding tie-points, since classical GPA assumes that correspondences among points are known in advance. However, this can represent a main limitation in several applications, where a manual identification of the correspondences can be unfeasible. This restriction can be overcome thanks to the Closest Point variation of the GPA (CP-GPA), described in Sect. 3.6, that embeds the correspondence estimation among point sets inside the classical framework of the GPA. In the following, examples of point cloud registration with both manual and automatic search of the point matches will be presented.

164

9 3D Model Registration by Generalized Procrustes Analysis

9.2.1 Global Registration with Known Correspondences This example, taken from Beinat and Crosilla (2001), refers to a global registration problem of LiDAR point clouds. Four range images of a van were acquired from four different viewpoints (Fig. 9.12) by means of a terrestrial laser scanning system. At first, a simple serial pairwise registration was performed. The manually selected tie-points are evidenced in the range images of Fig. 9.13 by square-contoured crosses. At first, assuming model B as reference (B = B R ), A was registered with respect to B, thus obtaining A R ; C over B, producing C R ; D on C R , finding D R . In this way, all the models were registered in the reference frame of B. To test the results, a set of five corresponding tie-points on D R and A R was selected and the corresponding coordinates were compared. Their differences were assumed as an index of the closure error, i.e., of the global registration. The average residual for the control set of five tie-points was −42.2 mm (σ = 50.3 mm), while the average residual of the full set of 22 tie-points was −9.6 mm (σ = 37.3 mm). The global registration of the point clouds was then carried out from scratch using GPA, with the same tie-points employed for the pairwise registration. At the end of the procedure, the average residual for the five check points was reduced to 0.6 mm (σ = 39.2 mm) while the average residual of the full set was −0.6 mm (σ = 30.3 mm). The obtained results confirm the quality of the GPA solution in terms of unbiasedness of residual distribution for the complete set of tie-points taken into account.

9.2.2 Embedding the Correspondence Estimation Inside GPA When the point correspondences are unknown, one can resort to CP-GPA, choosing an appropriate heuristic for the matching step. As highlighted in Sect. 3.6, CP-GPA can be considered a generalization of the Iterative Closest Point algorithm (ICP) (Besl and McKay 1992), the most common solution in literature to align two point sets.

Fig. 9.12 Viewpoints on the object scene

9.2 Solving the Point Cloud Registration Problem with GPA

(a) Model A

(c) Model C

165

(b) Model B

(d) Model D

Fig. 9.13 Four range images of the van obtained by a laser scanner, and the tie-points selected for the registration

In Toldo et al. (2010) correspondences are assigned at every step of the GPA by first finding the closest neighbor of each point. To improve the efficiency of the method, only at a subset of compatible matches is then used, as already suggested in Rusinkiewicz and Levoy (2001). Instead of detecting compatible matches by fixing a threshold on the distance between corresponding points, only the matches of points that are mutually nearest neighbour are considered. A point x ∈ A is mutually nearest neighbor to another point y ∈ B, if the closest neighbor ∈ B of x is y and the closest neighbor ∈ A of y is x (Fig. 9.14a). This is a simple yet powerful approach to discard pairs of points that are very far each other without introducing any threshold. It is in fact more probable that the mutual nearest neighbor relation is verified for closer matches. Let us build a graph, considering every pair of points that are in a mutual nearest neighbor relation. The edges of the graph are then represented by the mutual nearest neighbor relations. In this way, the graph is composed by connected components, as shown in Fig. 9.14b. Each connected component can be represented by a unique point in C, computed as the centroid of the points of the connected component. Every point of a connected component is then matched with the newly computed centroid in C (Fig. 9.14c). Using these correspondences, it is thus possible to update simultaneously

166

9 3D Model Registration by Generalized Procrustes Analysis

(a) Nearest Neighbour and Mutual Nearest Neighbour relations

(b) Connected components

(c) Centroids correspondences

Fig. 9.14 Example with three different clouds of bidimensional points (red, green, and black points). In the left figure, an arrow depicts a closest neighbor relation and a double-pointed arrow denotes a mutual closest neighbor relation. In the central image, the resulting independent sets are shown. The rightmost independent set contains two points from the green set and is thus discarded. In the right image, the arisen correspondences with the updated centroid (gray points - elements of C) of each connected component are shown

every transformation from the views to C. Empirically, points of different views belonging to the same connected component of the graph are matched with the same element in C and are thus driven into the same position. It may occur that more than one point of the same view is present in the same connected component; this is a rare degenerate case and the set is simply discarded. The general outline of the algorithm proposed in Toldo et al. (2010), called ICPGPA for its similarity with the ICP algorithm, is listed in Algorithm 9. Algorithm 9 ICP- GPA Input: m clouds of points [A1 . . . Am ] Output: m similarity transformations (rotation, scale and translation) 1. 2. 3. 4. 5.

For every view, find the pairs of points that are mutually nearest neighbours; Define the centroid of each connected component of mutually nearest neighbours; Estimate the similarity transformation between each view and the set of centroids; Transform each view using the estimated parameters; Iterate until global convergence or for a fixed number of steps.

Up to now, only unit-weighted correspondences have been considered. However, it is possible to easily apply a different weight for every match. At each step, a subset of points in a view is in correspondence with a subset of elements of C, which are computed as the centroids of the connected components. The natural solution is thus to define a different weight for every element of C that can be derived from a compatibility criteria among the points of each connected component.

9.2 Solving the Point Cloud Registration Problem with GPA

167

In Toldo et al. (2010), the Shape Index (S I ) was employed that represents a concise way to express the shape type of a region (Petitjean 2002):  k1 + k2 2 S I = − arctan π k1 − k2

k1 > k2

(9.2.1)

where k1 , k2 are the principal curvatures of a generic vertex. The curvature of a vertex is computed by fitting a quadric surface on the neighboring vertices. The Shape Index is bounded and varies in [−1, 1]: a negative value corresponds to concavities, whereas a positive value represents a convex surface. A weight is thus associated to each element of C and consequently to each correspondence. The weight is computed by making use of the Median of Absolute Deviations (MAD) among the points of each connected component (CC), defined as follows: MADCC = median(|S Ii − median(S I j )|)

∀i, j ∈ CC

(9.2.2)

The MAD varies in [0, 1]; the weight associated to each match is the inverse of the MAD, i.e., w = 1 − MADCC . The developed approach (ICP-GPA) and its weighted variant (ICP-WGPA) were tested (Toldo et al. 2010) with eight global alignment experiments: four composed of ten views and 10,000 points per view, and four obtained subsampling the views to 1000 points (simplified models). The initial configurations were misaligned to some extent. The views were produced with a laser scanner, and the subjects are a screw, a dog, a bunny, and a dinosaur. The method was compared with the classical ICP algorithm applied in a sequential way for each pair of subsequent views (ICP), and a variant where distant matches are rejected at each step according to the X84 rule (ICP-x84) (Hampel et al. 1986). The order of the views is know, but, unlike for the sequential ICP, is not needed for the proposed approach. For simplicity, the alignment results for the different methods are reported in Figs. 9.15 and 9.16 only for the bunny model. It can be noticed how both the weighted and non-weighted variants of the proposed method converge to global minima, whereas ICP and its variant ICP-x84 fail. The residual plots at each step of the algorithms for the bunny model (Fig. 9.16) were also analyzed. The residuals are computed as the mean square distance between the points of every view and their closest neighbor considering every other view. ICP-GPA and ICP-WGPA show in general a slightly faster convergence rate. In some experiments, the sequential ICP and its threshold variant converge to a solution that increases the global residual. Only in a few experiments the final residuals are comparable to the ICP-GPA, but never lower. The matching procedure proposed in Toldo et al. (2010) can be further refined by adopting multi-view matching algorithms, i.e., methods that jointly optimize the correspondences across the whole set, instead of computing matches between pairs

168

9 3D Model Registration by Generalized Procrustes Analysis

(a) Final configuration ICP

(b) Final configuration ICP-X84

(c) Final configuration ICP-GPA

(d) Final configuration ICP-WGPA

(e) Final configuration simplified ICP

(f) Final configuration simplified ICP-X84

(g) Final configuration simplified ICP-GPA

(h) Final configuration simplified ICP-WGPA

Fig. 9.15 Registration results for the bunny model

9.2 Solving the Point Cloud Registration Problem with GPA Fig. 9.16 Mean square distance residuals at each step of the algorithms for the bunny model

169

Error (10000 vertices per view)

40

ICP−GPA Weighted ICP−GPA ICP ICP X84

Mean Square Error

35 30 25 20 15 10 5 0 0

20

40

60

80

100

120

140

160

180

200

Step

of views in isolation (Zhou et al. 2015). A key concept in this context is that of cycle consistency, namely the composition of pairwise matches along any loop should give the identity. Since this property is not guaranteed by the method previously described, we propose a different heuristic for the matching step of CP-GPA, composed by the following phases: (a) Compute mutual closest points between pairs of views; (b) Apply the multi-view matching algorithm presented in Maset et al. (2017); (c) Connect point matching into tracks and discard the inconsistent ones. Preliminary experiments were carried out to test the developed procedure. In particular, we used the Bunny dataset from Stanford repository1 which contain ten point sets. Point clouds were initially randomly downsampled, in order to have approximately 1000 points per views. We perturbed the available ground-truth motions by a rotation with random axis and angle uniformly distributed over [0, α], with α = {5◦ , . . . , 20◦ }. In all cases, the proposed version of CP-GPA managed to register the views. For result visualization purposes, we report in Figs. 9.17 and 9.18 the alignments obtained with a noise of α = 5◦ and α = 20◦ , respectively.

1 http://graphics.stanford.edu/data/3Dscanrep/.

170

9 3D Model Registration by Generalized Procrustes Analysis

(b) CP-GPA

(a) Initial configuration

(c) Cross sections

Fig. 9.17 Results of the point cloud registration via CP-GPA. Initial configuration is obtained by perturbing the ground-truth motions by a rotation with random axis and angle uniformly distributed over [0, 5◦ ].

9.2 Solving the Point Cloud Registration Problem with GPA

171

(b) CP-GPA

(a) Initial configuration

(c) Cross sections

Fig. 9.18 Results of the point cloud registration via CP-GPA. Initial configuration is obtained by perturbing the ground-truth motions by a rotation with random axis and angle uniformly distributed over [0, 20◦ ]

172

9 3D Model Registration by Generalized Procrustes Analysis

9.3 Summary In this chapter, we presented the application of the Generalized Procrustes Analysis for solving the independent models block adjustment problem and the registration of multiple point clouds. In particular, we showed the potentialities of the Closest Point variant of the GPA that can be applied to the case in which point matches are unknown. This allows to extend the use of GPA to several real applications where a manual identification of the correspondences is unfeasible.

References A. Beinat, F. Crosilla, Generalized procrustes analysis for size and shape 3D object reconstruction, in Optical 3-D Measurement Techniques (Wichmann Verlag, 2001), pp. 345–353 P. Besl, N. McKay, A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992) Y. Chen, G. Medioni, Object modeling by registration of multiple range images, in Proceedings of the IEEE International Conference on Robotics and Automation (1991), pp. 2724–2729 F. Crosilla, A. Beinat, Use of generalised procrustes analysis for the photogrammetric block adjustment by independent models. ISPRS J. Photogrammetry Remote Sens. 56(3), 195–209 (2002) S. Du, N. Zheng, S. Ying, Q. You, Y. Wu, An extension of the ICP algorithm considering scale factor, in IEEE International Conference on Image Processing, ICIP 2007, vol. 5 (IEEE, 2007), pp. 193–196 F.R. Hampel, P.J. Rousseeuw, E.M. Ronchetti, W.A. Stahel, Robust Statistics: the Approach Based on Influence Functions. Wiley Series in Probability and Mathematical Statistics (Wiley, 1986) E. Maset, F. Arrigoni, A. Fusiello, Practical and efficient multi-view matching, in Proceedings of IEEE International Conference on Computer Vision, vol. 2 (2017), pp. 4578–4586 S. Petitjean, A survey of methods for recovering quadrics in triangle meshes. ACM Comput. Surv. 34(2), 211–262 (2002) S. Rusinkiewicz, M. Levoy, Efficient variants of the ICP algorithm, in Proceedings of the International Conference on 3-D Digital Imaging and Modeling (2001), pp. 145–152 J. Salvi, C. Matabosch, D. Fofi, J. Forest, A review of recent range image registration methods with accuracy evaluation. Image Vision Comput. 25(5), 578–596 (2007) R. Toldo, A. Beinat, F. Crosilla. Global registration of multiple point clouds embedding the generalized procrustes analysis into an ICP framework, in International Symposium on 3D Data Processing, Visualization and Transmission (2010), pp. 109–122 X. Zhou, M. Zhu, K. Daniilidis, Multi-image matching via fast alternating minimization, in Proceedings of the International Conference on Computer Vision (2015), pp. 4032–4040