Theoretical and Applied Statistics: In Honour of Corrado Gini SIS 2015 Treviso, Italy [1st ed.] 978-3-030-05419-9, 978-3-030-05420-5

Table of contents :
Front Matter ....Pages i-xv
Entropy Measures: An Health Care Study (Enrico Ciavolino, Corrado Crocetta, Amjad D. Al-Nasser)....Pages 1-11
A Review on Heterogeneity Test: Some Permutation Procedures (Stefano Bonnini, Eleonora Carrozzo, Luigi Salmaso)....Pages 13-20
Robust Estimation of Skew-Normal Parameters with Application to Outlier Labelling (Mario Romanazzi)....Pages 21-30
Asymptotics of S-Weighted Estimators (Jan Ámos Víšek)....Pages 31-42
Gini’s Delta to Measure Intensity of Multidimensional Performance (Silvia Terzi, Luca Moroni)....Pages 43-47
Frequentist and Bayesian Small-Sample Confidence Intervals for Gini’s Gamma Index in a Gaussian Bivariate Copula (Valentina Mameli, Alessandra R. Brazzale)....Pages 49-60
Bayesian Estimation of Gini-Simpson’s Index Under Mainland-Island Community Structure (Annalisa Cerquetti)....Pages 61-70
Spatial Residential Patterns of Selected Foreign Groups. A Study in Four Italian Cities (Federico Benassi, Fabio Lipizzi)....Pages 71-81
Minority Segregation Processes in an Urban Context: A Comparison Between Paris and Rome (Oliviero Casacchia, Luisa Natale, Gregory Verdugo)....Pages 83-90
Similarity of GPS Trajectories Using Dynamic Time Warping: An Application to Cruise Tourism (Mauro Ferrante, Christian Bongiorno, Noam Shoval)....Pages 91-101
The Financial Stress Spillover: Evidence from Selected Asian Countries (Zulfiqar Ali Shah, Muhammad Ejaz Majeed, Biagio Simonetti, Corrado Crocetta)....Pages 103-121

Citation preview

Springer Proceedings in Mathematics & Statistics

Corrado Crocetta Editor

Theoretical and Applied Statistics In Honour of Corrado Gini - SIS 2015, Treviso, Italy, September 9–11

Società Italiana di Statistica

Springer Proceedings in Mathematics & Statistics Volume 274

Springer Proceedings in Mathematics & Statistics This book series features volumes composed of selected contributions from workshops and conferences in all areas of current research in mathematics and statistics, including operation research and optimization. In addition to an overall evaluation of the interest, scientific quality, and timeliness of each proposal at the hands of the publisher, individual contributions are all refereed to the high quality standards of leading journals in the field. Thus, this series provides the research community with well-edited, authoritative reports on developments in the most exciting areas of mathematical and statistical research today.

More information about this series at http://www.springer.com/series/10533

Corrado Crocetta Editor

Theoretical and Applied Statistics In Honour of Corrado Gini - SIS 2015, Treviso, Italy, September 9–11

123

Editor Corrado Crocetta Department of Economics University of Foggia Foggia, Italy

ISSN 2194-1009 ISSN 2194-1017 (electronic) Springer Proceedings in Mathematics & Statistics ISBN 978-3-030-05419-9 ISBN 978-3-030-05420-5 (eBook) https://doi.org/10.1007/978-3-030-05420-5 Library of Congress Control Number: 2018965439 Mathematics Subject Classification (2010): S11001, S17040, S17010, S17030, X25000 © Springer Nature Switzerland AG 2019 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface

This volume includes a selection of the papers presented at the conference of the Italian Statistical Society: Statistics and Demography - the Legacy of Corrado Gini (SIS 2015), held in Treviso on September 9–11, 2015. It covered a wide variety of topics linked in various ways to the scientific legacy of Corrado Gini, ranging from the theory of statistical inference to demography, biology, sociology, and poverty studies. The topics addressed were of wide relevance to the social sciences, demography, and economics. This volume contains many interesting contributions on entropy measures (Ciavolino et al.), permutation procedures for the heterogeneity test (Salmaso et al.), robust estimation of skew-normal parameters (Romanazzi et al.), the S-weighted estimator (Víšek), measures of multidimensional performance using Gini’s delta (Terzi), small-sample confidence intervals for Gini’s gamma index (Mameli et al.), Bayesian estimation of the Gini–Simpson index (Cerquetti), spatial residential patterns of selected foreign groups (Benassi et al.), minority segregation processes (Natale et al.), dynamic time warping to study cruise tourism (Ferrante et al.), and financial stress spillover (Crocetta et al.). All papers published in the book have undergone rigorous peer review based on initial editor screening, anonymous refereeing by independent expert referees, and subsequent revision by article authors when required. Foggia, Italy October 2018

Corrado Crocetta

v

Contents

Entropy Measures: An Health Care Study . . . . . . . . . . . . . . . . . . . . . . . Enrico Ciavolino, Corrado Crocetta and Amjad D. Al-Nasser

1

A Review on Heterogeneity Test: Some Permutation Procedures . . . . . . Stefano Bonnini, Eleonora Carrozzo and Luigi Salmaso

13

Robust Estimation of Skew-Normal Parameters with Application to Outlier Labelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mario Romanazzi Asymptotics of S-Weighted Estimators . . . . . . . . . . . . . . . . . . . . . . . . . . Jan Ámos Víšek

21 31

Gini’s Delta to Measure Intensity of Multidimensional Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Silvia Terzi and Luca Moroni

43

Frequentist and Bayesian Small-Sample Confidence Intervals for Gini’s Gamma Index in a Gaussian Bivariate Copula . . . . . . . . . . . Valentina Mameli and Alessandra R. Brazzale

49

Bayesian Estimation of Gini-Simpson’s Index Under Mainland-Island Community Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Annalisa Cerquetti

61

Spatial Residential Patterns of Selected Foreign Groups. A Study in Four Italian Cities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Federico Benassi and Fabio Lipizzi

71

Minority Segregation Processes in an Urban Context: A Comparison Between Paris and Rome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Oliviero Casacchia, Luisa Natale and Gregory Verdugo

83

vii

viii

Contents

Similarity of GPS Trajectories Using Dynamic Time Warping: An Application to Cruise Tourism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Mauro Ferrante, Christian Bongiorno and Noam Shoval

91

The Financial Stress Spillover: Evidence from Selected Asian Countries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Zulfiqar Ali Shah, Muhammad Ejaz Majeed, Biagio Simonetti and Corrado Crocetta

About the Editor

Corrado Crocetta is Full Professor of Statistics at the University of Foggia, Italy. He received his Ph.D. in Statistics from the University of Bari in 1994. Dr. Crocetta has been an elected member of the SIS board (2012–16) and was appointed as Chairman of the Scientific Committee for the conference Statistics and Demography - the Legacy of Corrado Gini (SIS 2015). He has published over 60 papers in peer-reviewed national and international journals and proceedings, as well as a number of book chapters. He is an editorial board member or reviewer for a number of well-known journals. He has delivered many invited talks at national and international conferences.

ix

Gini’s Influence on Today’s Statistics

Corrado Gini was born in Motta di Livenza, a small town near Treviso. His parents were Luciano Gini and Lavinia Locatelli, who were wealthy land-owning farmers. Gini graduated in Bologna with a law degree in 1905, awarded for a thesis entitled Il sesso dal punto di vista statistico (Sex from a statistical point of view) that was published in 1908. In the same year, he published Sul concetto di probabilità (On the concept of probability) and was awarded his “libero docente” (similar to the habilitation), which gave him the right to lecture in universities. In 1909, he was appointed as a lecturer assisting the Chair of Statistics in the Faculty of Law at the University of Cagliari. In the following year, he was appointed as Full Professor of Statistics at Cagliari, and he founded a statistical laboratory there. He took up a similar position at the University of Padua in 1913. He founded the statistical journal Metron in 1920, which he edited until his death. Gini married Valentina Poggioli in 1921; they had two daughters. It was around this time that fascism gripped Italy. Both Gini and Mussolini were interested in demography, but Gini was interested in the evolution and equilibrium of population whereas Mussolini was interested in controlling population growth. Gini became a professor at the University of Rome in 1925. There, he founded a lecture course on Sociology, which he maintained until his retirement. He also set up the School of Statistics in 1928 and the Faculty of Statistical, Demographic, and Actuarial Sciences in 1936. In 1929, Gini founded the Comitato italiano per lo studio dei problemi della popolazione, which, 2 years later, organized the first Population Congress in Rome. In 1926, Gini was appointed as President of the Central Institute of Statistics in Rome. He resigned in 1931, in protest at interference in his work by the fascist state. In 1934, Gini founded the journal Genus, which became the official journal of the Italian Committee for the Study of Population Problems. Before World War II, Gini received numerous honors: he became an Honorary Member of the Royal Statistical Society (1920), Vice President of the International Institute of Sociology (1933; he later became president in 1950), President of the

xi

xii

Gini’s Influence on Today’s Statistics

Italian Society of Genetics and Eugenics (1934), President of the International Federation of the Society of Eugenics of the Latin language countries (1935), President of the Italian Society of Sociology (1937), Honorary Member of the ISI (1939), and President of the Italian Statistical Society (1941–1944). In 1943, Mussolini resigned and the Fascist party in Italy was dissolved. Gini’s position was not an easy one - he had been close to Mussolini but he had also shown his opposition to racist policies. During the summer of 1944, Gini was investigated for his role in the wartime regime. He had to leave teaching and his positions as Dean of the Faculty of Statistical Sciences and President of the Italian Statistical Society. In 1945, Gini was suspended from all academic duties and his salary was not paid for 1 year. In 1946, he resumed his duties at the Faculty and in 1949 again became President of the Italian Statistical Society, a position he held until his death. After this experience, his departure from politics was almost complete, and the last 20 years of his life were mainly devoted to his studies. In 1955, he was nominated as Professor Emeritus. In 1962, he became a national member of the Accademia dei Lincei. Corrado Gini passed away in the early morning of March 13, 1965. Gini’s main goal was the development of the theory of statistics and its applications to real contexts, without forgetting the connections of statistics with probability and its fundamental principles. Gini was a great collector of data, and he directed several scientific expeditions studying the demographic and medical profiles of populations in many countries. His most widely known contribution is the Gini coefficient, a measure of the inequality of distribution of wealth (or any other variable). In the paper The Dangers of Statistics, Gini attacked Fisher’s fiduciary methods and the Neyman–Pearson theory of hypothesis testing. In this, he took a somewhat Bayesian line, arguing for the importance of prior probabilities in judging the measures of a sample. Gini’s view of probability was limited, being related more to the traditional ideas of classical authors, such as Bernoulli, Laplace, and Pearson, than to the neo-Bayesian approach introduced by de Finetti. Actually, he personally knew de Finetti, and they also collaborated at the Italian National Institute of Statistics, but Gini did not really pay attention to his innovative idea of probability. We can conclude that 53 years after his death, it is still possible to appreciate his contribution to ISTAT’s organization and the reorganization of public statistics as well as his scientific work, which has made the Italian school competitive at the international level and has given momentum to the methodological progress of statistics. Today, we can still appreciate Gini’s heritage in terms of both the scientific content that he proposed and his vast work for institutional renovation. Corrado Crocetta

Gini’s Influence on Today’s Statistics

xiii

Bibliography 1. Atkinson, A.B.: On the measurement of inequality. J. Econ. Theory 2, 244–263 (1970) 2. Benedetti, C.: Ricordando Corrado Gini. Riv. di Politica Economica 55(3), 3–9 (1965) 3. Benedetti, C.: A proposito del centenario della nascita di Corrado Gini. Metron 42(1–2), 3–19, Bologna, Cuppini (1984) 4. Cassata, F.: Cronaca di un’epurazione mancata. Popolazione e Storia 2, 89–119 (2004) 5. Cassata, F.: Il Fascismo Razionale. Corrado Gini fra Scienza e Politica. Carocci, Roma (2006) 6. Cassese, S.: Lo Stato Fascista. Bologna, il Mulino (2010) 7. Castellano, V.: Corrado Gini: a memoir. Metron 24(1–4), 3–84 (1965) 8. Cocchi, D., Favero, G.: Gli Statistici Italiani e la Questione della Razza. In: Le leggi antiebraiche del 1938, le società scientifiche e la scuola in Italia, pp. 207–235. Accademia Nazionale delle Scienze detta dei XL, Roma (2009) 9. Costantini, D.: Il “metodo dei risultati” e le ipotesi profonde. Statistica 39, 35–43 (1979) 10. Dalla Zuanna, pp. 69–86. L’ancora del Mediterraneo, Napoli 11. de Finetti, B.: La probabilità secondo Gini nei rapporti con la concezione soggettivistica. Metron 25(1–4), 85–88 (1966) 12. Fisher, R.A.: Statistical Methods and Scientific Inference. Oliver & Boyd, London (1956) 13. Forcina, A.: Gini’s contribution to the theory of inference. Int. Stat. Rev. 50(1), 65–70 (1982) 14. Gini, C.: Contributo alle applicazioni statistiche del calcolo delle probabilità. G. economisti 35 (12), 1199–1237 (1907) 15. Gini, C.: Il sesso dal punto di vista statistico. Sandron, Napoli (1908) 16. Gini, C.: Il diverso accrescimento delle classi sociali e la concentrazione della ricchezza. G. Economisti 38, 27–83 (1909) 17. Gini, C.: Prezzi e consumi. G. Economisti 40(1, 2), 99–114, 235–249 (1910) 18. Gini, C.: Considerazioni sulle probabilità a posteriori e applicazioni al rapporto dei sessi alla nascita. Studi Economico-Giuridici, Facoltà di Giurisprudenza Della Regia Università di Cagliari, Anno III (1911) Reprinted in Metron 1949, 15(1–4). English translation in Gini (2001) 19. Gini, C.: Variabilità e mutabilità: contributo allo studio delle distribuzioni e delle relazioni statistiche. In: Studi Economico-Giuridici della Facoltà di Giurisprudenza della Regia Università di Cagliari, vol. 3, no. 2 (1912) 20. Gini, C.: L’ammontare e la composizione della ricchezza delle nazioni. Torino, Bocca (1914) 21. Gini, C.: Sulla misura della concentrazione e della variabilità dei caratteri. In: Atti del Reale Istituto Veneto di Scienze, Lettere ed Arti, a.a. 1913–14, 73, vol. 63, part II, pp. 1203–1248 (1914) English translation in Metron (2005) 22. Gini, C.: Measurement of inequality of incomes. Econ. J. 31, 124–126 (1921) 23. Gini, C.: Quelques considérations au sujet de la construction des nombres indices des prix et des questions analogues. Metron 4(1), 3–162 (1924) 24. Gini, C.: The contributions of Italy to modern statistical methods. J. R. Statist. Soc. A 89, 703–724 (1926) 25. Gini, C.: Present condition and future progress of statistics. J. Am. Stati. Assoc. 25, 295–304 (1930) 26. Gini, C.: Note. Bulletin de l’Institut International de Statistique, XIX Session (Tokio, 1930), tome XXV(3ème livraison, 2ème partie), 317K–320K (1931) 27. Gini, C.: Observations à la communication du Prof. L. von Bortkiewicz. Bulletin de l’Institut International de Statistique, XIX Session (Tokio, 1930), tome XXV(3ème livraison, 2ème partie), 299–306 (1931) 28. Gini, C.: Open letters. J. Am. Stati. Assoc. 29, 200–202 (1934) 29. Gini, C.: On the measure of concentration with special reference to income and wealth. In: Cowles Commission Research Conference on Economics and Statistics, July 6–August 8, Colorado College Publications, General Series, vol. 208, pp. 73–80 (1936)

xiv

Gini’s Influence on Today’s Statistics

30. Gini, C.: I pericoli della statistica. In: Atti della I Riunione scientifica della Società Italiana di Statistica, (Pisa, 9 Oottobre 1939), Roma (1939) 31. Gini, C.: I Testi Di Significatività. In: Atti Della VII Riunione Scientifica Della Società Italiana Di Statistica, pp. 1945. Roma, (Roma, 27–30 Giugno 1943): published in Atti della VI e VII Riunione scientifica della Società Italiana di Statistica (1943) 32. Gini, C.: Le applicazioni induttive del calcolo delle probabilità. Riv. di Politica Economica 12, 1059–1083. Reprinted in Gini (1968, 2, pp. 254–279) (1964) 33. Gini, C.: On the characteristics of Italian statistics. J. R. Statist. Soc. A 128(1), 89–109 (1965) 34. Gini, C.: Questioni fondamentali di probabilità e statistica (I, II). Tipografia Failli, Roma. Gini, C. (2001). Induction and Statistics. Bologna: Clueb (1968) 35. Giorgi, G.M., Gigliarano, C.: The Gini concentration index: a review of the inference literature. J. Econ. Surv. 31(4), 1130–1148 (2017) 36. Giorgi, G.M., Gubbiotti, S.: On Corrado Gini’s 1932 paper “Intorno alle curve di concentrazione”. A selection of translated excerpts. Metron 73(1), 1–24 (2015) 37. Giorgi, G.M., Gubbiotti, S.: Celebrating the memory of Corrado Gini: a personality out of the ordinary. Int. Stat. Rev. 85(2), 325–339 (2017) 38. Giorgi, G.M., Pallini, A.: Di alcune misure di disuguaglianza del benessere e della loro determinazione mediante trasformazione di variabile. Note Economiche 18, 182–194 (1985) 39. Giorgi, G.M.: A methodological survey of recent studies for the measurement of inequality of economic welfare carried out by some Italian statisticians. Econ. Notes 13, 146–157 (1984) 40. Giorgi, G.M.: Bibliographic portrait of the Gini concentration ratio. Metron 48(1–4), 183–221 (1990) 41. Giorgi, G.M.: Il rapporto di concentrazione di Gini: genesi, evoluzione ed una bibliografia commentata. Editrice Ticci, Siena (1992) 42. Giorgi, G.M.: A fresh look at the topical interest of the Gini concentration ratio. Metron 51 (1–2), 83–98 (1993) 43. Giorgi, G.M.: Encounters with the Italian statistical school: a conversation with Carlo Benedetti. Metron 54(3–4), 3–23 (1996) 44. Giorgi, G.M.: Income inequality measurement: the statistical approach. In: Silber, J. (Ed.) Handbook on Income Inequality Measurement, pp. 245–260. Kluwer Academic Publishers, Boston (1999) 45. Giorgi, G.M.: Intervista a Italo Scardovi. Statistica & Società 1(1), 15–23 (2002) 46. Giorgi, G.M.: Gini’s scientific work: an evergreen. Metron 63, 299–315 (2005) 47. Giorgi, G.M.: Corrado Gini: the man and the scientist. Metron 69, 1–28 (2011) 48. Giorgi, G.M.: The Gini index decomposition: an evolutionary study. In: Deutsch, J., Silber, J. (eds.) The Measurement of Individual Well-being and Group Inequalities, pp. 185–218. Oxon, Routledge (2011) 49. Herzel, A., Leti, G.: Italian contributions to statistical inference. Metron 35(1–2), 3–48 (1977) 50. Lambert, P.J., Decoster, A.: The Gini coefficient reveals more. Metron 63, 373–400 (2005) 51. Leti, G.: The international activities of Italian statisticians prior to the second world war. Statistica 64(1), 317–331 (2004) 52. Leti, G.: L’Istat e il Consiglio superiore di statistica dal 1926 al 1945. Istat, Roma (1996) 53. Lijoi, A., Prünster, I.: A conversation with Eugenio Regazzini. Stat. Sci. 26(4), 647–672 (2011) 54. Mieli, P.: Così Fermi scoprì la natura vessatoria del fascismo. In: Corriere della Sera, Milano 41 (2001) 55. Misiani, S.: I numeri e la politica. Statistica, programmazione e mezzogiorno nell’impegno di Alessandro Molinari. Il Mulino, Bologna (2007) 56. Muliere, P.: Misure della concentrazione: alcune osservazioni sull’impostazione assiomatica. In: Zenga, M. (ed.) La distribuzione personale del reddito: problemi di formazione, di ripartizione e di misurazione, pp. 190–213. Vita e Pensiero, Milano (1987) 57. Pareto, V.: La legge della domanda. G. Economisti 10, 59–68 (1895)

Gini’s Influence on Today’s Statistics

xv

58. Pareto, V.: Aggiunta allo studio sulla curva delle entrate. G. Economisti 14, 15–26 (1897) 59. Perfetti, F.: Lo stato fascista. Le basi sindacali e corporative. Casa editrice Le Lettere, Firenze (2010) 60. Piccinato, L.: Gini’s criticisms to the theory of inference: a missed opportunity. Metron 69(1), 101–117 (2011) 61. Pietra, G.: Delle relazioni tra indici di variabilità (I, II). Atti del Reale Istituto Veneto di Scienze, Lettere ed Arti 74(2), 775–804 (1915) 62. Pietra, G.: Note. Bulletin de l’Institut International de Statistique, XIX Session (Tokio, 1930), tome XXV(3ème livraison, 2ème partie), 320L–320M (1931) 63. Pietra, G.: Observations à la communication du Prof. L. von Bortkiewicz. Bulletin de l’Institut International de Statistique, XIX Session (Tokio, 1930), tome XXV(3ème livraison, 2ème partie), 310 (1931) 64. Pretolani, D.: Apportionments with minimum Gini index of disproportionality: a quadratic knapsack approach. Ann. Oper. Res. 215(1), 257–267 (2014) 65. Prévost, J.G.: A total science. Statistics in liberal an fascist Italy. McGill-Queen’s University Press, Montreal (2009) 66. Regazzini, E.: Gini Corrado. In: Johnson, N.L., Kotz, S. (eds.) Leading Personalities in Statistical Sciences, pp. 291–296. Wiley, New York (1997) 67. Romano, S.: 1931: i professori giurano fedeltà al fascismo. In: Corriere della Sera, Milano 39 (2006) 68. Savorgnan, F.: Observations à la communication du prof. L. von Bortkiewicz. Bulletin de l’Institut International de Statistique, XIX Session (Tokio,1930), tome XXV(3ème livraison, 2ème partie), 307–309 (1931) 69. Sen, A.: Utilitarianism and inequality. Econ. Polit. Wkly. 7(5–7), 343–344 (1972) 70. Sen, A.: On Economic Inequality. Clarendon Press, Oxford (1973) 71. Sen, A.: Poverty: an ordinal approach to measurement. Econometrica 44, 219–231 (1976) 72. Sen, A.: Foreword. In: Silber, J. (ed.) Handbook of Income Inequality Measurement, pp. xvii–xxvi. Boston, Kluwer Academic Publishers (1999) 73. Treves, A.: Le nascite e la politica nell’Italia del Novecento.Milano, Edizioni Universitarie di Lettere, Economia e Diritto (2001) 74. Trivellato, U.: Al crocevia fra scienza, ideologia e regime: uno sguardo allo sfondo e ad alcuni statistici e demografi eminenti. In: Numeri e potere, Statistica e Demografia nella cultura italiana tra le due guerre, Ed. G. (2004) 75. Ventura, A.: Le leggi razziali all’Università di Padova. In: L’Università dalle leggi razziali alla resistenza, Ed. (1996) 76. Yitzhaki, S., Schechtman, E.: The Gini Methodology. A Primer on a Statistical Methodology. Springer, New York (2013)

Entropy Measures: An Health Care Study Enrico Ciavolino, Corrado Crocetta and Amjad D. Al-Nasser

Abstract In medical emergency situations, the triage process allows patients in potentially life-threatening condition to receive the fastest and most appropriate medical treatment. Triage consists in an evaluation of patients’ medical condition on a colour-based scale, reflecting from major to minor urgency. Shannon’s entropy measures are applied to such process in order to evaluate concordance, overestimation and underestimation of triage codes assigned to patients in two different moments and by different health-care professionals: during the acceptance phase, by nurses (variable X ), and by physicians after deepened diagnostic evaluation (variable Y ). Entropy indexes were also used to compare the years 2016 and 2015, showing a little increment of equivocal transmission with respect to year 2015. Keywords Entropy measures · Emergency department triage · Information theory

1 Introduction Particularly in the health field there is a need of complexity reduction [8] and attempts to clarify aspects of multifaceted occurrences through appropriate statistical methods [4, 9]. In this paper, Shannon’s entropy measures are used in order to study the amount of uncertainty in a triage process.

E. Ciavolino (B) University of Salento, Edificio 5, Stanza 39 Via di Valesio, 73100 Lecce, Italy e-mail: [email protected] C. Crocetta University of Foggia, L.go Papa Giovanni Paolo II, 1, Foggia, Italy e-mail: [email protected] A. D. Al-Nasser Yarmouk University, Irbid 21163, Jordan e-mail: [email protected] © Springer Nature Switzerland AG 2019 C. Crocetta (ed.), Theoretical and Applied Statistics, Springer Proceedings in Mathematics & Statistics 274, https://doi.org/10.1007/978-3-030-05420-5_1

1

2

E. Ciavolino et al.

In medicine many techniques are being developed to help physician to establish a diagnosis and to carry out treatment under relative uncertainty, but in this study we will concentrate on the concept of entropy and in particular on the idea of the information degradation during the triage process, in which patients’ condition are evaluated. In this paper we consider triage process the phase in which patients are labeled into five categories according to a colour-coding system that classifies the level of critical injuries: red, yellow, green, white, and black tags. Measures of entropy are used to evaluate the total amount of information in all the triage process and the level of agreement between the evaluation given at the acceptance step and the one given by the doctors after medical tests. Moreover, we compare the differences, in term of information, between years 2015 and 2016. The remaining part of this paper is divided as follows: Sect. 2 introduces the details of the triage process; Sect. 3 gives a review of Shannon’s entropy measures and related indexes; the triage case study is showed in Sect. 4; discussion and conclusions are presented in the last Sect. 5.

2 Triage Process Triage refers to the evaluation and categorization of the sick or wounded when there are insufficient resources for medical care for everyone at once. Historically, triage is believed to arise from systems developed for categorization and transportation of wounded soldiers on the battlefield. Triage is used in a number of situations in modern medicine, including: • In mass casualty situations, triage is used to distinguish the most urgent situation that require immediate transportation to a hospital for health-care (generally, patients who have a chance of survival but who would die without immediate treatment) and situation of less severe injuries that must wait for medical care. • Triage is also commonly used in crowded emergency rooms and walk-in clinics to determine which patients should be seen and treated immediately. • Triage may be used to prioritize the use of space or equipment, such as operating rooms, in a crowded medical facility. In a walk-in clinic or emergency department, an interview with a triage nurse is a common first step to receiving care. He or she generally takes a brief medical history of the complaint and takes vital signs (heart rate, respiratory rate, temperature, and blood pressure) in order to identify seriously ill persons who must receive immediate care. In a hospital, triage might prevent an operation for an elective facelift from being performed if there are numerous emergency cases requiring use of operating facilities and surgical nursing staff.

Entropy Measures: An Health Care Study

3

A system that has been used in mass casualty situations is an example of advanced triage implemented by nurses or other skilled personnel. This advanced triage system involves a colour-coding scheme using red, yellow, green, white, and black tags: • Red tags - (immediate) are used to label those who cannot survive without immediate treatment but who have a chance of survival. • Yellow tags - (observation) for those who require observation (and possible later re-triage). Their condition is stable for the moment and, they are not in immediate danger of death. These victims will still need hospital care and would be treated immediately under normal circumstances. • Green tags - (wait) are reserved for the “walking wounded” who will need medical care at some point, after more critical injuries have been treated. • White tags - (dismiss) are given to those with minor injuries for whom a doctor’s care is not required. • Black tags - (expectant) are used for the deceased and for those whose injuries are so extensive that they will not be able to survive given the care that is available.

3 Entropy Measures Given two discrete random variables X and Y whose outcomes are xi (i =1...R) and R piX = y j ( j = 1...C) respectively, with mass probabilities piX and p Yj , with i=1 C 1 and j=1 p Yj = 1. The joint probability distribution is defined as pi j ≥ 0, with  R C i=1 j=1 pi j = 1. In the following sections we give an introduction to the Entropy’ Measures with the main features. In Fig. 1 Shannon’s Entropy, the relative measures and the mutual information are graphically represented, showing how they are related. Moreover, we briefly introduce also Maximum Entropy Principle (MEP) and the Generalized Cross Entropy (GCE) Method within the simple case of linear models. Fig. 1 Entropy measure and mutual information

4

E. Ciavolino et al.

3.1 Shannon’s Entropy It is well known as Entropy can be seen as a measure of information carried out from a probability distribution. Given the above discrete random variable X , the Shannon’s entropy is defined as follows: H ( p) = −

R 

piX · ln( piX )

(1)

i=1

which is a convex function whose minimum is zero when P = ( p1X , p2X , ..., p RX ) is a degenerated probability distribution (perfect certainty), while it assumes its maximum when P is a uniform distribution (perfect ignorance). Jaynes [11] experimentally derived a measure of entropy [12] getting the same result that Shannon introduced [14]. In developing this approach, results will lead to Bernoulli trial. Lets suppose to carry out N trials at random with repetitions of an experiment with K possible outcomes. Each event of this experiment will occur a specific number of times among the N trials. Then, we have KN total outcomes in the experiment. This random experiment will generate the (frequency) probability assignment: ni with i = 1, 2, . . . , K pi = N where n i is number of times that the ith event occurs among the N trials, with  n i = N . Thus, we can represent the number of ways a particular set of n i by the multinomial coefficient: W =

N! N! = K (N p1 )!(N p2 )! . . . (N p K )! i=1 n i !

(2)

or the monotonic function of W , namely, K 

ln(W ) = ln(N !) −

ln(n i !)

(3)

i=1

When N becomes large, one can use Stirling’s approximation to simplify this function, where Stirling’s formula is given by:   √ 1 1 (4) +0 ln(N !) ≈ N ln(N ) − N + 2π N + 12N N2 As N → ∞ this approximation will lead to:   ln(W ) ≈ N ln(N ) − N − n i ln(n i ) + ni i

i

(5)

Entropy Measures: An Health Care Study

but we have

 i

n i = N and pi =

5

ni , hence: N

 ln(W ) pi ln( pi ) = H (P) =− N i

(6)

which gives the same Shannon’s entropy measure as reported in the Eq. 1.

3.2 Conditional Entropy Conditional Entropy, that is, the quantification of the amount of information needed to describe the outcome of a random variable Y given the known value of another random variable X , can be expressed as: H (Y |X ) =

C R  

 pi j ln

i=1 j=1

H (Y |X ) =

C R  

1 p j|i

 pi j ln

i=1 j=1

pi· pi j

 (7)  (8)

where pi j represents the probability of the joint distribution of X and Y , while p j|i represents the probability of the joint distribution of Y conditioned on X . In essence, in order to communicate something about an event Y , conditional entropy requires less amount of information. In fact, its possible values will always be between 0 and the non-conditioned entropy value of event X : 0 ≤ H (Y |X ) ≤ H (X )

(9)

Extreme cases occur when the event Y it completely determined by X , and when the two events are independent. Conditional entropy of a variable Y , plus the entropy value of variable X , will result in joint entropy (Fig. 2).

Fig. 2 Conditional entropy

6

E. Ciavolino et al.

Fig. 3 Joint entropy

3.3 Joint Entropy Given two discrete random variables X and Y and the mass probabilities pi and p j , the Joint Entropy can be formalized as follows: H (X, Y ) =

R  C 

pi, j ln

i=1 j=1

1 pi, j

(10)

The joint entropy has the following properties: • H (X, Y ) > 0 • H (X, Y ) ≥ max[H (X ), H (Y )] • H (X, Y ) ≤ H (X ) + H (Y ) It can be represented by Venn’s Diagram reported in the Fig. 3.

3.4 Mutual Information The amount of information provided on average by two joint events X and Y can be defined as follows: I (X, Y ) =

R  C  i=1 j=1

 pi, j log

pi, j pi · p j

 (11)

The mutual information is a measure of interdependence between two random variables, that can be seen as the level of association, or the amount of information shared by X and Y , as reported in (11) (Fig. 4). The mutual information has the following properties: • I (X, Y ) > 0 • I (X, Y ) = 0 when the variables are statistically independent

Entropy Measures: An Health Care Study

7

Fig. 4 Mutual information

3.5 Cross Entropy The cross entropy (also called relative entropy) indicates the amount of divergence between two events. In particular, given X and Y, D(X ||Y ) is the measure of the distance between the probability distributions P(X ) and P(Y ), in terms of informational content.  X  pi X (12) D(X ||Y ) = pi log piY i The cross entropy has the following properties: • D(X ||Y ) ≥ 0 • D(X ||Y ) = 0 i f piX = piY ∀i • D(X ||Y ) = D(Y ||X ) The generalized version, called Generalized Cross Entropy (GCE), was first proposed by Golan et al. [7] as a generalization of the well-known Maximum Entropy principle described by E.T. Jaynes [10, 11]. Jaynes’ idea is mainly based on the principles of Shannon’s Information Theory and Shannon’s entropy [6, 14]. In qualitative terms, entropy is a measure of the average information carried out by a probabilistic source of data. In many statistical applications [5], entropy can be used as an information recovering device (e.g., from ill-posed problems: short and fat matrices, multicollinearity) as well as a method of estimation for both linear [1, 3] and more complex models [2, 13].

4 Health Care Study 4.1 Triage Data The measures described above have been used to test the level of appropriateness of the triage colour code assigned by the nurses at the acceptance office of the Emergency Department of Ospedali Riuniti di Foggia (Italy) and the colours assigned by physicians after completing the diagnostic tests. According to the See & Treat

8

E. Ciavolino et al.

Table 1 Contingency table for the years 2015 and 2016 Triage codes - evaluation Green Yellow Red 2015 Triage code - acceptance

2016 Triage code - acceptance

Green Yellow Red Total Green Yellow Red Total

17969 1789 28 19786 18810 2823 85 21718

3773 25841 974 30588 3858 21227 872 25957

35 885 2519 3439 43 926 2678 3647

Total 21777 28515 3521 53813 22711 24976 3635 51322

protocol, as a patient gets to the emergency department, he receives a colour code based on the severity of the diagnosis and the level of care required. The colours used are white, green, yellow, red and black, where white code is associated to minors problems, while red colour code is assigned to patients with serious problems or life-threatening condition, and black code are used for deceased patients. As patients with white or black codes are very few, we decided to exclude them from the analysis, focusing on the 53,813 admissions of 2015 and the 51,322 admissions of 2016. As we can see in Table 1, the main diagonal of the two contingency tables shows the cases of concordance between the code assigned during acceptance and after the diagnostic tests. The lower triangle of the table reports cases in which the severity of the diagnosis has been reduced after the necessary examinations, while the upper one reports cases for which the colour code got worse after the checks have been carried out.

4.2 Descriptive Analysis As briefly discussed in the previous section, frequencies on the main diagonal show correct prediction between the acceptance and doctors’ evaluation. Although frequencies on the diagonal are very high, there are some extra-diagonal frequencies that indicate discordance in the triage process. Frequencies below the main diagonal refer to an overestimation of the diagnosis (e.g. green patient is classified as red) which is the least dangerous situation. On the contrary, frequencies above the main diagonal refer to underestimation of the triage process (e.g. red patient is classified as green): this is a very dangerous situation, since the patient could not receive the appropriate care in due time. Table 2 shows conditional frequencies of Y (doctor evaluation) given X (acceptance evaluation).

Entropy Measures: An Health Care Study

9

Table 2 Contingency table for the years 2015 and 2016 (percentages) Triage codes - evaluation Green Yellow Red 2015 Triage code - acceptance

2016 Triage code - acceptance

Green Yellow Red Green Yellow Red

82.51 6.27 0.80 82.82 11.30 2.34

Table 3 Entropy measures and mutual information Year 2015 H (X ) H (Y ) H (X, Y ) I (X, Y ) D(Y ||X )

0.881 0.865 1.310 0.436 0.429

17.33 90.62 27.66 16.99 84.99 23.99

0.16 3.10 71.54 0.19 3.71 73.67

Total 100 100 100 100 100 100

Year 2016 0.899 0.897 1.399 0.396 0.501

4.3 Data Analysis Table 3 reports values of entropy measures and the mutual information, computed per year. The aim is to evaluate differences in terms of entropy between 2015 and 2016. In Table 3, columns indicate the years, whereas each row shows the entropy measure explained in Sect. 3.1. For 2015, the value for mutual information in both acceptance and evaluation process is equal to 0.436, corresponding to about 33% of the total amount of information. Considering the evaluation made by the doctor (variable Y ), as the receiver in a transmission system, and the evaluation (e.g. one the three codes considered: green, yellow, red) during the acceptance step (variable X ) as the source of the signal, it is possible to determine that the noisy background, that is, the disturbing element in the transmission (degrading the information) is equal to 0.429, corresponding to 32.7%. In the same way, for year 2016, it is possible to define the amount of information shared by both acceptance and evaluation phases, which is equal to I (X, Y ) = 0, 396, corresponding to about 28%. The transmission process of Y given X reports a value of D(Y ||X ) = 0.501, that is about 36%. This means that between the two years there is an increment of equivocal transmission (degraded information) equal to 3% (D(Y ||X )2016 − D(Y ||X )2015 = 3%). In order to evaluate the difference over the years, in terms of total amount of entropy, cross entropy (12) is computed, showing a value of D((X, Y )2015 || (X, Y )2016 ) = 0.013.

10

E. Ciavolino et al.

5 Discussion and Conclusion The present work illustrated how to use entropy measures in order to examine the triage process: from one side, it was possible to compare the ability to transfer information between acceptance and evaluation phase; from the other side, differences in terms of information between years 2015 and 2016 were taken into account and explored. The empirical evidence was based on data collected by the Emergency Department of Ospedali Riuniti di Foggia (Italy), where colour codes were assigned in two different moments: first, in the acceptance office; last, by physicians after all the diagnostic tests took place. Over the years 2015 and 2016, a decrease in the amount of information shared by both the acceptance phase and phisicians’ evaluation has been observed, while the amount of noisy information increased. Such results highlight two main aspects: first, the appropriateness of entropy measures in catching variations of complex phenomena and their potential to sum them up; also, data collected from two years might be not enough to allow researchers to draw certain conclusions about the phenomena under study. As described above, it implies exchange of information between different sources (both nurses and phisicians) in different moments and in differently critical situation: for these reasons, further research based on a broader time series might strenghten results obtained from entropy measures and shed light on any cue useful to an improved code assignment, whether it concerns instruments used for evaluation or the qualitative nature of communication between professionals throughout the triage process.

References 1. Carpita, M., Ciavolino, E.: A generalized maximum entropy estimator to simple linear measurement error model with a composite indicator. Adv. Data Anal. Classif. 11(1), 139–158 (2016) 2. Ciavolino, E., Al-Nasser, A.: Comparing generalised maximum entropy and partial least squares methods for structural equation models. J. Nonparametric Stat. 21(8), 1017–1036 (2009) 3. Ciavolino, E., Calcagnì, A.: Generalized cross entropy method for analysing the servqual model. J. Appl. Stat. 42(3), 520–534 (2015) 4. Ciavolino, E., Carpita, M., Al-Nasser, A.: Modelling the quality of work in the italian social cooperatives combining NPCA-RSM and SEM-GME approaches. J. Appl. Stat. 42(1), 161–179 (2015) 5. Ciavolino, E., Dahlgaard, J.: Simultaneous equation model based on the generalized maximum entropy for studying the effect of management factors on enterprise performance. J. Appl. Stat. 36(7), 801–815 (2009) 6. Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley-Interscience, Hoboken (2006) 7. Golan, A., Judge, G.: Maximum Entropy Econometrics: Robust Estimation with Limited Data. Wiley, New York (1996) 8. Grover, G., Dutta, R.: Survival analysis of acute myocardial infarction patients using nonparametric and parametric approaches. Electron. J. Appl. Stat. Anal. 2(1), 22–36 (2009)

Entropy Measures: An Health Care Study

11

9. Jayakumar, D.S., et al.: Heteroscedasticity in survey data and model selection based on weighted schwarz Bayesian information criteria. Electron. J. Appl. Stat. Anal. 7(2), 199–217 (2014) 10. Jaynes, E.: Information theory and statistical mechanics. Phys. Rev. 106(4), 620 (1957) 11. Jaynes, E.: Prior probabilities. IEEE Trans. Syst. Sci. Cybern. 4(3), 227–241 (1968) 12. Jaynes, E.T.: Probability Theory: The Logic of Science. Cambridge University Press, Cambridge (2003) 13. Papalia, R.B., Ciavolino, E.: Gme estimation of spatial structural equations models. J. Classif. 28(1), 126–141 (2011) 14. Shannon, C.: A mathematical theory of communications. Bell Syst. Tech. J. 27, 379–423 (1948)

A Review on Heterogeneity Test: Some Permutation Procedures Stefano Bonnini, Eleonora Carrozzo and Luigi Salmaso

Abstract When dealing with categorical data, generally the notion of heterogeneity may be used instead of that of variability. There are many fields where data may be only represented by nominal categorical variables or by ordinal variables, e.g. opinion polls, performance qualitative assessments, psycho aptitude tests and so on. In this paper we provide a review of some nonparametric methods concerning testing for heterogeneity, based on permutation procedures. Examples of real applications in different frameworks are also shown. Keywords Heterogeneity tests · Permutation test · Nonparametric framework

1 Introduction and Heterogeneity Indexes Let us suppose to compare C populations (C ≥ 2) that may take K categories. Let f jk denote the absolute frequency of the j-th sample for the k-th category ( j = 1, . . . , C; k = 1, . . . , K ). Thus the data of the problem have the form of a C × K contingency table of the observed frequencies. We consider a categorical variable X and let us suppose that it takes categories in (A1 , . . . , A K ), with unobserved probability distribution Pr{X = Ak } = πk , k = 1, . . . , K . The following properties should be satisfied by an index η for measuring the degree of heterogeneity:

S. Bonnini Department of Economics and Management, University of Ferrara, Ferrara, Italy e-mail: [email protected] E. Carrozzo · L. Salmaso (B) Department of Management and Engineering, University of Padova, Padua, Italy e-mail: [email protected] E. Carrozzo e-mail: [email protected] © Springer Nature Switzerland AG 2019 C. Crocetta (ed.), Theoretical and Applied Statistics, Springer Proceedings in Mathematics & Statistics 274, https://doi.org/10.1007/978-3-030-05420-5_2

13

14

S. Bonnini et al.

1. It reaches its minimum when the distribution is degenerate (and X is said to be minimally heterogeneous), i.e. when there is an integer r ∈ (1, . . . , K ) such that πr = 1 and πk = 0, ∀k = r ; 2. It assumes increasingly greater values when moving away from the degenerate towards the uniform distribution; 3. It reaches its maximum when the distribution is uniform (and X is said to be maximally heterogeneous), i.e. πk = 1/K , ∀k ∈ (1, . . . , K ). Let’s see the most popular indicators of heterogeneity: Gini’s index (Gini, 1912): ηG =

K 

πk (1 − πk ) = 1 −

K 

πk2 ,

(1)

πk log πk

(2)

K  1 log πkδ (1 − δ) k=1

(3)

k=1

k=1

Shannon’s entropy index (Shannon, 1948): η S = −

K  k=1

R´enyi entropy of order δ(R´enyi, 1996): η Rδ =

where log(·) is the natural logarithm and assuming that 0 · log(0) = 0. For interesting proposals about indexes for measuring heterogeneity see [2–4, 6, 7, 10, 12]. For a detailed discussion on heterogeneity in descriptive statistics see [17]. Consider two populations P1 and P2 , and suppose to be interested to test the hypotheses H0 : H et (P1 ) = H et (P2 ) against the alternative H1 : H et (P1 ) > H et (P2 ). where H et (P j ) represents the heterogeneity of population P j ,and η j is an index (Gini’s, Shannon’s, Rényi’s or others) for measuring H et (P j ), j = 1, 2. Let us suppose to have two independent samples with i.i.d. observations X j = {X ji , i = 1, . . . , n j ; n j > 2}, j = 1, 2. Observed data  can be displayed in a 2 × K contingency table with absolute frequencies f jk = k≤n j I(X ji = Ak ), k = 1, . . . , K , j = 1, 2. Let f •k = f 1k + f 2k , k = 1, . . . , K be the marginal frequency of the kth column (category), whereas the sample size n j , j = 1, 2 is the marginal frequency of j-th row. Consider the following representation of the observed data: X = {X (i), i = 1, . . . , n; n 1 , n 2 } and observe that marginal frequencies are permutation invariant. Thus the contingency table of a permuted dataset X∗ (the permuted table) has the same marginal frequencies of the observed contingency table, since a permutation of the rows of the data implies that some units of sample 1 are reassigned to sample 2 and viceversa.

A Review on Heterogeneity Test: Some Permutation Procedures

15

It is worth observing that if the exchangeability condition is satisfied, in univariate two-sample tests the set of marginal frequencies (n 1 , n 2 , f •1 , . . . , f •K ), the data set X, and any of its permutations X∗ , are equivalent sets of sufficient statistics [19]. If probabilities { π jk , k = 1, . . . , K , j = 1, 2} were known, they could be arranged in non-increasing order: π j (1) ≥ . . . ≥ π j (K ) , according to the Pareto diagram, and the null hypothesis could be written as: H0 : {H et (P1 ) = H et (P2 )} = {π1(k) = π2(k) , k = 1, . . . , K } Under the null hypothesis, exchangeability holds and the permutation testing principle is applicable exactly. The alternative hypothesis of the problem may be defined as H1 : {H et (P1 ) > H et (P2 )}  k  k   = π1(s) ≤ π2(s) , k = 1, . . . , K s=1

s=1

and the strict inequality holds for at least one k = 1, . . . , K . Since ordered parameters π j (k) , k = 1, . . . , K , j = 1, 2, are unknown and can only be estimated using the observed ordered frequencies:  π j (k) = f j (k) /n j , the ordering within each population, estimated through relative frequencies within each sample, is a data-driven ordering and so it may differ from the true one and it presents sampling variability. Hence exchangeability under H0 is not exact but only approximated. Data-driven and true ordering are equal with probability one only asymptotically and, thanks to the well-known Glivenko–Cantelli theorem [20], we can say that exchangeability of data with respect to samples is asymptotically attained under the null hypothesis. To solve the testing problem, a reasonable test statistic could be a function of the sampling estimates of the indexes described in Sect. 1. Specifically, considering the sampling index  η j = η( f j1 , . . . , f j K ) as estimate of the index η j = η1 −  η2 , where η η(π j1 , . . . , π j K ), j = 1, 2, a possible test statistic could be Tη =  is an index of heterogeneity (Gini’s, Shannon’s, Rényi’s or others). H0 should be rejected in favor of H1 for large values of Tη . Once computed the observed ordered table for both the samples, let us apply a testing procedure for stochastic dominance on ordered categorical variables (see e.g. [1, 8, 9, 13–16, 18]). When the observed value of the test statistics Tηo is calculated, B independent permutations of the data set are performed and at each permutation we compute a permuted table { f j∗(k) ; k = 1, . . . , K ; j = 1, 2} and the corresponding permuted value of the test statistic  Tη∗ . The p-value λη = # Tη∗ > Tηo /B can be computed and compared with the significance level α as usual: Ho is rejected in favor of H1 if λη < α, otherwise it cannot be rejected.

16

S. Bonnini et al.

2 Two-Sided and Multi-sample Test In Sect. 1 we described the one-sided test for heterogeneity comparisons when a stochastic dominance is of interest. In the present section we considered the case in which we are interested only to know if two populations significatively differ. If the null hypothesis is not true, then there is a difference between the populations. Since the alternative hypothesis specifies no direction for the difference, we have the so called two-sided test. Formalizing the problem, let us consider two populations P1 and P2 and let’s denote with H et (P j ) the heterogeneity of P j . The system of hypotheses we want to test is: H0 : H et (P1 ) = H et (P2 ) against the alternative H1 : H et (P1 ) = H et (P2 ). Such a problem is very common in several real application problems. Let’s think to a customer satisfaction survey where the interviewed are asked to give a qualitative judgement about a product (e.g.: Poor, Fair, Good, Excellent) or to express their level of satisfaction about it (e.g.: very dissatisfied, moderately dissatisfied, moderately satisfied, very satisfied). We could be interested in comparing two groups of customers in terms of heterogeneity of the judgements, for checking if the representativeness of medians, as indicators representing the global evaluation of the groups, are equal or not. In the case of quantitative data, this test is equivalent to the two-sample test for variance comparison. The procedure of the permutation test for this problem is similar to that of dominance in heterogeneity described in the previous section. For the two-sided problem the test statistic should be based on the absolute value of the difference between the η1 −  η2 |. High values of the test statistic lead to the rejection sampling indices: Tη = | of the null hypothesis in favor of the alternative. The rest of the procedure follows the same steps already described for the test on the dominance in heterogeneity: (1) compute the observed value of the test statistic Tηo ; (2) perform B independent permutations of the data set and for each permutation, obtain permuted table { f j∗(k) ; k = 1, . . . , K ; j = 1, 2} and the corresponding permuted value of the test statistic  Tη∗ ; (3) the p-value λη = # Tη∗ > Tηo /B can be computed and Ho is rejected in favor of H1 when λη < α. Generalizing it could be possible to include the case of multi-sample test. When we are interested to compare the heterogeneities of C populations, with C > 2, the hypotheses of the problem ca be written H0 : H et (P1 ) = H et (P2 ) = · · · = H et (PC )

A Review on Heterogeneity Test: Some Permutation Procedures

17

and H1 : H et (P j ) = H et (Pr ) for some j, r ∈ {1, . . . , C} . Here the test statistic could be based on the absolute values of the differences between the sampling indices and the index computed on the pooled dataset. Assume to have C independent samples with i.i.d. observations X j = {X ji , i = 1, . . . , n j ; n j > 2}, j = 1, . . . , C. Observed data can be displayed in a C × K contingency table with absolute frequencies { f jk = i≤n j I(X ji = Ak ), k = 1, . . . , K , j = 1, . . . , C}. Let f •k = f 1k + · · · + f Ck , k = 1, . . . , K indicates the marginal frequency of the k-th column (category) and the sample size n j , j = 1, . . . , C be marginal frequency of j-th row. The test statistic based on the index η is C     ηj −  η j = η f j1 , . . . , f j K is the sampling index for sample Tη = η• , where  j=1

j and  η• = η ( f •1 , . . . , f •K ) is the sampling index for the pooled dataset. The null hypothesis is rejected for large values of the test statistic, hence the steps of the testing procedures are similar to what previously described.

3 Some Applications In this section we introduce some applications of the presented tests for heterogeneity to solve real problems in different contexts. We will show examples for testing dominance, for two-sided test and multi-sample test, using all the three indices of heterogeneity presented in Sect. 1.

3.1 Dominance in Heterogeneity: Market Segmentation A Tourist Association in the North of Italy, in the Natural Park of Dolomites of Sesto wants to know the satisfaction of their guests about the facilities and offerings of the district of Sesto Dolomites/South Tyrol. A sample of tourists completed a questionnaire about satisfaction, habits and preferences. One of the questions was related to the type of accomodation during their stay in Sesto. One of the objective was to analyze the ability to satisfy the demand for accomodation for vacationers, distinguishing between new and loyal visitors. To this aim the hypothesis that the choice regarding the accommodation by tourists who had previous stays in Sesto is less heterogeneous, mainly oriented towards hotels, than that of tourists who were at their first presence in Sesto could be of interest. The categorical response variable Accomodation can take five modalities: Camping, Hotel, Bed & Breakfast, Farm and House/Flat. The variable/factor useful to define the compared groups is First presence in Sesto, and it can take two levels: no and yes. Table 1 shows the distributions of observed frequencies. More formally

18

S. Bonnini et al.

Table 1 Contingency table of the type of accomodation for the tourists in the “Sesto Nature Survey 2010”, with the distinction between visitors at their first presence or not Accomodation First presence Total No Yes Camping Hotel Bed & Breakfast Farm House or Flat Total

1 45 2 2 4 54

4 97 10 4 26 141

5 142 12 6 30 195

the hypothesis we wish to test is H0 : H et (Accomodation1 ) = H et (Accomodation2 ) against H1 : H et (Accomodation1 ) < H et (Accomodation2 ). Fixing the significance level α = 0.10, 5000 permutations have been computed for the computation of the p-values of the test. We report the observed values of the three test statistics, based respectively on Gini’s index, Shannon’s entropy index and Rényi entropy of order 3, TηG = 0.189, TηS = 0.296 and Tη R3 = 0.277, with respective p-values pηG = 0.0276, pηS = 0.0597 and pη R3 = 0.0273. Thus we can conclude that, at a significance level α = 0.10, the choice about the accomodation by the tourists for whom this is not the first visite in Sesto is less diversified.

3.2 Two-Sided Test: Heterogeneity as a Measure of Uncertainty A survey to determine the level of perception of the hazards of smoking, has been conducted among smokers and nonsmokers. The results are summarized in Table 2. The sample consisted of one hundred persons randomly selected (see [11]). The response variable is ordinal categorical with K = 4 ordered modalities corresponding to increasing levels of hazard. As a matter of fact the larger the heterogeneity of answers is, the higher the uncertainty about the awareness of risks. Looking at Table 2 the group of nonsmokers seems to be more homogeneous because a large proportion choose the answer “Very dangerous”, whereas in the group of smokers the frequencies related to the higher levels of hazard are very similar, denoting a higher level of uncertainty about the awareness of risk. Let us suppose to be not interested to know whether the uncertainty of a group is greater than that of the other one but just to know if uncertainties are not equal. Formalizing we want to test the hypothesis H0 : H et (Perceptionsmokers ) = H et (Perceptionnonsmokers ) against H1 : H et ( Perceptionsmokers ) = H et (Perception nonsmokers ) at the significance level α = 0.01, using the permutation test with 10,000 permutations. The three test statistics, based respectively on Gini’s index, Shannon’s entropy index and Rényi entropy

A Review on Heterogeneity Test: Some Permutation Procedures

19

Table 2 Contingency table of perception of the hazard of smoking, for smokers and nonsmokers [11] Hazard Group Total Smokers Nonsmokers Not dangerous Somewhat dangerous Dangerous Very dangerous Total

9 14 15 11 49

3 6 16 26 51

12 20 31 37 100

Table 3 Contingency table of Medical School Applicants broken down by ethnicity and program choice [5] Ethnicity Program Total Medicine Pediatrics Medicine/Pediatrics White Black Hispanic Asian Total

30 11 3 9 53

35 6 9 3 53

19 9 6 8 42

84 26 18 20 148

of order 3 are TηG = 0.116, TηS = 0.241 and Tη R3 = 0.436, with respective p-values pηG = 0.008, pηS = 0.008 and pη R3 = 0.007. Thus we can conclude that, at a significance level α = 0.01, the heterogeneities of the answers of smokers and nonsmokers are not equal because the uncertainty about hazards of smoking is different in the two groups.

3.3 Multi-sample Test: Ethnic Heterogeneity We are interested to assess wether ethnicity affects the primary care program choice of students at Tulane University. Let us consider the results of a survey [5] reported in Table 3, where the frequency distributions of Medical School applicants according to program (Medicine, Pediatrics or Medicine/Pediatrics) and the race (white, black, hispanic and asian) are reported. We wish to test the hypothesis that heterogeneity of ethnicity in the three groups of applicants are different at the significance level α = 0.05, with a multisample permutation test with 10,000 permutations. More formally the set of hypotheses we want test is H0 : H et (EthnicityMed ) = H et (Ethnicity Ped ) = H et (EthnicityMedPed ) and H1 : H0 not true. The three test statistics, based respectively on Gini’s index, Shannon’s entropy index and Rényi entropy of order 3 are TηG = 0.183, TηS = 0.340 and Tη R3 = 0.500, with respective p-values pηG =

20

S. Bonnini et al.

0.125, pηS = 0.108 and pη R3 = 0.119. Hence the null hypothesis of equality in ethnic heterogeneity of the applicants to three programs should not be rejected at the significance level α = 0.05. Acknowledgements The work was also partially supported by University of Ferrara, which funded the FIR (Research Incentive Fund) project “Advanced Statistical Methods for data analysis in complex problems”. The work was also partially supported by University of Padova BIRD185315/18.

References 1. Agresti, A., Klingenberg, B.: Multivariate tests comparing binomial probabilities, with application to safety studies for drugs. J. R. Stat. Soc. Ser. C (Appl. Stat.) 54, 691–706 (2005) 2. Al-Nasser, A.D.: Customer satisfaction measurment models: generalized maximum entropy approach. Pak. J. Stat. 19(2), 213–226 (2003) 3. Ciavolino, E., Carpita, M.: The GME estimator for the regression model with a composite indicator as explanatory variable. Qual. Quant. 49(3), 955–965 (2015) 4. Ciavolino, E., Al-Nasser, A.D.: Comparing generalized maximum entropy and partial least squares methods for structural equation models. J. Nonparametric Stat. 21(8), 1017–1036 (2009) 5. Doucet, H., Shah, M.K., Cummings, T.L., Kahm, M.J.: Comparison of internal medicine, pediatric and medicine/pediatric applicants and factors influencing career choices. South. Med. J. 92, 296–299 (1999) 6. Frosini, B.V.: Heterogeneity indeces and distances between distributions. Metron 34, 95–108 (1981) 7. Golan, A., Judge, G.G., Miller, D.: Maximum Entropy Econometrics: Robust Estimation with Limited Data. Wiley, New York (1996) 8. Han, K.E., Catalano, P.J., Senchaudhuri, P., Mehta, C.: Exact analysis of dose-response for multiple correlated binary outcomes. Biometrics 4(60), 216–224 (2004) 9. Hirotsu, C.: Cumulative chi-squared statistic as a tool for testing goodness-of-fit. Biometrika 73, 165–173 (1986) 10. Jaynes, E.T.: Information theory and statistical mechanics. Phys. Rev. 106(4), 620–630 (1957) 11. Kvam, P.H., Vidakovic, B.: Nonparametric Statistics with Applications to Science and Engineering. Wiley, Hoboken (2007) 12. Leti, G.: Sull’entropia, su un indice del Gini e su altre misure dell’eterogeneità di un collettivo. Metron 24, 332–378 (1965) 13. Loughin, T.M.: A Systematic comparison of methods for combining p-values from independent tests. Comput. Stat. Data Anal. 47, 467–485 (2004) 14. Loughin, T.M., Scherer, P.N.: Testing for association in contingency tables with multiple column responses. Biometrics 54, 630–637 (1998) 15. Lumley, T.: Generalized estimating equations for ordinal data: a note on working correlation structures. Biometrics 52, 354–361 (1996) 16. Nettleton, D., Banerjee, T.: Testing the equality of distributions of random vectors with categorical components. Comput. Stat. Data Anal. 37, 195–208 (2001) 17. Piccolo, D.: Statistica, 2nd edn. Il Mulino, Bologna (2000) 18. Pesarin, F., Salmaso, L.: Permutation tests for univariate and multivariate ordered categorical data. Aust. J. Stat. 35, 315–324 (2006) 19. Pesarin, F., Salmaso, L.: Permutation Tests for Complex Data. Theory, Applications and Software. Wiley, Chichester (2010) 20. Shorack, G.R., Wellner, J.A.: Empirical Processes with Applications to Statistics. Wiley, New York (1986)

Robust Estimation of Skew-Normal Parameters with Application to Outlier Labelling Mario Romanazzi

Abstract We suggest to estimate the parameters of the skew-normal distribution by the method of moments, modified so as to achieve robustness. A type of trimmed estimator is used, with the trimming fraction depending on a given scaled deviation from the center. An application to outlier labelling is illustrated. Keywords Trimmed mean · δ-trimmed moments · Quantile-based estimator · Skew-t · Boxplot

1 Introduction Skew-normal distributions (SND) [1] provide a broad class of unimodal probability distributions allowing a flexible yet simple modelling of data in terms of location, spread and skewness. Maximum likelihood (ML) is the standard criterion to fit SND to data. However, ML estimates are not resistant to outliers and it is advisable to resort to some type of robust estimation procedure. Quantile-based estimation, though appealing in principle, in practice proves difficult, just because of the type of parameterization of SND. Moment-based estimation, modified so as to achieve robustness, appears more convenient and is the route pursued here. In Sect. 2 details of both methods are provided for the univariate skew-normal distribution. Section 3 is devoted to an application to outlier labelling. A final discussion is reported in Sect. 4.

M. Romanazzi (B) Department of Environment, Informatics and Statistics, Ca’ Foscari University, Venice, Italy e-mail: [email protected] © Springer Nature Switzerland AG 2019 C. Crocetta (ed.), Theoretical and Applied Statistics, Springer Proceedings in Mathematics & Statistics 274, https://doi.org/10.1007/978-3-030-05420-5_3

21

22

M. Romanazzi

2 Robust Estimation SND considerably enlarge the standard paradigm of fitting a normal distribution to data so as to cover symmetry exceptions that are frequently observed in practice and sometimes suggested by subject-matter considerations. Practical implementations are not exempt from risks of distortions, however. We argue below that ML fit tends to overweight sparse data sometimes encountered in real samples, much more dangerously than the well-documented normal situation. Flexible modelling should then be accompanied by fitting methods able to reduce sensitivity of estimators to sparse data. It is in this sense that robust estimation is employed in the present contribution. Below the notation about the skew-normal distribution and its parameters is recalled and two robust estimation procedures are described, the first one based on quantiles, the other based on trimmed moments.

2.1 Skew-Normal Distribution The density function of the standard skew-normal distribution U is f S N (z) = 2φ(z)Φ(θ3 z) ,

(1)

where φ (Φ) denotes the density (cumulative distribution) function of the standard normal distribution. The parameter θ3 ∈ R accounts for the asymmetry of the distribution. When θ3 = 0, the ordinary, symmetric, standard normal distribution is obtained. Setting z = (x − θ1 )/θ2 in (1), it is possible to change location and scale by means of the parameters θ1 ∈ R and θ2 > 0. The general, non standardized, skewnormal distribution will be denoted S N (θ ), with parameter vector θ = (θ1 , θ2 , θ3 )T . The components of θ are called direct parameters (DP). An alternative parameterization is in terms of the centred parameters (CP) μ, σ , γ1 , where μ and σ 2 are the expectation and variance operators and, for a general random variable X , γ1 = E{(X − μ)/σ }3 . The skewness parameter γ1 is equal to zero at the normal distribution. CP vector θ = (μ, σ, γ1 )T is often preferred to DP vector because of a more intuitive interpretation. Reference [1] reports expressions of CP as functions of DP for both skew-normal and skew-t families and thoroughly discusses pros and cons of both parameterizations. Maximum likelihood is the standard criterion to fit a skew-normal distribution to data. An easy-to-use suite of optimization functions is the R package sn available at CRAN [2]. However, ML estimates are not outlier resistant and to prevent masking in outlier labelling it is imperative to use robust estimation procedures. Roughly speaking, two broad families of robust estimators of CP parameters can be envisaged, quantile-based and moment-based estimators.

Robust Estimation of Skew-Normal Parameters …

23

2.2 Quantile-Based Estimation For 0 < p < 1, we denote with Q F ( p) the p-th order quantile of distribution F. The standard quantile measure of location is the median Q F (.5). Popular quantile measures of spread are the interquartile range I Q R F = Q F (.75) − Q F (.25) and the median absolute deviation (from the median) M AD F . For 0 < p < 0.5, a general quantile measure of skewness is S K F ( p) = (Q F (1 − p) − 2Q F (.5) + Q F ( p))/(Q F (1 − p) − Q F ( p)) [4]. By putting p = 0.25 or p = 0.125, one obtains quartile skewness Q S K F (0.25) or octile skewness Q S K F (0.125), respectively. Quartile skewness is very resistant to outliers because it uses the 50% most central part of the distribution, only. On the other hand, octile skewness is more informative on the actual asymmetry of the distribution while preserving good outlier resistance because the extreme octiles are used. A quantile measure of kurtosis also dependent on octiles is K S F = (Q F (.875) − Q F (.625) + Q F (.375) − Q F (.125))/(Q F (.75) − Q F (.25)). The reader is referred to [5] for a discussion of octile kurtosis. The idea is to extend the familiar method of moments to quantiles. Let Fϑ be a parametric family of probability distributions depending on a K -dimensional parameter ϑ = (ϑ1 , . . . , ϑ K )T and let Fˆn be the empirical distribution function of a random sample X 1 , . . . , X n from Fϑ . We denote with ψˆ n = (ψˆ n,1 , . . . , ψˆ n,K )T a K -dimensional vector of functions of the sample quantiles. Typical ψ functions are ψˆ n,1 = Q n (.5), the median, ψˆ n,2 = I Q Rn , the interquartile range and so on. To each empirical ψ function there corresponds the theoretical version ψk ≡ ψk (Fϑ ). For example, the theoretical ψ function associated to the interquartile range is Q Fϑ (.75) − Q Fϑ (.25). The notation emphasizes that the theoretical ψ’s are functions of ϑ parameter. To derive the estimates of ϑ1 , . . . , ϑ K we consider the system of K (possibly non linear) equations ⎧ n,1 ⎨ ψ1 (Fϑ ) = ψ ... = ... ⎩ n,K ψ K (Fϑ ) = ψ to be solved with respect to ϑ1 , . . . , ϑ K . The solution for ϑk has the form ϑˆ k = τk (ψˆ ) and is a function of the empirical ψ’s, k = 1, . . . , K . Example 1 For the exponential distribution K = 1 and a family of ψ functions is offered by Q F ( p), 0 < p < 1. Quantile estimators of ϑ are explicitly obtained from Fϑ (Q Fϑ ( p)) = 1 − exp(−Q Fϑ ( p))/ϑ) = p, i. e., ϑˆ p = −Q n ( p)/ ln(1 − p), a scale transformation of p-th order sample quantile. The parameter p can be chosen so as to achieve specific properties, i. e., robustness and efficiency. For instance, to minimize the asymptotic variance, it must be p = 0.7968. The corresponding estimator can be considered the quantile counterpart of ML (and moment) estimator, the sample mean. Example 2 For the normal distribution K = 2 and ϑ1 = E(X ), ϑ2 = V ar (X ). Using the median and the interquartile range as ψ functions, it is not difficult to

24

M. Romanazzi

see that the quantile estimates are ϑˆ 1 = Q n (.5) and ϑˆ 2 = (I Q Rn /I Q R N (0,1) )2 , to be compared with ML (and moment) estimators, the sample mean and the sample variance. Here I Q R N (0,1)  1.34898 is the interquartile range of the standard normal distribution. Example 3 Consider the uniform distribution on the interval (ϑ1 , ϑ2 ), ϑ1 < ϑ2 . Again using the median and the interquartile range as ψ functions, as (ϑ1 + ϑ2 )/2 = Q Fϑ (.5) and (ϑ2 − ϑ1 )/2 = I Q R Fϑ , we get the estimators ϑˆ 1 = Q n (.5) − I Q Rn , ϑˆ 2 = Q n (.5) + I Q Rn . The corresponding ML estimators are the minimum and maximum of the sample observations. For the skew-normal distribution K = 3 and the median, the interquartile range and the octile skewness are used as ψ functions. Explicit solutions are not available but numerical solutions are easily obtained by non linear equation system solvers. As in the general case, the estimates identify the specific skew-normal distribution whose parameters agree with the empirical ψ’s. Some general properties follow from the properties of sample quantiles. For example, if Fϑ is continuous, then the quantile estimators of ϑ are consistent and asymptotically normally distributed [6, Theorem A, p. 122]. Example 4 For 0 < p < 1, the quantile estimators ϑˆ p in Example 1 have an asymptotically normal distribution N (ϑ, c p ϑ/n 1/2 ). For p = .5, c.5  1.443 and for the optimal case p = p ∗ , c p∗  1.243. It is clear that outlier resistance is obtained at the price of a large efficiency loss with respect to ML (moment) estimator.

2.3 Moment-Based Estimation For the skew-normal distribution, CP are based on moments and therefore it seems more natural to look for estimators that are robust versions of sample moments. Trimmed and winsorized means are popular examples. A recent proposal by [8] is to trim data outside a given scaled deviation from the center. The resulting estimator of the mean definitely improves over the classical sample trimmed mean and appears to be competitive with quantile-based estimators, both in terms of robustness and efficiency. Let Δn (X i ) = (X i − Q n (.5))/M ADn ), i = 1, . . . , n, and let δ be a positive value of Δn . The δ-trimmed mean Mδ (X 1 , . . . , X n ) ≡ Mδ,n is defined as follows n i=1 wΔ (X i )X i , (2) Mδ,n =  n i=1 wΔ (X i ) where wΔ (.) is a weighting function that is identically zero if |Δn (X i )| > δ. Two sim(1) (X i ) = a if |Δn (X i )| ≤ δ and ple weighting functions are the constant function wΔ (2) −1 wΔ (X i ) = {1 + |Δn (X i )|} if |Δn (X i )| ≤ δ. Reference [8] recommends choosing δ in the interval [4, 7] or making an adaptive choice of δ, according to sample

Robust Estimation of Skew-Normal Parameters …

25

0.20

Left and Right Trimmed Probabilities Left trim SN(0, 1, −1) Right trim SN(0, 1, −1) Left trim SN(0, 1, 5) Right trim SN(0, 1, 5)

●

0.15

●

0.10

● ●

● ●

0.05

Trimmed Probability

●

●

●

●

●

0.00

● ●

1

2

● ● ●

● ● ●

● ●

3

● ●

4

●

●

5

●

●

6

●

●

7

K

Fig. 1 Left and right trimmed probabilities for two skew-normal distributions. Left trim is P(U < Q U (.5) − k × M ADU ), right trim is P(U > Q U (.5) + k × M ADU ). U ∼ S N (0, 1, ϑ3 ), ϑ3 ∈ {−1, 5}

features. The main difference between the classical trimmed mean and the new δtrimmed mean is that in the second case trimming takes place only if sparse data are present, i. e., observations X ∗ with |Δn (X ∗ )| > δ. For each δ, the trimming fraction of the sample estimator is random but the behaviour can be illustrated from the corresponding population functional. Figure 1 shows the trimmed probabilities corresponding to several values of δ for two skew-normal distributions. The (total) trim of δ ≥ 4 is lower than 1% suggesting a good compromise between outlier resistance and efficiency. We suggest extending (2) to δ-trimmed moments and central moments of the order k through (k) Mδ,n

n n (1) k k (k) i=1 wΔ (X i )(X i − Mδ,n ) i=1 wΔ (X i )X i n , M δ,n = . = n i=1 wΔ (X i ) i=1 wΔ (X i )

(3)

This gives at once trimmed moment (TM) estimators of CP. The estimator of μ is (1) Mδ,n ≡ Mδ,n , the estimator of σ 2 follows from the second of (3) with k = 2 and the (3)

(2)

3/2 estimator of γ1 is G (1) . δ,n = M δ,n /{M δ,n }

26

M. Romanazzi

Table 1 Monte Carlo expectations and standard errors of maximum likelihood and alternative estimators. Parent distribution is S N (0, 1, 2) Sample Parameter Perturbation θ1 = 0 θ2 = 1 θ3 = 2 Unperturbed Perturbed, c = 4 Perturbed, c = 6 Unperturbed Perturbed, c = 4 Perturbed, c = 6 Unperturbed Perturbed, c = 4 Perturbed, c = 6

Maximum likelihood estimates .070 ± .29 .983 ± .14 −.098 ± .12 1.14 ± .10 −.191 ± .09 1.29 ± .08 δ-trimmed moment estimates .113 ± .29 .949 ± .13 .084 ± .28 .969 ± .15 .110 ± .29 .951 ± .13 Quantile-based estimates .243 ± .49 .963 ± .16 .212 ± .48 .987 ± .16 .213 ± .48 .987 ± .16

2.33 ± 6.0 3.95 ± 13 4.22 ± 6.1 1.86 ± 1.69 2.16 ± 2.45 1.91 ± 1.90 1.74 ± 2.02 1.92 ± 2.09 1.92 ± 2.09

Example 5 M = 1000 samples of n = 100 observations were simulated from U ∼ S N (ϑ), ϑ = (0, 1, 2)T and ML, TM (with δ = 4) and quantile estimators were evaluated. The same estimators were recomputed on the samples augmented by a single observation c located on the right tail of the distribution. Two different values of c were tried, c = 4 and c = 6. The scaled deviations of the contaminating data with respect to the nominal distribution U are ΔU (4)  4.96 and ΔU (6)  7.93. Monte Carlo expectations and standard errors of the estimators are reported in Table 1. The nominal values of the parameters are outside the 50% innermost interval of the distribution of the perturbed ML estimators which is a clear signal of non resistance. (1) provide a reasonable On the contrary, the TM estimators with weight function wΔ compromise in terms of bias, standard error and outlier resistance. The quantile estimators are very stable under the perturbation but they tend to exhibit slightly higher (2) standard errors than TM estimators. The TM estimators with weight function wΔ (results not shown in Table 1) are also outlier resistant but they are much more biased than TM estimators with constant weight function. Example 5 shows that ML estimation is exposed to the risk of distortion by just a few outliers. In particular, the estimate of θ3 seems highly vulnerable. Therefore, the sample must be carefully cleaned before fitting a distribution or some type of robust fit must be used. The TM estimators, especially with constant weight function, are appealing because, in addition to providing outlier resistance, they are scaled as the CP and hence directly comparable with them. A particular problem encountered with TM estimators is discussed in the following remark. Remark 1 For skew-normal distributions it must hold |γ1 | ≤ .995 but G (1) δ,n need not comply with such requirement. When this happens, we suggest to use projection

Robust Estimation of Skew-Normal Parameters …

27

on the feasible interval, which amounts to replace the TM estimator of γ1 with (1) (1) G˜ (1) δ,n = min{|G δ,n |, .995}sign(G δ,n ). By definition, quantile-based estimators do not suffer from this problem.

3 Outlier Labelling Let F0 be the skew-normal distribution best fitting the observed sample X 1 , . . . , X n and let 0 < α < 1. Similarly to Tukey’s outlier labelling rule, we look for functions, called fences, FI(0) (α, n) = Q n (.25) − K I(0) (α, n)I Q Rn , FS(0) (α, n)

= Q n (.75) +

K S(0) (α, n)I Q Rn

(4)

,

satisfying n (FI(0) (α, n) ≤ X i ≤ FS(0) (α, n))) = 1 − α . P(∩i=1

(5)

Therefore, under F0 , the interval [FI(0) (α, n), FS(0) (α, n)] has a probability 1 − α of including all sample observations, with α playing the same role as the significance level in hypothesis testing [3]. Let αn be the probability of a random observation X from F0 to be outside the fences (4). As the X i ’s are assumed to be independent and identically distributed, it follows   n FI(0) (α, n) ≤ X i ≤ FS(0) (α, n) = (1 − αn )n (6) 1 − α = P ∩i=1 or αn = P(X < FI(0) (α, n) ∪ X > FS(0) (α, n)) = 1 − (1 − α)1/n .

(7)

Assuming the disjoint events A I (α, n) : X < FI(0) (α, n), A S (α, n) : X > FS(0) (α, n) to have the same probability, it follows that FI(0) (α, n) and FS(0) (α, n) are quantiles of F0 of orders αn /2 = (1 − (1 − α)1/n )/2 and 1 − αn /2. When F0 is a parametric model, a naive estimate is obtained by replacing the unknown parameters by their sample estimates, e. g., ML or TM estimates illustrated in Sect. 2. A possible difficulty is low outlier resistance and high standard error of extreme quantiles. An alternative approach is to express the fences as functions of the quartiles as in (4) and estimate the quantities K I(0) (α, n), K S(0) (α, n) complying with the required αn . Reference [7] suggests an iterative algorithm based on properties of the order statistic.

28

M. Romanazzi

Example 6 The performance of the outlier labelling rule was studied with Monte Carlo experiments. The proportions of detected outliers (true positives, TP) and regular data wrongly labelled as outliers (false positives, FP) were estimated under several conditions using the naive percentile method. Sample size was n = 200 and the number of Monte Carlo replicates was N = 1000. Contamination was modelled as gaussian error in the far right tail of the nominal distribution. Contamination proportions 0, 5, 10% were used. The value of all-inside-rate 1 − α was set to 99%. Ideally, TP rate should match contamination proportion and FP rate should match the pre-set value of αn = 1 − (1 − α)1/n . Tukey’s rule was used as a benchmark. When the nominal distribution of the sample was skew-normal, both methods performed

Table 2 Contaminated skew-normal. Monte Carlo evaluation of Tukey’s (TK) and skew-normal (SN) outlier labelling rule (FP: false positives, TP: true positives, %) Contamination 0% γ1 = .4 γ1 = .8 TK SN TK SN F P = 1.1, T P = 0 F P = 0.0, T P = 0 F P = 1.6, T P = 0 F P = 0.0, T P = 0 Contamination 5% γ1 = .4 γ1 = .8 TK SN TK SN F P = 0.6, T P = 4.6 F P = 0.0, T P = 4.0 F P = 0.9, T P = 4.6 F P = 0.1, T P = 3.9 Contamination 10% γ1 = .4 γ1 = .8 TK SN TK SN F P = 0.2, T P = 9.0 F P = 0.3, T P = 7.5 F P = 0.3, T P = 9.0 F P = 0.6, T P = 7.4

Table 3 Contaminated exponential. Monte Carlo evaluation of Tukey’s (TK) and skew-normal (SN) outlier labelling rule (FP: false positives, TP: true positives, %) Contamination 0% λ=1 λ = 1.5 TK SN TK SN F P = 4.9, T P = 0 F P = 0.6, T P = 0 F P = 4.9, T P = 0 F P = 0.6, T P = 0 Contamination 5% λ=1 λ = 1.5 TK SN TK SN F P = 3.2, T P = 4.7 F P = 0.4, T P = 4.3 F P = 3.2, T P = 4.8 F P = 0.4, T P = 4.3 Contamination 10% λ=1 λ = 1.5 TK SN TK SN F P = 1.7, T P = 9.2 F P = 0.3, T P = 8.2 F P = 1.8, T P = 9.4 F P = 0.3, T P = 8.9

Robust Estimation of Skew-Normal Parameters …

29

well (Table 2), with FP and TP rates near to the optimal values even for |γ1 | values as high as .8. Different results were obtained when the nominal distribution of the sample was exponential with parameters λ = 1 and λ = 1.5 (Table 3). In this case, Tukey’ rule has a high FP rate while keeping a good TP rate. The newly suggested method exhibits a FP rate close to the right value and a TP rate a little worse than Tukey’s. Overall, it appears superior. This result should not be underrated because we are fitting a skew-normal distribution with unimodal density to an exponential sample.

4 Conclusion There is a wealth of potential applications of SND in data analysis and the goal of the present work is to suggest an estimation method less prone to distortion by outliers than ML. Evidence provided in Example 5 confirms that ML estimates can be made to miss the true parameters by just a few observations far enough from the center. As shown by Table 1, in these situations the skewness estimate is particularly vulnerable. The suggestion for practitioners is to use ML method on the sample cleaned of outliers, or to use a robust method. TM method has several favourable features, beyond robustness. Reference parameters are just CP parameters. It extends easily to skew-t, through TM estimate of (4) (2) 2 (4) kurtosis G (2) δ,n = M δ,n /{M δ,n } , where Mδ,n is obtained by setting k = 4 in (3). It also extends to the multivariate case, where the trimming is based on depth induced regions [9]. On the negative side, as shown by Remark 1, TM estimators may require projection on parameter space to identify a proper skew-normal (skew-t) distribution. Quantile-based estimators are theoretically appealing but they seem to have poorer efficiency. The outlier labelling rule proposed in Sect. 3 has two useful features. First, it gives control of FP rate through the pre-set value of α. Second, it is flexible enough to adapt to sample features, that is, outliers are extreme observations with respect to the skew-normal distribution best fitting the sample, which includes asymmetry. Here also, extension to skew-t is not difficult. Results given here are based on the naive percentile method and may require refinements with low sample sizes. A surprising result is that Tukey’s outlier labelling rule behaves well under a (contaminated) skew-normal distribution.

References 1. Arellano-Valle, R.B., Azzalini, A.: The centred parameterization and related quantities of the skew- t distribution. J. Multivar. Anal. 113, 73–90 (2013) 2. Azzalini A (2014) The skew-normal and skew-t distributions. R package version 1.1-1

30

M. Romanazzi

3. Davies, L., Gather, U.: The identification of multiple outliers. J. Am. Stat. Assoc. 88, 782–792 (1993) 4. Hinkley, D.V.: On power transformations to symmetry. Biometrika 62, 101–111 (1975) 5. Moors, J.J.A.: A quantile alternative for kurtosis. The Statistician 37, 25–32 (1988) 6. Serfling, R.: Approximation Theorems of Mathematical Statistics. Wiley, New York (1980) 7. Sim, C.H., Gan, F.F., Chang, T.C.: Outlier labeling with boxplot procedures. J. Am. Stat. Assoc. 100, 642–652 (2005) 8. Wu, M., Zuo, Y.: Trimmed and winsorized means based on a scaled deviation. J. Stat. Plan. Inference 139, 350–365 (2009) 9. Zuo, Y.: Multidimensional trimming based on projection depth. Ann. Stat. 34, 2211–2251 (2006)

Asymptotics of S-Weighted Estimators Jan Ámos Víšek

Abstract The paper studies S-weighted estimator - a combination of S-estimator and the least weighted squares. The estimator allows to adjust the properties, namely the level of robustness of estimator in question to the processed data better than the S-estimator or the least weighted squares can do. The paper offers the proof of its √ n-consistency. Keywords Robustness · Implicit weighting · The order statistics of the absolute values of residuals · The consistency of the SW-estimator under heteroscedasticity

1 An Overview of Present State The presence of order statistics of squared residuals and above all the discontinuity of objective function in the definitions of the least median of squares (LMS) [10] and the least trimmed squares (LTS) [6] represented a nontrivial problem when looking for asymptotic properties of these estimators.1 S-estimators [12] get rid of both problems at the cost of a bit, but acceptably, restricted objective function. May be that more restrictive is the fact that the influence of individual observation is controlled only by the objective function. It seems - as the results of simulations hint - that allowing for prescribing the weights to the observations according to the magnitude of the respective order statistics of the squared residuals can bring substantially larger flexibility of estimator. It is than able to accommodate not only to the level of contamination but also to the character of data and of contamination, their mutual position. Nevertheless, at the middle of eighties we hoped that the battle for finding 50% breakdown point estimator which can be (relatively) easy implemented and for which we can prove (again relatively easy) the consistency has 1 That

is why that although LTS was defined in eighties, the general proofs of its properties appeared as late as in 2006, see [16].

J. Á. Víšek (B) Charles University, Opletalova 26, 110 01 Prague, Prague 1, Czech Republic e-mail: [email protected] © Springer Nature Switzerland AG 2019 C. Crocetta (ed.), Theoretical and Applied Statistics, Springer Proceedings in Mathematics & Statistics 274, https://doi.org/10.1007/978-3-030-05420-5_4

31

32

J. Á. Víšek

Table 1 Estimated coefficients of the regression model for correct and damaged data, respectively Data Intercept SPARK AIR TEMP-IN TEMP-EX Correct data 35.11 (x22 = 14.1) Damaged data −88.7 (x22 = 15.1)

−0.028

2.949

0.477

4.72

1.06

1.57

−0.009 0.068

(for more details see [17]). A very first idea can “yield” an academic explanation given in the Fig. 1

been won. But results by Hettmansperger and Sheather [8] “discovered” an inevitable high sensitivity of estimators with the zero-one objective function and with the high breakdown point with respect to a small change of one item (they made the only error - wrote wrongly x22 = 15.1 instead of correct value 14.1, see [8]). Although it appeared later that their results had been misleading due to a bad algorithm which they employed for computing LMS, they have started a discussion on the topic. One immediately notice that the dataset (nowadays referred to as the Engine Knock Data) is so small (it contains 16 observations, see again [8]) that we can compute βˆ (L T S,n,h) exactly just computing the ordinary least squares (OLS) for all subsamples of data, the subsamples which contain 11 observations (the recommendation to minimize the sum of 11 smallest order statistics of the squared residuals is based on the theoretical results by Rousseeuw and Leroy [11]). We obtain Table 1. At the first glance it may seem that the “switch” of the estimator of underlying model is due to the high breakdown point of the estimator in question. However, taking into account the results for βˆ (L T S,n,h) given in Table 1 and the fact that h = 11, we conclude that the “switch effect” can appear even for much lower breakdown point - in the case of employing βˆ (L T S,n,h) on Engine Knock Data we have the breakdown point adjusted on the level only a bit larger than 30%. It indicates that the reason for βˆ (L T S,n,h) to exhibit the “switch effect” is the zero-one objective function.2 So, the conclusion is that the high breakdown point and/or zero-one objective function can cause this unpleasant behaviour of estimator. The least weighted squares (LWS) removed this drawback. On the other hand, LWS allows for only one objective function - the quadratic one. In other words, the problem of depressing the influence of suspicious observations is left on the weight function. The experiences from the numerical experiments (see e.g. a very large study in [19]) it follows that the weight function (for examples see Appendix) can very well cope with any type of contamination. √ The present paper proves the n-consistency of S-weighted estimator - a combination on S-estimator and LWS which preserves pros of LTS, LWS and S-estimators, starting with scale- and regression-equivariance (in contrast with M-estimators) reached without any studentization, the high rate of convergence and ending e.g. with a reliable and quick algorithm. But it adds one notable benefit: A possibility to

2 It

is clear from Fig. 1 that any estimator with high breakdown point generally suffer by “switch effect” but the Engine Knock Data indicate that it can happen for much lower breakdown point.

Asymptotics of S-Weighted Estimators

33

Fig. 1 Notice that in both cases the estimator explains the most of data although the difference between the left and the right part of figure is a position of small bullet (close to origin) which is shifted from position below the axe x (in the left-hand part of the figure) to the position above the axe x (in the right-hand part)

tailor, by a combination of proper objective and weight functions, the estimator just for processed data.

2 Fixing the Framework Let N denote the set of all positive integers, R the real line and R p the p-dimensional Euclidean space. All vectors will be assumed to be the column ones and throughout the paper we assume that all r.v.’s are defined on a basic probability space P),  (Ω, A , ∞ say. For a sequence of ( p + 1)-dimensional random variables (r. v.’s) (X i , ei ) i=1 , for any n ∈ N and a fixed β 0 ∈ R p the linear regression model given as Yi = X i β 0 + ei , i = 1, 2, . . . , n,

(1)

will be considered.

3 S-Weighted Estimator and Its Consistency ∞  Conditions C 1. The sequence (X i , ei ) i=1 is sequence of independent ( p + 1)-dimensional random variables (r.v.’s) with distribution function FX,ei (r, u) = (2) (r, u) where F (1) (r (1) ) : R 1 → [0, 1] is distribution function (d. f.) F (1) (r (1) ) · FX,e i (2) (x, r ) = FX (x) · Fei (r ) where degenerated at 1 (to allow for intercept) and FX,e i −1 Fei (r ) = Fe (r σi ). Moreover, Fe (r ) (a parent d. f.) is absolutely continuous with distributed according to d.f. density f e (r ) bounded by Ue . Further, let e be a r. v. n σi = 1. Finally, there is Fe with IE Fe e = 0, var Fe (e) = σ0 = 1 and limn→∞ n1 i=1

34

J. Á. Víšek

∞ q > 1 so that IE FX X 1 2q < ∞ (as FX (x) doesn’t depend on i, the sequence {X i }i=1 is sequence of independent and identically distributed (i.i.d.) r.v.’s). Conditions C 2. The weight function w : [0, 1] → [0, 1] is a continuous, nonincreasing function with w(0) = 1. Moreover, w is Lipschitz in absolute value, i. e. there is L such that for any pair u 1 , u 2 ∈ [0, 1] we have |w(u 1 ) − w(u 2 )| ≤ L · |u 1 − u 2 | . Further, the objective function ρ : (0, ∞) → (0, ∞), ρ(0) = 0, is non-decreasing on (0, ∞), symmetric and differentiable (denote the derivative of ρ ) by ψ). Finally, ψ(r )/r is non-increasing for r ≥ 0 with limr →0+ ψ(r = 1. r

Definition 1 Let w : [0, 1] → [0, 1] and ρ : [0, ∞] → [0, ∞] be a weight function and an objective function, respectively. Further, let F|e| (v) = P (|e| < v). Moreover, put for any β ∈ R p ri (β) = Yi − X i β and denote (a bit non-traditionally) the i-th order statistics among the absolute values of residuals by |r |(i) (β), i. e. |r |(1) (β) ≤ |r |(2) (β) ≤ · · · ≤ |r |(n) (β). Then  βˆ (SW,n,w,ρ) = arg min σ (β) ∈ R + : β∈R P

    n |r |(i) (β) 1 i −1 ρ =b w n i=1 n σ

(2)



  where b = IE w F|e| (s|e|) ρ(e) , is called the S-weighted estimator (SW estimator, for short), see [20].

Notice please that we cannot write in (2) simply ρ ri σ(β) because then we would

 to other residual. A chain of technicalities, see [12], leads assign the weight w i−1 n to a conclusion that βˆ (SW,n,w,ρ) has to be one of solutions of the normal equations  

Yi − X i β (n) =0 w Fβ,σ (|ri (β)|) X i ψ σ i=1

n 

(3)

(n) where Fβ,σ (r ) is the empirical distribution function of the absolute values of residuals. Moreover, the technical steps show that the solution does not depend on σ . Now, we are going to employ the idea of Frank Hampel (see [6]) that the information given by observations z 1 , z 2 , . . . , z n (say) is the same as the information represented by the corresponding empirical distribution function. Taking into account some techni(n) (n) (r ) be a continuous strictly increasing modification of Fβ,σ (r ) cal reasons, let F˜β,σ (n) (n) ˜ defined as follows: Let Fβ,σ (r ) coincide with Fβ,σ (r ) at |ri (β)|, i = 1, 2, . . . , n and let it be continuous and strictly monotone between any pair of |r |(i) (β)

 and ˜ ) = w(r ) · ψ σr · σr . |r |(i+1) (β). Moreover, recalling that ψ = ρ  , let us put w(r Finally, the asymmetry of ψ together with the fact that ψ(0) = 0 allows to write the normal Eq. (3) as



  r (β)   (n)

 σ i · X i Yi − X i β w Fβ,σ (|ri (β)|) ψ σ ri (β) {i : r (β) =0} i

Asymptotics of S-Weighted Estimators

=

n 





 (n) w˜ F˜β,σ (|ri (β)|) X i Yi − X i β = 0.

35

(4)

i=1

Notice please, that w˜ is well defined and it fulfill C 2. We will need the following identification condition. Conditions C 3. For any fixed σ > 0 there is the only solution of   n



 (n) 



0   0 =0 w˜ F β 0 ,σ (|ei |) X i ei − X i β − β β − β IE

(5)

i=1

namely β = β 0 where (n)

F β 0 ,σ (|r |) =

n  

 1 Fi,β 0 (r ) with Fi,β (r ) = P Yi − X i β  < r . n i=1

(6)

Theorem 1 Let Conditions C 1, C 2 and C 3 be fulfilled. Then any sequence  ∞ βˆ (SW,n,w,ρ) of the solutions of sequence of normal Eq. (4) for n = 1, 2, . . ., n=1 is weakly consistent. For the proof see [20] and the discussion there.

4

√

n-Consistency of S-Weighted Estimator

Conditions N C 1. The derivative f e (r ) exists and it is bounded in absolute value by Be . The derivative w˜  (α) exists and it is Lipschitz of the first order (with the corresponding constant Jw ). Moreover, for any i ∈ N   (n) E w˜  (F β 0 ,σ (|ei |)) ( f e (|ei |) − f e (−|ei |)) · ei = 0.

Theorem 2 Let Conditions C 1, C 2, C 3 and N C 1 hold. Then any sequence  ∞ √ βˆ (SW,n,w,ρ) of solutions of the normal Eq. (4) is weakly n-consistent, i. e. n=1 ∀(ε > 0) ∃(K ε < ∞) ∀(n ∈ N )    √    P ω ∈ Ω : n βˆ (SW,n,w,ρ) − β 0 ) < K ε > 1 − ε. The key tool for proving Theorem 2 is the lemma: Lemma 1 Let Condition C 1 hold. Then for any ε > 0 there is a constant K ε and n ε ∈ N so that for all n > n ε

36

J. Á. Víšek

 P

 √  (n) (n)  ω ∈ Ω : sup sup sup n  F˜β,σ (r ) − F β,σ (r ) < K ε



r ∈R + β∈R p σ ∈R +

> 1 − ε.

(7)

Lemma 1 is a slight generalization of Lemma 1 of [18]. The generalization is allowed (n) (r ) and by the fact that R p × R + is separable space. by the monotonicity of F˜β,σ The proof employs Skorohod’s embedding into Wiener process (see [3]) which was for statisticians “rediscovered” by Stephan Portnoy [9]. We will need several other lemmas. The proofs of them consist of a chain of straightforward steps of findings on the upper bound of respective expressions. Hence we give only some hints. Lemma 2 Let Conditions C 1, C 2, C 3 and N C 1 hold. Then, as n → ∞,   (n)  (n) sup sup sup sup F β,σ (r ) − F β 0 ,σ (r ) − f e (−r · σi−1 ) β∈R p σ ∈R + r ∈R + i∈N

−2      + f e (r · σi−1 ) IE FX X 1 · β − β 0  · β − β 0  = O(1)    −1 (n)  (n)  sup sup sup  F β,σ (r ) − F β 0 ,σ (r ) · β − β 0  = O(1).

and

β∈R p σ ∈R + r ∈R +

(8)

(9)

Proof of lemma is just a computation of some integrals with utilization of Taylor expansion. Lemma 3 Let Conditions C 1, C 2, C 3 and N C 1 hold. Then, as n → ∞,  n  1  

(n)

 (n)  w˜ F β,σ (|ri (β)|) − w˜ F β 0 ,σ (|ri (β)|) sup sup  β∈R p σ ∈R +  n i=1  − w˜



(n) (F β 0 ,σ (|ri (β)|)) ·

(n) F β,σ (|ri (β)|) −

 (n) F β 0 ,σ (|ri (β)|)

   X i ei  · β − β 0 −2 = O p (1) 

and for any , k = 1, 2, . . . , p  n  1  

(n)

 (n)  w˜ F β,σ (|ri (β)|) − w˜ F β 0 ,σ (|ri (β)|) sup sup sup  β∈R p σ ∈R + v∈R +  n i=1   (n)  (n) (n)  −w˜  (F β 0 ,σ (|ri (β)|)) · F β,σ (|ri (β)|) − F β 0 ,σ (|ri (β)|) X i X ik  × × β − β 0 −2 = O p (1). Proof of this lemma is a slight generalization of Lemma 3 in [17]. Let for any a, b ∈ R [a, b]or d = [min{a, b}, max{a, b}].

Asymptotics of S-Weighted Estimators

37

Lemma 4 Let Conditions C 1, C 2, C 3 and N C 1 hold. Then, as n → ∞, sup β∈R p

n  1    (n) (n)  w˜ (F β 0 ,σ (|ri (β)|)) − w˜  (F β 0 ,σ (|ri (β 0 )|)) × n i=1

   −2 (n)  (n)  × F β,σ (|ri (β)|) − F β 0 ,σ (|ri (β 0 )|) · X i ei  · β − β 0  = O p (1) (n)

(n)

and for any , k = 1, 2, . . . , p and any ξi ∈ [F β,σ (|ri (β)|), F β 0 ,σ (|ri (β)|)]or d  n 1   (n)  w˜  (ξi ) − w˜  (F β 0 ,σ (|ri (β)|)) ×  n i=1    (n)  −2 (n)  × F β,σ (|ri (β)|) − F β 0 ,σ (|ri (β)|) · X i · X ik  · β − β 0  = O p (1). Proof of Lemma 4 as well as of Lemma 5 is again a chain of simple steps establishing the upper bounds of respective expressions. Lemma 5 Let Conditions C 1, C 1, C 3 and N C 1 hold. Then, as n → ∞, n 

(n)

 1  (n) β − β 0 −2 ·  w˜ F β 0 ,σ (|ri (β)|) − w˜ F β 0 ,σ (|ri (β 0 )|) n i=1

 

 (n)  − w˜  (F β 0 ,σ (|ei |)) ( f e (|ei |) − f e (−|ei |)) · X i β − β 0 · X i ei  = O p (1).   j=1,2,..., p  ∞ Lemma 6 Let for some p ∈ N , V (n) n=1 , V (n) = vi(n) be a sequence j i=1,2,..., p

of ( p × p) matrixes such that for i = 1, 2, . . . , p and j = 1, 2, . . . , p lim v(n) n→∞ i j

= qi j

in probability

(10)

  j=1,2,..., p where Q = qi j i=1,2,..., p is a fixed nonrandom regular matrix. Moreover, let  (n) ∞ θ be a sequence of p–dimensional random vectors such that n=1 ∃ (ε > 0)

∀ (K > 0)

 lim sup P θ (n)  > K > ε. n→∞

Then ∃ ( δ > 0) ∀ (H > 0) so that  

 lim sup P V (n) · θ (n)  > H > δ. n→∞

For the proof see [15].

38

J. Á. Víšek

Proof of Theorem 2. From the normal Eq. (4) we have for βˆ = βˆ (SW,n,w,ρ) n n

√ 1  ˜ (n) 1  ˜ (n)  0 ˆ ˆ ˆ . w˜ Fβ,σ (|r ( β)|) X e = w ˜ F (|r ( β)|) X X · n β − β √ i i i i i i ˆ ˆ β,σ n i=1 n i=1 (11) Recalling that w˜ is Lipschitz and employing Lemma 1 we arrive at

 n    

(n)

 1   (n) ˆ w˜ F˜β,σ (|ri (β)|) − w˜ F β,σ X i ei  √ sup  ˆ (|ri (β)|) ˆ  n β∈R p  i=1 ≤

n   1 √ (n)  (n)  X i  · |ei | = O p (1) n · Jw · sup sup sup Fβ,σ (r ) − F β,σ (r ) · n i=1 r ∈R + β∈R p σ ∈R +

(for Jw see N C 1). It implies by CLT that the left-hand-side of (11) is O p (1). By a couple of similar approximations, employing Lemmas 2, 3, 4 and 5, we show that for Q n = n1 X  X the right-hand-side of (11) is equal to Q n · (1 + qn ) ·

√ (SW,n,w,ρ) n βˆ − β0

(12)



where qn = o p βˆ (SW,n,w,ρ) − β 0 . As Q n → Q, Q regular matrix, we can utilize Lemma 6 to show that assuming (12) not being O p (1) implies that also left-hand-side

√ (SW,n,w,ρ) ˆ − β 0 is of (11) can’t be O p (1). But it is the contradict. Hence also n β O p (1).

5 Patterns of Results of Simulation Study We have considered βˆ (O L S,n) , βˆ (S,n,ρ) and βˆ (SW,n,w,ρ) . Data were generated by the model Yi = 1 + 2 · X i2 −3 · X i3 + 4 · X i4 −5 · X i5 + ei , i = 1, 2, . . . , n

(13)

(notice please the “true” values of regression coefficients). The i. i. d. framework with explanatory variables X i ’s generated by standard normal d. f., independent from error terms ei ’s which were normally distributed with zero mean and heteroscedastic variances which were uniformly distributed on [0.5, 6]. Algorithms from [5] (employing [4, 13]) and from [14] (which basically coinside with [7]) were used for S- and SW -estimators, respectively (MATLAB codes are available on request). Tukey’s function

Asymptotics of S-Weighted Estimators

ρc (x) =

39

x4 x2 x6 − + for abs(x) ≤ c 2 2 · c2 6 · c4

and

ρc (x) =

c2 otherwise, 6

(see e.g. [4], the constant c is specified at the head of tables) and the quadratic function were utilized as the ρ for S-, and SW -estimators, respectively. The

objective  functions  value b = IE w F |e| (|e|) ρ(e) (remember that σ02 = 1) was for given c computed as ( p is dimension of model, i. e. p = 5) b= p

2 (c2 ) χ p+2

2

− p · ( p + 2)

2 (c2 ) χ p+4

2 · c2

+ p · ( p + 2) · ( p + 4)

2 (c2 ) χ p+6

6 · c4

+

c2 (1 − χ p2 (c2 )), 6

see again [4]. Finally, the weight function w(r ) = 1 for r ∈ [0, h], w(r ) = 0 for r ∈ [g, 0] (h < g) and on [h, g] it has the shape of the “left-wing” of Tukey’s function, decreasing from 1 to 0, see Appendix. Due to rather good PC3 the all constants c, h ˆ MSE, see and g were assigned to minimize an “aggregated” (over coordinates of β) below. We have generated 100 datasets,4 each contained 500 observations and we computed the estimates of regression coefficients 

100 . βˆ (method,k) = (βˆ1(method,k), βˆ2(method,k), βˆ3(method,k), βˆ4(method,k), βˆ5(method,k) ) k=1

The abbreviations O L S, S and SW at the position of “method” inluedicate the method employed for the computation. Finally, we report values (for j = 1, 2, 3, 4 and 5) βˆ (method) = j

1   (method,k) 0 2 1  (method,k)  βˆ j = βˆ and M SE βˆ (method) − βj . j 100 k=1 100 k=1 j (14) 100

100

6 Conclusions The SW -estimator combines plausible properties of S-estimator and of the least weighted squares removing simultaneously a drawback of S-estimator of bounded objective function and the exclusivity of quadratic function utilized by the least weighted squares. Its considerable advantage is the possibility to tailor the estimator - by selecting appropriately the weight and objective functions - to the level and character of contamination of data by something like forward search, see [1]. The 3 HP

Elite 7500 with Intel Core i7-3770 Processor (3.4 GHz, 8 MB cache).

4 We experimented with various numbers of repetitions - smaller than 100 exhibited some instability

in MSE, in the sense that repeated simulations (yielding one particular table - see below) gave (rather) different information about the dispersion of the estimates for individual datasets, - the number of repetitions larger than 100 gave a lower information about the preciseness of estimation by βˆ (method) j (see (14)) just resulting in exact “true values of coefficients”, see (13).

40

J. Á. Víšek original

Table 2 Contamination by outliers: For randomly selected observations we put Yi = 5 ∗ Yi original and data contained also good leverage points X i = 10 ∗ X i Number of observations in each dataset = 500 Contamination level = 1%, h = 0.973, g = 0.989, c = 8.9 (O L S) βˆ(M 0.972(0.063) 1.975(0.022) −2.934(0.027) S E) βˆ (S) 1.010(0.024) 2.003(0.031) −3.021(0.032)

3.912(0.035)

−4.866(0.052)

3.975(0.035)

−4.974(0.023)

−3.012(0.011)

3.986(0.010)

−4.990(0.011)

Contamination level = 2%, h = 0.963, g = 0.978, c = 7.74 (O L S) 0.837(0.115) 1.975(0.011) −2.934(0.013) βˆ(M S E) (S) βˆ 0.993(0.027) 2.014(0.032) −2.973(0.027)

3.927(0.017)

−4.899(0.019)

3.985(0.028)

−4.996(0.023)

−3.000(0.004)

4.000(0.005)

−5.003(0.004)

Contamination level = 3%, h = 0.949, g = 0.963, c = 8.26 (O L S) 0.846(0.158) 1.965(0.009) −2.955(0.008) βˆ(M S E) βˆ (S) 0.979(0.027) 2.017(0.026) −2.958(0.029)

3.927(0.016)

−4.915(0.013)

3.949(0.026)

−4.984(0.032)

−3.002(0.003)

3.997(0.003)

−4.996(0.003)

Contamination level = 5%, h = 0.921, g = 0.942, c = 8.37 (O L S) 0.735(0.253) 1.959(0.004) −2.959(0.005) βˆ(M S E) (S) βˆ 0.985(0.028) 1.948(0.040) −2.967(0.034)

3.940(0.007)

−4.918(0.010)

3.919(0.038)

−4.955(0.030)

−3.006(0.002)

3.998(0.002)

−5.003(0.001)

Contamination level = 10%, h = 0.821, g = 0.863, c = 9.14 (O L S) βˆ(M 0.487(0.641) 1.966(0.003) −2.957(0.004) S E) (S) βˆ 0.922(0.068) 1.871(0.076) −2.792(0.115)

3.943(0.005)

−4.931(0.007)

3.772(0.106)

−4.658(0.213)

−3.005(0.001)

4.002(0.001)

−5.000(0.001)

(M S E) (SW ) βˆ(M S E)

1.002(0.022)

2.001(0.013)

(M S E)

(SW ) βˆ(M S E)

0.992(0.030)

2.008(0.005)

(M S E)

(SW ) βˆ(M S E)

0.999(0.026)

2.001(0.004)

(M S E)

(SW ) βˆ(M S E)

1.014(0.027)

2.002(0.002)

(M S E)

(SW ) βˆ(M S E)

0.989(0.039)

1.999(0.001)

√ paper proves the n-consistency of SW -estimator. The consistency was proved in [20] and the asymptotic representation can be found in [21]. MSE’s of βˆ (SW,n,w,ρ) in Table 2 indicate that it can work rater effectively, especially under significant heteroscedasticity. Numerical experiences indicate that it can be improved by estimating the model of heteroscedasticity, see also [22]. Acknowledgements This paper was written with the support of the Czech Science Foundation project No. P402/12/G097 DYME Dynamic Models in Economics.

Asymptotics of S-Weighted Estimators

41

Appendix See Fig. 2. 1.2

1.2

1

1

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0 −0.2 −0.2

0

0.2

0

g

h 0.4

0.6

0.8

1

1.2

−0.2

g

h 0

0.2

0.4

0.6

0.8

1

Fig. 2 Examples of the weight function of Tukey’s shape for SW -estimator. Experiences from simulations hint that under the serious heteroscedasticity the left-hand side weight function gives better results

References 1. Atkinson, A.C., Riani, M., Cerioli, A.: Exploring Multivariate Data with the Forward Search. Springer, New York (2004) 2. Boˇcek, P., Lachout, P.: Linear programming approach to L M S-estimation. Meml. Vol. Comput. Stat. Data Anal. 19(1995), 129–134 (1993) 3. Breiman, L.: Probability. Addison-Wesley Publishing Company, London (1968) 4. Campbell, N.A., Lopuhaa, H.P., Rousseeuw, P.J.: On calculation of a robust S-estimator of a covariance matrix. Stat. Med. 17, 2685–2695 (1998) 5. Desborges, R., Verardi, V.: A robust instrumental-variable estimator. Stata J. 12, 169–181 (2012) 6. Hampel, F.R., Ronchetti, E.M., Rousseeuw, P.J., Stahel, W.A.: Robust Statistics - The Approach Based on Influence Functions. Wiley, New York (1986) 7. Hawkins, D.M.: The feasible solution algorithm for least trimmed squares regression. Comput. Stat. Data Anal. 17, 185–196 (1994) 8. Hettmansperger, T.P., Sheather, S.J.: A cautionary note on the method of least median of squares. Am. Stat. 46, 79–83 (1992) 9. Portnoy, S.: Tightness of the sequence of empiric c.d.f. processes defined from regression fractiles. In: Franke, J., H˝ardle, W., Martin, D. (eds.) Robust and Nonlinear Time-Series Analysis, pp. 231–246. Springer, New York (1983) 10. Rousseeuw, P.J.: Least median of square regression. J. Am. Stat. Assoc. 79, 871–880 (1984) 11. Rousseeuw, P.J., Leroy, A.M.: Robust Regression and Outlier Detection. Wiley, New York (1987) 12. Rousseeuw, P.J., Yohai, V.: Robust regression by means of S-estimators. In: Franke, J., H˝ardle, W., Martin, R.D. (eds.) Robust and Nonlinear Time Series Analysis. Lecture Notes in Statistics, vol. 26, pp. 256–272. Springer, New York (1984)

42

J. Á. Víšek

13. Verardi, V., McCathie, A.: The S-estimator of multivariate location and scatter in Stata. Stata J. 12, 299–307 (2012) 14. Víšek, J.Á.: Empirical study of estimators of coefficients of linear regression model. Technical report of Institute of Information Theory and Automation, Czechoslovak Academy of Sciences, number 1699 (1990) 15. Víšek, J.Á.: Sensitivity analysis of M-estimates of nonlinear regression model: influence of data subsets. Ann. Inst. Stat. Math. 54, 261–290 (2002) √ 16. Víšek, J.Á.: The least trimmed squares. Part I - consistency. Part II - n-consistency. Part III asymptotic normality and Bahadur representation. Kybernetika 42, 1–36; 181–202; 203–224 (2006) √ 17. Víšek, J.Á.: Weak n-consistency of the least weighted squares under heteroscedasticity. Acta Univ. Carol. Math. Phys. 2/51, 71–82 (2010) 18. Víšek, J.Á.: Empirical distribution function under heteroscedasticity. Statistics 45, 497–508 (2011) 19. Víšek, J.Á.: The least weighted squares with constraints and under heteroscedasticity. Bull. Czech Econ. Soc. 20(31), 21–54 (2013) 20. Víšek, J.Á.: S-weighted estimators. In: Bozeman, J.R., Oliveira, T., Skiadas, C.H. (eds.) Stochastic and Data Analysis Methods and Applications in Statistics and Demography, pp. 437–448 (2015) 21. Víšek, J.Á.: Representation of SW -estimators. In: Skiadas C.H. (ed.) Proceedings 4th Stochastic Modeling Techniques and Data Analysis International Conference with Demographics Workshop, SMTDA 2016, pp 425–438 (2016) 22. Wooldridge, J.M.: Introductory Econometrics. A Modern Approach. MIT Press, Cambridge (2006); 2nd edn. (2009)

Gini’s Delta to Measure Intensity of Multidimensional Performance Silvia Terzi and Luca Moroni

Abstract In the present paper we suggest the use of Alkire-Foster [1, 3] dual cut-off method as a measure of multidimensional performance and a modified version of Gini’s delta to measure the concentration of achievements and/or deprivations. On either tails, the information on the joint distribution of achievements or deprivations can be used to measure their concentration. In particular we suggest the use of Gini’s delta to measure concentration within a same tail - for different second order cut-off values k. We call this measure a measure of performance intensity. As an example we apply it to both tails of an OECD regional well-being performance indicator, and to the Multidimensional Poverty Index (MPI). Keywords Multivariate performance · Multidimensional poverty · Well being

1 Introduction In the present paper we suggest the use of Alkire-Foster [1, 3] dual cut-off method as a measure of multidimensional performance and a modified version of Gini’s delta to measure the concentration of achievements and/or failures (deprivations). The Alkire Foster (AF) methodology has been introduced as a multivariate deprivation measure, and has led to the currently used Multidimensional Poverty Indicator (MPI). However it can be naturally extended to measure achievements and/or performance on both tails of a multidimensional distribution. It consists first of all in defining the different dimensions and deprivation-thresholds on which the multidimensional indicator is to be based, and in counting how many deprivations for each person. The second step consists in defining as multidimensionally poor a person that is deprived in at least k dimensions, and subsequently aggregating across poor people to construct a poverty measure. One key feature of this methodology is it applies also to ordinal-scale S. Terzi (B) · L. Moroni Department of Economics, Università Roma Tre, Rome, Italy e-mail: [email protected] L. Moroni e-mail: [email protected] © Springer Nature Switzerland AG 2019 C. Crocetta (ed.), Theoretical and Applied Statistics, Springer Proceedings in Mathematics & Statistics 274, https://doi.org/10.1007/978-3-030-05420-5_5

43

44

S. Terzi and L. Moroni

based dimensions. A second important feature is that it reflects the joint distribution of deprivations. Our suggestion is to use the information on the joint distribution of deprivations (or vice-versa of achievements) to measure their concentration by means of Gini’s delta, for different cut-off values k. We thus introduce what we call a measure of performance intensity. As an example we apply it to both tails of an OECD regional well-being performance indicator, and to the Multidimensional Poverty Index (MPI).

2 The Multidimensional Performance Indicator P0 Assume we have several (d ) different dimensions of performance (or of poverty, as in the original context) and no natural definition of an aggregate variable. These dimensions could be, and in fact this will often be the case, attributes measured on an ordinal scale, as for example: satisfactory environment, health, education - within a public economics framework; we could define for each dimension a specific reference value or cut-off (as AF call it) and thus determine who lies above and who lies below each of these uni-dimensional references or thresholds. By defining cut-off values for achievements rather than deprivations we can use AF methodology as an overall performance measure. Thus within our suggested framework the first order cut-off values identify success or un-success in each dimension or key performance indicator. The second step consists in setting a second threshold (or cut-off, usually denoted by k) to define as multi-dimensionally successful whatever unit has at least a certain number of accomplishments. In other words the second order cut-off defines how widely successful and well-performing an individual or unit must be to be defined as effective/successful tout court. If we set this second cut-off value equal to d , this leads us to the intersection approach, i.e. the definition of multi-dimensionally effective a unit that achieves its goals in all the key indicators. Vice-versa, setting this cutoff value equal to 1 leads us to the union approach: multidimensional effectiveness as achievement in any one key indicators. Setting 1 < k < d we have intermediate solutions. Let wj (j = 1, . . . , d ) represents the weight that is applied to dimension j and set j wj = d , so that the dimensional weights wj sum to the total number of dimensions d . Let ci (i = 1, . . . , n) represents the weighted number of achievements obtained by person i; select a performance cut-off k, such that 0 < k ≤ d , and define as multi-dimensionally effective anyone whose achievement count is ci ≥ k. Let q be the number of effective units, and let ci (k) be the weighted achievement count only of effective  units. The multidimensional performance indicator P0 can be defined as: P0 = i ci (k)/nd , the average weighted achievements among the population. It can also be expressed as the product of two measures: the (multidimensional) headcount ratio (H ) and the average achievement share among the effective (A). In other words: P0 = HA, where: H = q/n and A = ci (k)/dq. H - the headcount ratio is simply the proportion of well performing units, whereas A is the average fraction of achievements among the well performing (also called average intensity). To compute the concentration of achievements by means of Gini’s δ let k1 and k2 (k2 > k1 ) be

Gini’s Delta to Measure Intensity of Multidimensional Performance

45

two different second order cut off values; let P0 (k1 ), P0 (k2 ), H (k1 ) and H (k2 ) be the performance and the head counts corresponding to the selected second order cut-offs. By definition, δ is such that: 

P0 (k2 ) P0 (k1 )

δ

= 

Thus: δ=

log log



H (k2 ) H (k1 )

H (k2 ) H (k1 ) P0 (k2 ) P0 (k1 )

(1)

 

(2)

and this corresponds to the suggested measure of performance intensity.

3 Multivariate Performance of Regional Well Being: A First Example Application As a first example let us take OECD indicators in the Regional Well-Being Database [4]. These 11 indicators1 relate to: Income; Jobs; Housing; Health; Education; Environment; Safety; Civic engagement; Accessibility of services. We set wj = 1(j = 1, . . . , d ), the first order cut-off level at 20% and k1 = 4, while k2 varies from 6 to 8. With these positions we obtain δ + (6/4) = 1.26, δ + (7/4) = 1.24 and δ + (8/4) = 1.22. On the bad performance tail of the distribution we obtain δ − (6/4) = 1.46, δ − (7/4) = 1.33 and δ − (8/4) = 1.19. As k grows the performance intensity declines (on both tails); however the bad performance intensity on the lower tail is greater than the intensity of good performance. This means that among the well performing regions there are greater performance differences than among the excellently performing. This also holds on the bad performing tail of the distribution, where the decline in performance intensity is greater.

4 A Closer Look at MPI: A Second Example Multidimensional Poverty Index is a multidimensional performance indicator based on three dimensions (Education, Health, Living Standards) and ten indicators.2 Each 1 Household disposable income per capita; Employment rate; Unemployment rate; Number of rooms

per person; Life expectancy at birth; Age adjusted mortality rate; Share of labour force with at least secondary education; Estimated average exposure to air pollution in PM2.5; Homicide rate; Voter turnout; Share of households with broadband access. 2 Years of schooling; School attendance; Child Mortality; Nutrition; Electricity: Drinking Water; Sanitation; Flooring; Cooking Fuel; Assets.

46

S. Terzi and L. Moroni 12

Frequency

10 8 6 4 2 0

1.20

1.44

1.68

1.92

2.16

2.40

2.64

Fig. 1 Histogram of δ

dimension is weighted 1/3; the indicators within the same dimension are equally weighted. As we have already outlined MPI can be decomposed in the product of A and H , where A is the average fraction of deprivations among the poor and H is the head count ratio. A natural question (see Alkire-Santos [2]) is how these two sub indices are related. There appears to be evidence that countries with higher MPI headcounts tend to have higher average intensity (in other words H and A are highly correlated). A second natural question could be to see how delta relates to either of these. From the Human Development Report 2014 [5], we make use of the MPI and H values with k1 equal to the 20% of the indicators considered (corresponding to vulnerability) and with k2 equal to the 33% of the indicators (poverty), to compute δ for each of the 105 countries as:   H (.33) log H (.20)   δ= (3) MPI (.33) log MPI (.20) It varies from 1.13 to 2.62, and Fig. 1 is the histogram of its distribution. The histogram suggests the existence of two groups of countries: one group centered around δ = 1.4, and the other around δ = 2; besides these two groups we have 5 countries with particularly high values. Of course high δ means high concentration of deprivations in the tail; thus the first cluster is made of countries with deprivation intensity ranging from low to medium whereas the second cluster is composed of countries whose deprivation intensity ranges from medium to high. The correlations between δ, A and H are all very high, as can be seen in Table 1.

Gini’s Delta to Measure Intensity of Multidimensional Performance

47

Table 1 Correlations between MPI components A, H and performance intensity δ H (.20) H (.33) A(.20) A(.33) δ H (.20) H (.33) A(.20) A(.33) δ

1 0.980 0.886 0.821 0.875

1 0.938 0.829 0.927

1 0.872 0.995

1 0.964

1

References 1. Alkire, S., Foster, J.: Counting and multidimensional poverty measurement. J. Public Econ. 95(7–8), 476–487 (2009) 2. Alkire, S., Santos, M.E.: Acute multidimensional poverty: a new index for developing countries, OPHI working paper No. 38 (2010) 3. Alkire, S., Foster, J.: Understandings and misunderstandings of multidimensional poverty measurement. J. Econ. Inequal. 9(2), 289–314 (2011) 4. OECD Regional well being: a users guide. www.oecdregionalwellbeing.org 5. UNDP: Human Development Report (2014). http://www.hdr.undp.org/sites/default/files/hdr14report-en-1.pdf

Frequentist and Bayesian Small-Sample Confidence Intervals for Gini’s Gamma Index in a Gaussian Bivariate Copula Valentina Mameli and Alessandra R. Brazzale

Abstract In this paper we consider frequentist and Bayesian likelihood-based smallsample procedures to compute confidence intervals for Gini’s gamma index in the bivariate Gaussian copula model. We furthermore discuss how the method straightforwardly extends to any measure of concordance which is available in closed form, and to any type of copula for which the considered measure of concordance has a closed-form expression. Keywords Equi-correlated bivariate normal model · Gaussian copula Gini’s gamma index · Modified signed likelihood root

1 Introduction Copula functions are useful tools to construct bivariate distributions as well as multivariate ones; see [11]. They are key tools in numerous fields of application as diverse as Biology, Genetics, and medical research to model dependent random variables. In these fields small sample sizes are rather common. However, as it is well known, inference based on the classical first order approximations may produce unreliable results when the sample size is small. Higher order approximations can provide appreciably better inferences, as has been confirmed by the large body of literature and the large quantity of examples, which illustrate the accuracy of these approximations; see, for instance, [2] and reference therein. Our research addresses the construction of small-sample confidence intervals based on the findings of [4] for Gini’s index, and the possible extension to further measures of concordance, in a bivariate Gaussian copula both from the frequentist and the Bayesian point of view. V. Mameli (B) Department of Environmental Sciences, Informatics and Statistics, Ca’ Foscari University of Venice, Venice, Italy e-mail: [email protected] A. R. Brazzale Department of Statistical Sciences, University of Padova, Padova, Italy e-mail: [email protected] © Springer Nature Switzerland AG 2019 C. Crocetta (ed.), Theoretical and Applied Statistics, Springer Proceedings in Mathematics & Statistics 274, https://doi.org/10.1007/978-3-030-05420-5_6

49

50

V. Mameli and A. R. Brazzale

More specifically, let F(x, y) be a joint cumulative distribution function with marginal cumulative distributions F1 (x) and F2 (y). For all real (x, y), the bivariate Gaussian copula C is defined as   C(F1 (x), F2 (y); ρ) = Φ2 Φ −1 (F1 (x)), Φ −1 (F2 (y)); ρ , ρ ∈ (−1, 1), where Φ2 (·, ·; ρ) and Φ(·) are the distribution functions of, respectively, the bivariate standard normal distribution with correlation ρ and of the univariate standard normal distribution. The Gaussian copula reduces to the bivariate standard normal model when F1 (x) = Φ(x) and F2 (y) = Φ(y), that is, when both marginals are standard normal. The dependence between two variables is quantified by a so-called measure of concordance. Note that these measures are completely determined by the copula, that is, they are independent of the marginal distributions F1 (x) and F2 (y) ([10]). In the case of the bivariate Gaussian copula the expression of these measures can be found in closed form and depends on the correlation coefficient ρ ([7]). In particular, we recall the expressions of Gini’s gamma index [5] γ (ρ) =

    1 4 arcsin (1 + ρ)(3 + ρ) − (1 − ρ)(3 − ρ) , π 4

(1)

of Blomqvist’s beta index β(ρ) =

2 arcsin(ρ), π

which for the bivariate Gaussian copula agrees with Kendall’s tau, and of Spearman’s rho index ρ

6 ρS (ρ) = arcsin . π 2 In this paper we focus our attention on the construction of small-sample confidence intervals for Gini’s gamma index (1). Although here interest relies on Gini’s index, our method can be extended to any measure of concordance such as Kendall’s tau, Spearman’s rho, and Blomqvist’s beta. Note that in the case of a bivariate Gaussian copula, these are monotone functions of the parameter ρ, and hence represent an interest-respecting reparametrization of the model; see Fig. 1. The paper is organized as follows. Section 2 reviews likelihood-based smallsample asymptotics. In Sect. 3 we assess numerically the finite-sample properties of the proposed confidence intervals, which are discussed in Sect. 4.

Frequentist and Bayesian Small-Sample Confidence Intervals …

1.0

0.5

0.0

0.5

1.0

Fig. 1 Three measures of concordance as functions of the correlation ρ for the bivariate Gaussian copula: Gini’s gamma (solid), Kendall’s tau (dashed), Sperman’s rho (dotted)

51

1.0

0.5

0.0

0.5

1.0

2 Background Theory Given a parametric statistical model with density function f (y; θ ), let y = (y1 , . . . , yn ) be a vector of n observations and θ = (ψ, λ) a k-dimensional parameter, where ψ is the scalar component of interest and λ is a nuisance parameter of dimension k − 1. Define the likelihood and the log-likelihood functions as, respectively, L(θ ) ∝ f (y; θ ) and l(θ ) = log L(θ ). Confidence intervals for ψ can be derived from the signed likelihood root  ˆ − lp (ψ)), r(ψ) = sign(ψˆ − ψ) 2(lp (ψ) which is asymptotically normal up to the order n−1/2 . Here, lp (ψ) = l(θˆψ ) denotes the profile log-likelihood and θˆψ = (ψ, λˆ ψ ) is the constrained maximum likelihood estimate. The standard normal reliably approximates the finite-sample distribution of r(ψ) for large n, but can be inaccurate if the sample size is small. Improvements can be obtained by using the modified signed likelihood root [1]   q(ψ) 1 log , r (ψ) = r(ψ) + r(ψ) r(ψ) ∗

(2)

whose finite-sample distribution is standard normal up to the order n−3/2 . On the Bayesian side, an expression similar to (2) rB∗ (ψ)

  qB (ψ) 1 log = r(ψ) + r(ψ) r(ψ)

(3)

52

V. Mameli and A. R. Brazzale

provides an approximation to the posterior distribution of ψ given y whose finitesample error is of the order n−3/2 . This approximation can be used to derive credible intervals for ψ. For the expression of the correction terms q in (2) and qB in (3) we refer the reader to [2, 4, 8] and references therein. In this paper, we will use the solution proposed by Fraser and co-authors [4, 9]. Its derivation requires the definition of a so-called canonical parameter

n ∂l(θ ; y)

ϕ (θ ) = l;V (θ ; y0 ) = ∂y 

i

i=1

Vi , y=y0

and involves the directional derivative, l;V (θ ; y0 ), of the log-likelihood function evaluated at the observed data point y0 . In particular, l;V is the derivative of l(θ ) in the directions given by the n rows V1 , . . . , Vn of the n × k matrix V . This matrix can be constructed by using a vector of pivotal quantities z = {z1 (y1 , θ ), . . . , zn (yn , θ )}, through     ∂z −1 ∂z

V =− , ∂y ∂θ 

y0 ,θˆ0

where θˆ0 is the maximum likelihood estimate at y0 . The correction term q results in ϕλ (θˆψ )| |ϕ(θˆ ) − ϕ(θˆψ ) q= |ϕθ (θˆ )|



|j(θˆ )| |jλλ (θˆψ )|

 21 ,

where ϕθ (θ ) = ∂ϕ(θ )/∂θ  represents the Jacobian matrix of ϕ(θ ) with respect to θ , while ϕλ (θ ) = ∂ϕ(θ )/∂λ identifies the k − 1 columns of this matrix which correspond to the nuisance parameter λ. Analogously, jλλ (θ ) represents the (k − 1) × (k − 1) sub-block of the observed information matrix j(θ ) with respect to λ. On the Bayesian side, the correction term qB results in  qB =

ˆ − 21 lp (ψ)jp (ψ)

|jλλ (θˆψ )| |jλλ (θˆ )|

 21

π(θˆ ) , π(θˆψ )

where lp (ψ) = dlp (ψ)/d ψ is the profile score function, jp (ψ) = −d 2 lp (ψ)/d ψ 2 the profile observed information function and π(θ ) denotes a suitable prior distribution for θ .

Frequentist and Bayesian Small-Sample Confidence Intervals …

53

3 Confidence Intervals for Gini’s Gamma Index In the special case of a Gaussian copula model with normal marginals, we can use the confidence intervals discussed in [6] for the correlation coefficient ρ. These apply to the three different parameter combinations of a bivariate normal distribution, that is, when the means and variances are, respectively, known and unknown, and the inbetween situation given by the equi-correlated bivariate normal distribution. Given that Gini’s gamma index (1) represents an interest-respecting reparametrization of the baseline model, and that the first and third order pivots r and r ∗ are invariant under such kind of reparametrization, the results by [6] immediately yield the desired largeand small-sample confidence intervals for γ (ρ). The same holds for the higher order Bayesian solution rB∗ , given that Φ(rB∗ ) represents an approximation of the posterior distribution of the parameter of interest ψ. Last but not least, recalling from Sect. 1 that the measures of concordance mentioned there do not depend on the specification of the two marginal distributions F1 (x) and F2 (y), our method generalizes to any type of Gaussian copula model.

3.1 Simulation Studies In this section we present the findings from a simulation study we designed to assess the coverage probabilities of nominal 95% confidence intervals based upon the smallsample pivots r ∗ and rB∗ , and to compare them with those obtained from the largesample counterpart r. The accuracy of the confidence intervals was evaluated in terms of empirical coverage, and upper and lower error probabilities. Three possible scenarios were considered. Simulation 1: a bivariate normal model with zero means, unit variances and unknown correlation ρ Simulation 2: the equi-correlated bivariate normal model with mean 0.7, variance 0.1 and correlation ρ Simulation 3: a bivariate normal model with means 7, variances 0.9 and correlation ρ. In all three scenarios, the correlation took on values in {−0.9, −0.8, . . . , 0.8, 0.9}. We considered the four sample sizes n = 5, 10, 15, 20, to emphasize small sample sizes. For every scenario, and combination of ρ and n, 10, 000 observations were generated. The Bayesian correction term qB was derived by using matching priors. In particular, in the scalar parameter case (Simulation 1), we considered Jeffrey’s prior, while in Simulation 2 and 3 we considered the matching prior of [12] obtained from an orthogonal parametrization of the models; see [6] for details. All simulations were run using the numerical computing environment R [3]. Figure 2 shows the empirical coverage of the 95% confidence intervals for Simulation 1. Tables 1, 2, 3 report the summary statistics for all three scenarios for sample size n = 5. The omitted results are available from the authors upon request.

V. Mameli and A. R. Brazzale

0.91

0.91

0.93

0.93

0.95

0.95

54

−0.5

0.0

0.5

−0.5

ρ

0.0

0.5

0.91

0.91

0.93

0.93

0.95

0.95

ρ

−0.5

0.0

0.5

ρ

−0.5

0.0

0.5

ρ

Fig. 2 Simulation 1. Empirical coverage of the 95% confidence intervals for γ (ρ) for varying values of ρ and sample sizes n. From top left to bottom right: n = 5, n = 10, n = 15, n = 20. Legend: –◦– 1st order; –– 3rd order; –– Bayes; — nominal

4 Discussion and Generalization The extensive numerical investigation, which we partially reported here, revealed that the higher order frequentist pivot r ∗ is highly accurate, especially for the rather small sample sizes which may be encountered in practice. Tables 1, 2, 3 highlight how the small-sample pivots r ∗ and rB∗ exhibit more reliable coverage than the confidence intervals obtained from their large-sample counterpart r, although the Bayesian solution rB∗ somewhat overestimates the nominal level. Furthermore, the higher order solutions guarantee symmetry on the tails. The differences among the pivots vanish as the sample size increases. Though our simulation study considered standard normal marginals, our method does not depend on the specification of F1 (x) and F2 (y) as the measures of concordance are invariant with respect to these. Furthermore, as mentioned at the beginning of the paper, our method applies to any measure of concordance for the bivariate Gaussian copula, provided it is available in closed form. Last, but not least, the method can be generalized to handle any type of copula for which the measures of concordance have a closed-form expression.

Frequentist and Bayesian Small-Sample Confidence Intervals …

55

Table 1 Summary statistics for Simulation 1: bivariate normal with means 0 and variances 1. Empirical coverage (CP), and upper (U E) and lower (LE) error probabilities of nominal two-sided 95% confidence intervals for γ , for varying values of ρ and sample size n = 5. Pivots used: likelihood root r (1st), modified likelihood root r ∗ (3rd); Bayesian modified likelihood root rB∗ (Bayes). Based on 10,000 replicates; simulation error: ±0.004 (a) ρ Summary 1st 3rd Bayes −0.9

−0.8

−0.7

−0.6

−0.5

−0.4

−0.3

−0.2

−0.1

(b) ρ 0

0.1

CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE

0.927 0.040 0.033 0.930 0.042 0.028 0.936 0.041 0.023 0.936 0.036 0.028 0.9330 0.0376 0.0294 0.925 0.044 0.031 0.923 0.043 0.034 0.929 0.037 0.034 0.926 0.040 0.034

0.954 0.024 0.022 0.944 0.027 0.029 0.946 0.025 0.029 0.944 0.024 0.032 0.950 0.023 0.027 0.946 0.027 0.027 0.944 0.027 0.029 0.952 0.023 0.025 0.949 0.026 0.025

0.953 0.024 0.023 0.941 0.027 0.032 0.937 0.026 0.037 0.942 0.024 0.034 0.945 0.024 0.031 0.941 0.028 0.031 0.937 0.032 0.031 0.940 0.029 0.031 0.935 0.034 0.031

Summary

1st

3rd

Bayes

CP UE LE CP UE LE

0.928 0.036 0.036 0.920 0.040 0.040

0.952 0.024 0.024 0.948 0.028 0.024

0.941 0.029 0.030 0.935 0.035 0.030 (continued)

56

V. Mameli and A. R. Brazzale

Table 1 (continued) (a) ρ Summary 0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE

1st

3rd

Bayes

0.924 0.037 0.039 0.926 0.032 0.042 0.926 0.033 0.041 0.926 0.033 0.041 0.935 0.026 0.039 0.940 0.021 0.039 0.933 0.026 0.041 0.922 0.037 0.041

0.945 0.029 0.026 0.948 0.025 0.027 0.945 0.029 0.026 0.942 0.032 0.026 0.947 0.029 0.024 0.950 0.028 0.022 0.948 0.027 0.025 0.952 0.023 0.025

0.935 0.032 0.033 0.939 0.029 0.032 0.944 0.030 0.026 0.938 0.036 0.026 0.942 0.033 0.026 0.942 0.037 0.021 0.943 0.032 0.025 0.950 0.025 0.025

Table 2 Summary statistics for Simulation 2: equi-correlated bivariate normal model with mean 0.7 and variance 0.1. Empirical coverage (CP), and upper (U E) and lower (LE) error probabilities of nominal two-sided 95% confidence intervals for γ , for varying values of ρ and sample size n = 5. Pivots used: likelihood root r (1st), modified likelihood root r ∗ (3rd); Bayesian modified likelihood root rB∗ (Bayes). Based on 10,000 replicates; simulation error: ±0.004 (a) ρ −0.9

−0.8

Summary

1st

3rd

Bayes

CP UE LE CP UE LE

0.914 0.066 0.019 0.914 0.064 0.022

0.949 0.028 0.023 0.950 0.026 0.024

0.970 0.016 0.014 0.972 0.013 0.014 (continued)

Frequentist and Bayesian Small-Sample Confidence Intervals … Table 2 (continued) (a) ρ Summary −0.7

−0.6

−0.5

−0.4

−0.3

−0.2

−0.1

(b) ρ 0

0.1

0.2

0.3

0.4

57

1st

3rd

Bayes

CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE

0.909 0.069 0.022 0.911 0.070 0.019 0.913 0.065 0.022 0.907 0.072 0.022 0.912 0.068 0.020 0.913 0.067 0.020 0.908 0.070 0.022

0.950 0.026 0.024 0.950 0.030 0.021 0.950 0.025 0.025 0.944 0.031 0.025 0.950 0.028 0.022 0.951 0.028 0.022 0.948 0.028 0.023

0.972 0.013 0.015 0.972 0.015 0.013 0.972 0.014 0.014 0.969 0.018 0.013 0.974 0.014 0.012 0.974 0.014 0.012 0.971 0.014 0.014

Summary

1st

3rd

Bayes

CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE

0.913 0.066 0.021 0.910 0.069 0.022 0.910 0.067 0.023 0.912 0.067 0.020 0.912 0.070 0.019

0.949 0.027 0.024 0.945 0.031 0.025 0.947 0.027 0.026 0.951 0.026 0.023 0.948 0.031 0.021

0.972 0.014 0.014 0.970 0.015 0.016 0.972 0.013 0.015 0.974 0.013 0.013 0.973 0.016 0.011 (continued)

58

V. Mameli and A. R. Brazzale

Table 2 (continued) (a) ρ Summary 0.5

0.6

0.7

0.8

0.9

CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE

1st

3rd

Bayes

0.905 0.072 0.023 0.911 0.069 0.020 0.912 0.068 0.020 0.907 0.073 0.020 0.916 0.066 0.018

0.943 0.030 0.027 0.948 0.028 0.024 0.947 0.030 0.023 0.946 0.031 0.023 0.951 0.028 0.021

0.970 0.015 0.015 0.973 0.014 0.013 0.973 0.014 0.013 0.971 0.015 0.014 0.974 0.013 0.012

Table 3 Summary statistics for Simulation 3: complete bivariate normal model with means 7 and variances 0.9. Empirical coverage (CP), and upper (U E) and lower (LE) error probabilities of nominal two-sided 95% confidence intervals for γ , for varying values of ρ and sample size n = 5. Pivots used: likelihood root r (1st), modified likelihood root r ∗ (3rd); Bayesian modified likelihood root rB∗ (Bayes). Based on 10,000 replicates; simulation error: ±0.004 (a) ρ −0.9

−0.8

−0.7

−0.6

−0.5

Summary

1st

3rd

Bayes

CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE

0.855 0.106 0.040 0.849 0.101 0.050 0.844 0.102 0.054 0.849 0.096 0.054 0.845 0.094 0.061

0.943 0.033 0.024 0.940 0.032 0.028 0.939 0.033 0.029 0.939 0.033 0.028 0.942 0.030 0.028

0.972 0.015 0.014 0.968 0.016 0.016 0.967 0.016 0.017 0.971 0.016 0.013 0.971 0.014 0.014 (continued)

Frequentist and Bayesian Small-Sample Confidence Intervals … Table 3 (continued) (a) ρ Summary −0.4

−0.3

−0.2

−0.1

(b) ρ 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

59

1st

3rd

Bayes

CP UE LE CP UE LE CP UE LE CP UE LE

0.838 0.097 0.065 0.842 0.090 0.068 0.842 0.086 0.072 0.837 0.086 0.078

0.935 0.034 0.031 0.938 0.033 0.029 0.940 0.031 0.029 0.937 0.033 0.030

0.965 0.017 0.017 0.969 0.017 0.014 0.970 0.015 0.015 0.967 0.017 0.016

Summary

1st

3rd

Bayes

CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE CP UE LE

0.846 0.076 0.078 0.839 0.078 0.083 0.839 0.072 0.090 0.837 0.069 0.094 0.848 0.065 0.087 0.843 0.063 0.094 0.847 0.053 0.100 0.851 0.048 0.101

0.940 0.029 0.032 0.936 0.031 0.033 0.938 0.030 0.032 0.936 0.031 0.033 0.938 0.031 0.030 0.935 0.031 0.034 0.939 0.029 0.032 0.941 0.026 0.032

0.970 0.014 0.016 0.969 0.014 0.017 0.970 0.015 0.015 0.969 0.014 0.017 0.968 0.018 0.014 0.966 0.016 0.018 0.970 0.015 0.015 0.967 0.014 0.018 (continued)

60

V. Mameli and A. R. Brazzale

Table 3 (continued) (a) ρ Summary 0.8

0.9

CP UE LE CP UE LE

1st

3rd

Bayes

0.848 0.046 0.106 0.853 0.037 0.110

0.939 0.029 0.032 0.945 0.022 0.032

0.970 0.016 0.014 0.970 0.013 0.016

References 1. Barndorff-Nielsen, O.: On a formula for the distribution of the maximum likelihood estimator. Biometrika 70, 343–365 (1983) 2. Brazzale, A.R., Davison, A.C., Reid, N.: Applied Asymptotics: Case Studies in Small-Sample Statistics. Cambridge University Press, Cambridge (2007) 3. R Core Team: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. Vienna, Austria (2013) 4. Fraser, D.A.S., Reid, N., Wu, J.: A simple general formula for tail probabilities for frequenstist and Bayesian inference. Biometrika 86, 249–264 (1999) 5. Gini, C.: L’ammontare e la composizione della ricchezza delle nazioni. Fratelli Bocca, Torino (1914) 6. Mameli, V., Brazzale, A.R.: Modern likelihood inference for the maximum/minimum of a bivariate normal vector. J Stat Comput Simul 86, 1869–1890 (2016) 7. Meyer, C.: The bivariate normal copula. Commun Stat-Theory Methods 42, 2402–2422 (2013) 8. Reid, N.: Asymptotics and the theory of inference. Ann Stat 31, 1695–1731 (2003) 9. Reid, N., Fraser, D.A.S.: Mean log-likelihood and higher-order approximations. Biometrika 97, 159–170 (2010) 10. Schmid, F., Schmidt, R., Blumentritt, T., Gaisser, S., Ruppert, M.: Copula-based measure of multivariate association. In: Jaworski, P., et al. (eds.) Copula Theory and Its Applications. Lecture Notes in Statistics, vol. 198, pp. 209–236. Springer, Berlin (2010) 11. Sklar, A.: Fonctions de répartition à n dimensions et leurs marges. Publications de l’Institut de Statistique de l’Université de Paris 8, 229–231 (1959) 12. Staicu, A.M., Reid, N.: On probability matching priors. Can J Stat 36, 613–622 (2008)

Bayesian Estimation of Gini-Simpson’s Index Under Mainland-Island Community Structure Annalisa Cerquetti

Abstract The mainland-island community structure is an ecological transposition of a popular model in population genetics in which a fixed number of subpopulations (islands) are connected, through differing immigration rates, to a single metapopulation (mainland) where diversity is generated through speciation. It has been recently shown that a large class of neutral models with this particular structure converges in the large population limit to the Hierarchical Dirichlet process. This finding provides the analogous, in the multipopulation setting, of the Ewens sampling formula for the single population neutral hypothesis. Here we apply some recent results for conditional moments of diversity measures under Gibbs-type priors to derive a Bayesian nonparametric estimator of Gini-Simpson’s index under the Hubbell Unified Neutral Theory of Biodiversity and Biogeography. Potential applications are also illustrated. Keywords Bayesian estimator · Gini Simpson index · Diversity

1 Introduction First introduced in Gini’s [12] monograph Variabilità e Mutabilità as a descriptive measure of heterogeneity for categorical variables, Gini-Simpson’s index H2 (P) = 1 −



Pi2

(1)

i

for P = (Pi )i≥1 a population of relative abundances, has been rediscovered by Simpson in [24] as an intrinsic characteristic of an infinite population of a finite number of groups corresponding to the probability that two individuals chosen at random and independently from the population will be found to belong to different groups. Since then, together with Shannon-Wiener index [23], (1) is one of the most popular A. Cerquetti (B) MEMOTEF, Sapienza Università di Roma, Via del Castro Laurenziano, 9, 00161 Roma, Italy e-mail: [email protected] © Springer Nature Switzerland AG 2019 C. Crocetta (ed.), Theoretical and Applied Statistics, Springer Proceedings in Mathematics & Statistics 274, https://doi.org/10.1007/978-3-030-05420-5_7

61

62

A. Cerquetti

measure of diversity, and its estimation is still one of the main concerns in many applied settings, like e.g. population genetics, community ecology and evolutionary biology. It is well-known that both Gini-Simpson and Shannon index belong to the family of Tsallis entropies [21, 26]    1 m Pi , Hm (P) = 1− m−1 i

(2)

for m > 0 a parameter specifying the sensitivity to common and rare species. For m < 1 the index reduces relative differences between abundant and rare species, while for m > 1 exacerbates such differences, disproportionately favoring the most common species. From (2) Simpson’s index is easily recovered for m = 2, while  Shannon entropy H1 (P) = − i Pi log Pi , the unique index that weighs all species exactly by their frequencies, is obtained for m → 1, by standard application of Hôpital rule. Recently in [2], the full sequence of conditional and unconditional moments of Hm (P) has been derived under general Gibbs-type priors [13], a tractable generalization of the Dirichlet process prior, extremely popular in Bayesian nonparametric approach to species sampling problems, but mostly investigated, until now, with respect to species richness and discovery probability estimation (see e.g. [1, 9, 10]. Here we briefly sketch the general setting.  Let (n 1 , . . . , n k ), for n j > 0 and j n j = n, be the vector of the multiplicities of k(n) different species observed in a sample of size n from a population of a possible countable infinite unknown species abundances. Under exchangeability, by de Finetti representation theory, there exists an infinite random discrete distribution P = (Pi )i≥1 , such that the law of the infinite exchangeable random partition n = {A1 , . . . , Ak } of [n], for n ≥ 1, induced by sampling from P is given by p(n 1 , . . . , n k ) =



⎡ E⎣

(i 1 ,...,i k )

k 

⎤ Pi j j ⎦ , n

(3)

j=1

where n j = |A j |, (i 1 , . . . , i k ) ranges over all ordered k-tuples of distinct positive inte↓ gers and (Pi )i≥1 is any rearrangements of the ranked atoms (Pi )i≥1 of P. Gibbs-type priors are the largest class of infinite random discrete distributions with corresponding exchangeable partition probability function (EPPF) (3) in the Gibbs product form pα,V (n 1 , . . . , n k ) = Vn,k

k 

(1 − α)n j −1 ,

(4)

j=1

for α ∈ (−∞, 1) and V = (Vn,k ) weights satisfying the backward recursive relation Vn,k = (n − kα)Vn+1,k + Vn+1,k+1 , where V1,1 = 1 and (x) y = (x)(x + 1) · · · (x + the multipliciy − 1) is the usual notation for rising factorials. Given (n 1 , . . . , n k ) ties of the first k species observed in a random sample of size n = i n i , then the

Bayesian Estimation of Gini-Simpson’s Index Under …

63

probabilities to observe the jth old species or a new yet unobserved species at step n + 1 follow from (4) and correspond respectively to pα,V (n +j ) = Vn+1,k (n j − α)/Vn,k and pα,V (n k+1 ) = Vn+1,k+1 /Vn,k . Additionally, by Theorem 12 in Gnedin and Pitman [13] each random partition belonging to the class (4) is a probability mixture of extreme partitions, namely: finite symmetric Dirichlet partitions for α < 0, Ewens (θ ) partitions [8] for α = 0, and Poisson-Kingman conditional partitions driven by the stable subordinator [22] for α ∈ (0, 1).

2 Moments of Gini-Simpson’s Diversity Under Gibbs-Type Priors Specializing the results in Cerquetti [2] we can state the following results for the prior and posterior moments of Gini-Simpson’s diversity index for a population of random abundances (Pi )i≥1 following a general Gibbs-type species sampling model of parameters (α, V ). Proposition 1 Given a random discrete distribution P = (Pi )i≥1 , distributed according to a (α, V ) Gnedin-Pitman model with EPPF (4), then first and second moments of H2 (P) are given respectively by E α,V (H2 (P)) = [1 − V2,1 (1 − α)],

(5)

and  E α,V [(H2 (P))2 ] = 1 + V4,1 (1 − α)3

+ V4,2 ((1 − α))2 − 2V2,1 (1 − α) .

(6)

The result is readily applicable under any prior in the Gibbs-type class for which explicit Vn,k weights are known. See e.g. Cerquetti [2] for some examples and explicit formulas. The posterior counterparts of (5) and (6) also arise as particular cases of the general results for Tsallis diversity established in Cerquetti [2]. Proposition 2 Let n = (n 1 , . . . , n k ) be the multiplicities of the first k species observed in a sample of size n drawn from a random population of abundances (Pi )i≥1 following a general (α, V ) Gnedin-Pitman model, then, conditionally given (n 1 , . . . , n k ), first and second moments of H2 (P) are as follows ⎡

⎤ k  V V n+2,k n+2,k+1 (n j − α)2 − (1 − α)⎦ , E V,α (H2 (P)|n) = ⎣1 − Vn,k j=1 Vn,k

(7)

64

A. Cerquetti

and ⎡

⎛ ⎞   V n+4,k ⎝ (n j − α)4 + 2 E V,α [(H2 (P)|n)2 ] = ⎣1 + (n j − α)2 (n i − α)2 ⎠ Vn,k j i= j Vn+4,k+2 [(1 − α)]2 Vn,k  Vn+4,k+1 + [(1 − α)3 + 2(1 − α) (n j − α)2 ] Vn,k j ⎞⎤ ⎛ k  Vn+2,k Vn+2,k+1 (n j − α)2 − (1 − α)⎠⎦ . − 2⎝ Vn,k j=1 Vn,k +

(8)

3 The Neutral Theory of Biodiversity in Community Ecology The neutral hypothesis originates in mathematical population genetics [18] as opposite to selective Darwinian theory. The theory asserts that a large fraction of observed genetic variation between and within populations is nonselective, but occurs purely by chance. In a community ecology transposition neutral models assume ecological equivalence among species and provide null models for community assembly. Those models combine stochastic population dynamics with the assumption that species are equivalent forms for all per capita demographic rates, like e.g. birth and death. As a consequence the abundances within a neutral population fluctuate and diversity arises as a balance between the immigration of new species and local extinction. In Hubbell [15] this theory is also extended to multiple sites using a mainland-island structure. Local communities governed by neutral dynamics are connected through migration to a neutral metacommunity with diversity generated through speciation. In mathematical population genetics, the data being expressed as the gene frequencies at various loci in a population, an exact sampling theory of selectively neutral alleles has been established in a celebrated paper by Ewens [8]. Ewens sampling formula (ESF) under the Wright-Fisher infinitely many alleles model with mutation parameter θ > 0 gives the probability, under neutrality, of the random allocation of a sample of n genes in k different allelic types at a single loci with allelic frequencies (n 1 , . . . , n k ) k  θ k−1 pθ (n 1 , . . . , n k ) = (n j − 1)! (9) (θ + 1)n−1 j=1 This formula is well-known to correspond to the EPPF induced by the PoissonDirichlet random discrete distribution, the law of the ranked random atoms of the θ k−1 Dirichlet process [11, 19], hence to be a particular case of (4) for Vn,k = (θ+1) and n−1

Bayesian Estimation of Gini-Simpson’s Index Under …

65

α → 0. It follows that specializing (5) and (6), expected first and second moment of Gini-Simpson’s index under Ewens neutral model are given by E θ (H2 (P)) =

θ , θ +1

(10)

2θ 2 + 9θ + 6 . (θ + 1)3

(11)

and E θ [(H2 (P))2 ] = 1 −

Additionally, from (7) and (8), conditional first and second moments of H2 (P) are as follows:    θ j (n j )2 − E θ (H2 (P)|n) = 1 − (12) (θ + n)2 (θ + n)2 and 



E θ [(H2 (P)) |n] = 1 + 2

 −2

j (n j )4

j (n j )2

+2

−θ

(θ + n)2



i= j (n j )2 (n i )2

+ θ 2 + θ (6 + 2



j (n j )2

(θ + n)4

 .

(13)

3.1 The Mainland Island Community Structure In a mainland-island community structure [20] a finite number of spatially separated local communities of species abundances are sufficiently far apart for one another to have a negligible contribution to immigration into another plot. Nevertheless the islands do depend on one another because they share a common source (mainland, the metacommunity) pool of immigrants, which is the only source of new species. Under the neutral assumption the mainland actually resembles the infinite alleles models [8] meaning that at the equilibrium the species abundances obeys the Ewens sampling formula with fundamental biodiversity or speciation parameter θ . Additionally immigration from the metacommunity provides new species to each local community (island) at a certain different rate Ii , usually called in this setting the fundamental dispersal or immigration parameter. Locally the single communities under the neutral hypothesis obeys the Ewens sampling formula with parameter Ii . It has been recently shown [14] that in the large size limit this community structure converges to the Hierarchical Dirichlet process (HDP) structure [25]. This model generalizes the partition structure induced by the Dirichlet process, allowing a two-level hierarchy in which, conditionally to a species sample from the top-level Dirichlet process,

66

A. Cerquetti

the child Dirichlet process samples from the very same set of atoms with different probabilities. A sequential urn construction of a sample from HDP structure will be as follows (see e.g. [28]). The stock urn at the top level (mainland) contains balls of colors (species) that are represented by at least one ball in one or multiple urns (islands) at the bottom level. Now, focusing on a single local population, suppose that upon drawing the n i + 1th ball for urn i at the bottom, the stock urn contains A balls of k distinct colors indexed by an integer set {1, . . . , k}. Now we either draw a ball randomly from urn i and place back two balls of that color in it, or with some immigration probability Ii /Ii + n i we return to the top level. From the stock urn we can either draw a ball randomly and put back two balls of that color in the stock urn and one in i or obtain a ball of a new color k + 1 with probability θ/(θ + A) and put back a ball of this color in both the stock urn and urn i of the lower level. The EPPF for a single local population under HDP neutral mainland-island structure has been introduced in Etienne [4] and further studied in Etienne [5, 6] and Etienne and Olff [7]. For (a1 , . . . , ak ) the abundances of species 1, . . . , k in the stock of the k different  species in a single subpopulation, urn and (n 1 , . . . , n k ) individuals  such that a j ≤ n j and j a j = A with A ≤ n = j n j then n  θ k−1 pθ,I (n 1 , . . . , n k ) = (I + 1)n−1 A=k k 

  (a1 ,...,ak : j a j =A)

Sn−1,0 (a j − 1)! j ,a j

j=1

I A−1 , (θ + 1) A−1

(14)

Stirling numbers of the first kind. for Sn−1,0 j ,a j

3.2 Moments of Gini-Simpson’s Diversity Under Mainland-Island Neutral Model By the theory of exchangeable partitions (see e.g. [22]), for Sm = ξ  1 E[(Sm ) ] = j! j=1 ξ

  (ξ1 ,...,ξ j : j ξ j =ξ

 j

P jm then

ξ! p(mξ1 , . . . , mξ j ). ξ1 ! · · · ξ j !

(15)

Therefore to obtain first and second prior moments of H2 (P), for P = (P j ) j≥1 for a single local population under mainland-island structure, we just need few values of (14), namely p(2), p(4) and p(2, 2). Notice that starting with both a single ball in the local urn and in the stock urn (of the same species), the one-step prediction rules are as follows

Bayesian Estimation of Gini-Simpson’s Index Under … old,l pθ,I (1) = pθ,I (2 ∩ local) =

1 , I +1

67

old,s pθ,I (1) = pθ,I (2 ∩ stock) =

I 1 I +1θ +1

(16) and new (1) = pθ,I (1, 1) = pθ,I

θ I I +1θ +1

(17)

where we call p(2 ∩ old) the probability to observe the same species drawn from the local urn and p(2 ∩ stock) the probability to observe the same species by immigration from the stock urn. In the following propositions we obtain our main results. Proposition 3 Under the mainland-island community structure with fundamental diversity parameter θ and immigration parameter I , first and second moments of Gini-Simpson’s diversity index in a single local population correspond to E θ,I (H2 (P)) =

  1 I 1 θI = 1− − , (θ + 1)(I + 1) I +1 I +1θ +1

(18)

and   3! 1 3!I 3 11I 12I 2 + E θ,I [(H2 (P)) ] = 1 + + + (I + 1)3 1 (θ + 1) (θ + 1)2 (θ + 1)3   2 3 I I θ I +2 + + (I + 1)3 (θ + 1) (θ + 1)2 (θ + 1)3   1 1 I , (19) −2 + I +1 I +1θ +1 2

Proof By (14) and (15) (18) follows as 1 − p(2), while (19) as 1 + p(4) + p(2, 2) − 2 p(2). Proposition 4 Let n = (n 1 . . . , n k ) be the multiplicities of the first k species observed in a local sample of size n, under the mainland-island (θ, I ) neutral model on the unknown relative abundances P = (P j ) j≥1 , then, conditionally given n, first moment of H2 (P) is as follows k E θ [H2 (P)|n] = 1 −

j=1 (n j )2

(I + n)2 n  1 I − (I + n)2 A=k (θ + A) −

n  1 I (I + n)2 A=k (θ + A)



k   n jaj

 (a1 ,...,ak : j a j =A) j=1



k   (n j + 1)a j

 (a1 ,...,ak : j a j =A) j=1

68

A. Cerquetti

−

n  1 I2 (I + n)2 A=k (θ + A)2



k  

 (a1 ,...,ak : j a j =A) j=1

(a j )2



n  Iθ 1 − (I + n)2 A=k (θ + A)

−

n 1 I 2θ  (I + n)2 A=k (θ + A)2

(20)

Proof The rigorous proof is extremely long. We sketch here the technique. The result arises telescoping the one-step prediction rules (16) and (17) and accounting: (a) for the four alternatives which may produce two new observations of the same sample old species j (local-local, local-immigrant, immigrant-local, immigrant-immigrant), and (b) for the two alternatives that may produce two new observations of a single species yet unobserved in the sample (immigrant-local, immigrant-immigrant). Remark 1 The posterior second moment of H2 (P) for a single local population under mainland-island community structure can be obtained by the same technique used for the posterior expectation of Proposition 4. Nevertheless, for brevity, we omit here this result since the full expression is extremely long and the derivation extremely cumbersome.

4 Discussion Despite its mathematical complexity and besides providing a Bayesian nonparametric point estimation of Gini-Simpson’s index under Hubbell’s neutral model of biodiversity, (20) may be relevant in some additional perspectives. We give here some preliminary insights. A fundamental role of neutral models in biodiversity is to serve as null models to check whether neutrality may be rejected, before to embark in constructing more complex nonneutral models. The first objective tests of selective neutrality were put forward by Ewens [8] and Watterson [27]. While Ewens’ testing procedure relies on Shannon information, Watterson chose observed Gini-Simpson’s index. Testing the neutral hypothesis under the mainland-island Hubbell’s structure is currently a central topic in community ecology. The typical approach is to first fit neutral parameters θ and I by maximum likelihood and then to perform Monte Carlo significance tests with test statistics Etienne’s EPPF ([5, 14] or Gini-Simpson’s and Shannon index [3, 16]. In a Bayesian frequentist hybrid approach (20) could provide an alternative statistic for testing neutrality under Hubbell’s biodiversity model as follows. Consider two local subpopulations with immigration parameters I1 and I2 , then, under HDP (I1 , I2 , θ ) neutral model, expected prior alpha (within) Gini-Simpson’s diversity corresponds to

Bayesian Estimation of Gini-Simpson’s Index Under …

E θ,I1 ,I2 [H2α (P)] =

69

θ I1 (I1 + 1) + I2 (I2 + 1) , θ + 1 (I1 + 1)2 + (I2 + 1)2

(21)

while expected prior gamma (total) Gini-Simpson’s diversity is given by γ E θ,I1 ,I2 [H2 (P)]

θ = θ +1



 I1 + I2 + 1 . I1 + I2 + 2

(22)

Then, according to Jost [17] decomposition, the expected beta (between) prior diversity under Hubbell’s neutral model with two islands populations follows by substituting (21) and (22) in β

H2 (P) =

γ

H2 (P) − H2α (P) . 1 − H2α (P)

(23)

Now, assume two samples (n 1,1 , . . . , n 1,k1 ) and (n 2,1 , . . . , n 2,k2 ) have been observed respectively from the two subpopulations, then, relying on (20) and on the expected posterior Gini-Simpson gamma diversity, the corresponding expected posterior GiniSimpson beta diversity under neutrality may be obtained by applying Jost’s decomposition with updated weights. An hybrid Bayesian-frequentist testing procedure for the mainland-island neutral hypothesis may be performed by a Monte Carlo significance test based on expected posterior beta diversity under HDP (I1 , I2 , θ ) in place of the hierarchical EPPF (14) used in the Etienne’s original approach.

References 1. Cerquetti, A.: Marginals of multivariate Gibbs distributions with applications in Bayesian species sampling. Electr. J. Stat. 169, 321–354 (2013) 2. Cerquetti, A.: Bayesian nonparametric estimation of Tsallis diversity indices under GnedinPitman priors (2014). arXiv:1404.3441v2 [math.ST] 3. Chao, A., Jost, L., Hsieh, T.C., Ma, K.H., Sherwin, W.B., Rollins, L.A.: Expected Shannon entropy and Shannon differentiation between subpopulations for neutral genes under the finite island model. Plos One 10, 1–2 (2015) 4. Etienne, R.S.: A new sampling formula for neutral biodiversity. Ecol. Lett. 7, 321–354 (2005) 5. Etienne, R.S.: A neutral sampling formula for multiple samples and an ’exact’ test of neutrality. Ecol. Lett. 10, 608–618 (2007) 6. Etienne, R.S.: Maximum likelihood estimation of neutral model parameters for multiple samples with different degrees of dispersal limitation. J. Theor. Biol. 257, 510–514 (2009) 7. Etienne, R.S., Olff, H.: A novel genealogical approach to neutral biodiversity. Ecol. Lett. 8, 253–260 (2004) 8. Ewens, W.J.: The sampling theory of selectively neutral alleles. Theoret. Popul. Biol. 3, 87–112 (1972) 9. Favaro, S., Lijoi, A., Mena, R.H., Prünster, I.: Bayesian non-parametric inference for species variety with a two-parameter Poisson-Dirichlet process prior. J. R. Stat. Soc. B. 71, 993–1008 (2009) 10. Favaro, S., Lijoi, A., Prünster, I.: Conditional formulae for Gibbs-type exchangeable random partitions. Ann. Appl. Probab. 23, 1721–1754 (2013)

70

A. Cerquetti

11. Ferguson, T.S.: A Bayesian analysis of some nonparametric problems. Ann. Stat. 1, 209–230 (1973) 12. Gini, C.W.: Variabilita e mutabilita. Studi Economico-Giuridici della R. Università di. Cagliari 3, 3–159 (1912) 13. Gnedin, A., Pitman, J.: Exchangeable Gibbs partitions and Stirling triangles. J. Math. Sci. 138(3), 5674–5685 (2006) 14. Harris, K., Parsons, T. L., Ijaz, U. Z., Lahti, L., Holmes, I., Quince, C. Linking statistical and ecological theory: Hubbell’s unified neutral theory of biodiversity as a hierarchical Dirichlet process. Proceedings of the IEEE, Issue: 99(2015) 15. Hubbell, S.P.: The Unified Neutral Theory of Biodiversity and Biogeography. Princeton University Press, Princeton (2001) 16. Jabot, F., Chave, J.: Analyzing tropical forest tree species abundance distributions using a nonneutral model and through approximate Bayesian inference. Am. Nat. 178(2), 38–47 (2011) 17. Jost, L.: Partitioning diversity into independent alpha and beta components. Ecology 88, 2427– 2439 (2007) 18. Kimura, M.: Evolutionary rate at the molecular level. Nature 217, 624–626 (1968) 19. Kingman, J.F.C.: Random discrete distributions. J. R. Stat. Soc. B 37, 1–22 (1975) 20. MacArthur, R.H., Wilson, E.O.: The Theory of Island Biogeography. Princeton University Press, Princeton (1967) 21. Patil, G.P., Taille, C.: Diversity as a concept and its measurement. J. Am. Stat. Assoc. 77, 548–561 (1982) 22. Pitman, J.: Poisson-Kingman partitions. In: Goldstein, D.R. (ed.) Science and Statistics: A Festschrift for Terry Speed. LN Monograph Series, vol. 40, pp. 1–34. IMS, Hayward (2003) 23. Shannon, C.E.: A mathematical theory of communication. Bell Syst. Tech. J. 27, 379–423 (1948) 24. Simpson, E.H.: Measurement of diversity. Nature 163, 688 (1949) 25. Teh, Y.W., Jordan, M.I., Beal, M.J., Blei, D.M.: Hierarchical Dirichlet processes. J. Am. Stat. Assoc. 101, 1566–1581 (2006) 26. Tsallis, C.: Possible generalization of Boltzmann-Gibbs statistics. J. Stat. Phys. 52, 479–487 (1988) 27. Watterson, G.A.: Heterosis or neutrality? Genetics 88, 405–417 (1977) 28. Xing, E.P., Sohn, K., Jordan, M.I., Teh, Y.W.: Bayesian Multi-Population Haplotype Inference via a Hierarchical Dirichlet Process Mixture. Proceedings of the 23rd International Conference on Machine Learning (2006)

Spatial Residential Patterns of Selected Foreign Groups. A Study in Four Italian Cities Federico Benassi and Fabio Lipizzi

Abstract What are the spatial residential patterns of the main foreign groups residing in some large Italian cities? Using data from the last Italian demographic census (2011) at sub-municipality level, the study investigates on this research question. A spatial approach is applied to analyze the geographical distribution of the main foreign groups enumerated in the cities of Milan, Rome, Naples and Palermo. The results provide some interesting insights: the distribution of foreign groups coming from central and eastern European countries is quite scattered and shows a comparative low level of dissimilarity to the spatial distribution of Italians. Conversely, foreign groups coming from more distant countries (like China, Bangladesh and Sri Lanka) show spatial distributions characterized by a comparative low level of dispersion and a comparative high level of dissimilarity to the spatial distribution of Italians. Keywords Spatial statistics · Spatial residential patterns · Foreign groups · Foreigners

1 Introduction Foreign immigration in Italy is not a new phenomenon. The first significant immigration wave, in fact, dates back to the late 1970s. New strong waves followed during the next decades [1]. However, the recent unexpected surge of foreign population residing in Italy was incredibly high. Foreigners passed in fact from approximately 1.3 million people by the end of 2001 to over 4 million people by the end of 2011, equal to 7% of the entire population. In this framework, the spatial distribution of foreign population becomes an important research topic, especially when it is referred to urban contexts where, due to several different factors, specific phenomena like F. Benassi (B) · F. Lipizzi Italian National Institute of Statistics, Piazza Guglielmo Marconi, 26/C, 00144 Rome, Italy e-mail: [email protected] F. Lipizzi e-mail: [email protected] © Springer Nature Switzerland AG 2019 C. Crocetta (ed.), Theoretical and Applied Statistics, Springer Proceedings in Mathematics & Statistics 274, https://doi.org/10.1007/978-3-030-05420-5_8

71

72

F. Benassi and F. Lipizzi

residential segregation are more frequent [2]. The spatial distribution of foreign population is a topic that scholars have been studying widely in many social sciences, relating in particular to the US and UK situation. Several approaches have been followed and a wide range of measures and indices have been developed concerning different geographical contexts and several geographical scales of analysis: see for instance [3–5]. Since the 1980s, the academic interest in the study of settlement patterns of foreign population grew in Italy as well, with an intensification in recent years [6–9]. The present study thus, contributes to the national and international literature by detecting the spatial residential patterns of the most numerous foreign groups in four of the largest Italian cities: Milan, Rome, Naples and Palermo. We will use synthetic spatial measures that will allow us to appreciate specific aspects of each observed distribution. The paper is organized as follows: the data used and the methods applied are described in Sect. 2; the results are presented in Sect. 3 and, finally, in Sect. 4 some conclusions are drawn.

2 Data and Methods The data used in this study is provided by the 2011 Italian Population and Housing Census: the distribution of usually resident population broken down by country of citizenship at sub-municipality level (enumeration areas) and the shape files of the four municipalities under analysis. The applied methodology follows a spatial approach. In particular, we computed two versions of the mean center (geographical and weighted) and two versions of the standard deviational ellipse (geographical and weighted). The geographical mean center (GMC) is a point identified in a Euclidean space by the arithmetic mean of geographical coordinates (longitude and latitude) related to all points – n centroids of each enumeration area in our case – constituting a specific territory at a certain time t.   n n   xi /n; yi /n (1) (XGMC ; YGMC )  i1

i1

The weighted mean center (WMC) is a synthetic spatial index as well and it represents the center of gravity of the spatial distribution of a population in a given territory at a given time t.1 WMC is a point identified in a Euclidean space by the arithmetic mean of geographical coordinates (longitude and latitude) related to all points – n centroids of each enumeration area in our case – constituting a specific territory, weighted by the population usually residing in each spatial unit (pi ) at a given time t.   n n n n     xi pi / pi ; yi pi / pi (2) (XW MC ; YW MC )  i1

1 In

i1

i1

i1

this perspective the WMC coincides with the Gini’s center of population firstly proposed by Gini and colleagues in the early 1930s [10, 11].

Spatial Residential Patterns of Selected Foreign Groups …

73

Like all mean values, the informative level of GMC and WMC can be improved by a spatial measure that takes into account the dispersion of the observed distribution around the mean center. This measure is known as Standard Deviational Ellipse (SDE), a measure firstly proposed by Lefever [12]. The SDE can capture the directional bias in a point distribution.2 It is an ellipse centered on the mean center (geographical or weighted) of a spatial distribution and it is defined by three parameters: angle of rotation, deviation along the major axis and deviation along the minor axis [13]. The angle of rotation is computed as:  tan θ 

n  i1

x˜ i2 −

n  i1





y˜ i2 +

n 

x˜ i2 −

i1 n 

2

n  i1

2 y˜ i2

 +4

n 

2 x˜ i y˜ i

i1

(3)

x˜ i y˜ i

i1

The deviation along x-axis and y-axis is computed as:

σx 





n

(˜xi cos θ − y˜ i sin θ )2  i1 n

; σy 





n

(˜xi sin θ − y˜ i cos θ )2  i1 n

(4)

The major axis is the major value between σx and σy while the minor axis is the minor value between σx and σy . In case of Geographical Standard Deviational Ellipse (GSDE), x˜ i and y˜ i are computed as: (˜xi ; y˜ i )  (xi − XGMC ; yi − YGMC )

(5)

while in case of Weighted Standard Deviational Ellipse (WSDE): (˜xi ; y˜ i )  (pi xi − XW MC ; pi yi − YW MC )

(6)

In each municipality, the spatial distribution of the selected foreign groups and the Italians – synthesized by the WMC and the WSDE – will be compared to the GMC and GSDE.3 We will also evaluate the degree of similarity/dissimilarity between the spatial distribution of each foreign group and the spatial distribution of Italians. In doing so we will take into account the shapes of SDE (geographical and weighted) and two additional parameters: the Euclidean distance and the ratio with geographical 2 When

the features have a spatially normal distribution, one standard deviation will encompass approximately 68% of all input feature centroids. 3 We computed GMC and GSDE taking into consideration all the enumeration areas (i) of each municipality. As known, some of them could not be populated. Nevertheless their number is limited and their impact on the measure is negligible. All the measures (geographical and weighted) were computed using ARCGIS ESRI 10.03.

74

F. Benassi and F. Lipizzi

axis. The first parameter measures the linear distance between the center of gravity of the spatial distribution of each foreign group and the geographical one. The second parameter measures the degree of spatial dispersion of each spatial distribution compared to the geographical one. This parameter is composed of two values: ‘minor’ and ‘major’. For each foreign group the value of the ‘minor’ parameter is obtained as the ratio between the minor axis of the SDE and the minor axis of the GSDE. In the same way, the value of the ‘major’ parameter is obtained as the ratio between the major axis of the SDE and the major axis of the GSDE. When the value of the ratio is lower than 1, this means that the observed spatial distribution has a degree of dispersion lower than the geographical one and vice versa when the ratio is more than 1. It is important to highlight that the measures here applied are affected by the geography of the territory under analysis. This implies that by these measures it is only correct to compare the spatial distributions of groups of populations that are related to the same space. Moreover, the proposed methodology has some additional limitations: (1) the possible bias induced by considering only official resident population data; (2) the use of descriptive statistical tools that roughly synthetize the spatial distribution of the foreign groups; (3) the (partial) use of visual comparisons of graphical results to assess differences among spatial distributions; (4) the lack of consideration concerning confounding factors that could drive the differences between groups. Despite these limitations, we believe that this kind of approach represents a useful explorative tool that allows an intuitive interpretation of the spatial residential patterns of specific groups of human populations.

3 Results Before presenting the results, we provide some figures about the foreign groups selected in the study for each municipality. The total number of foreigners enumerated in 2011 in the municipality of Milan was 176,303, corresponding to 14.2% of the total population of the municipality. The data for Rome, Naples and Palermo are, respectively: 224,493 (8.6%), 31,496 (3.3%) and 19,644 (3.0%). In Milan, the largest foreign group is represented by Filipinos, followed by Chinese, Egyptians, Peruvians and Sri Lankans. These communities represent 56.2% of the foreign population accounted in the same municipality and they represent 8.0% of the total population resident in Milan. In Rome, the largest foreign group is represented by Romanians, followed by Filipinos, Bangladeshis, Chinese and Peruvians. These five population groups represent 52.2% of the foreigners reported in Rome and they account for 4.5% of the total population resident in the same municipality. In Naples, the largest foreign group is represented by Sri Lankans, followed by Ukrainians, Chinese, Romanians and Filipinos. These communities represent 60.5% of foreign population enumerated in the same municipality and 2.0% of the total population resident in Naples. Finally, in Palermo, the biggest group is represented by Bangladeshis, followed by Sri Lankans, Romanians, Ghanaians and Filipinos. These communities represent 61.6% of foreigners reported in Palermo and 1.8% of the total population resident

Spatial Residential Patterns of Selected Foreign Groups …

75

Fig. 1 GMC, GSDE, WMC, WSDE. Italians and selected foreign groups. Milan. Source Our own elaboration on 2011 Population and Housing Census data, Istat

in the same city. Looking at Figs. 1, 2, 3, 4 and Table 1, we can appreciate a quite high heterogeneity amongst the spatial distributions of selected population groups inside each municipality. In the first one, Milan, the center of gravity of Italians is the closest to the GMC, 80.6 m, which is located in the “Centro” area. The mean centers of the other population groups are located not so far from the GMC as well. Euclidean distances are in fact less than 1 km, with the exception of the Chinese community which presents the highest distance from the GMC, almost 2 km, towards the north-west part of the city (Fig. 1 and Table 1). From a geographical point of view, the shape of WSDE related to Italians and to other population groups is not so different among them and versus the GSDE. This means that the spatial distributions of foreign groups are not characterized by a particular degree of concentration. In this perspective, the Chinese community represents, once again, an exception: as we can see from Fig. 1, Chinese WSDE indicates that this population group is mainly concentrated in the north-west part of the municipality, in particular in “Porta Nuova”, “Bovisa”, “Niguarda” and “Fulvio-Testi” areas. With regard to the ratio with geographical axis, Chinese community is the only one with both values slightly lower than 1 (Table 1). In the case of Rome, the center of gravity of Italians is the closest to the GMC, almost 410 m, which is located in “San Saba” area. As in the case of Milan, the Chinese community records the highest distance from the GMC, almost 4 km; the

76

F. Benassi and F. Lipizzi

Table 1 Euclidean distancea and Ratio with geographical axisb . Italians and selected foreign groups. Milan, Rome, Naples and Palermo. Source Our own elaboration on 2011 Population and Housing Census data, Istat Euclidean distance

Ratio with geographical axis Minor

Major

Milan Italians

80.6

1.0

1.0

Egyptians

131.4

1.1

1.0

Sri Lankans

296.0

1.1

0.9

Peruvians

486.7

1.0

1.1

Filipinos

591.5

0.9

1.0

Chinese

1887.1

0.9

0.8

409.9

0.9

0.9

Romanians

1742.7

1.1

1.2

Peruvians

1968.0

0.9

0.8

Filipinos

2354.9

0.7

0.6

Bangladeshis

2825.1

0.5

0.6

Chinese

3913.3

0.5

0.7

Rome Italians

Naples Italians

483.8

1.1

1.0

Ukrainians

488.1

0.9

0.9

Romanians

548.9

0.9

1.0

Sri Lankans

583.1

0.5

0.5

Filipinos

1982.0

0.5

0.7

Chinese

2030.8

0.3

0.4

Romanians

402.4

1.2

1.1

Italians

612.0

0.9

0.9

Palermo

Filipinos

800.5

0.5

0.6

Bangladeshis

906.4

0.5

0.4

Sri Lankans

947.9

0.5

0.9

Ghanaians

952.8

0.6

0.6

a The b The

Euclidean distances are expressed in meters and they are computed from the GMC ratios are calculated with respect to the GSDE

Spatial Residential Patterns of Selected Foreign Groups …

77

Fig. 2 GMC, GSDE, WMC, WSDE. Italians and selected foreign groups. Rome. Source Our own elaboration on 2011 Population and Housing Census data, Istat

Chinese’s WMC is located towards the north-east part of the city near the “Tuscolana” area (Fig. 2 and Table 1). Mean center of other foreign groups are located about 2 km far from the GMC: Romanians, 1.7 km in “Appio Latino” area; Peruvians, 2.0 km in the “Monti” area; Filipinos, 2.3 km in the “Colonna” area; Bangladeshis, 2.8 km in the “Tuscolana” area (Fig. 2 and Table 1). Observing the shape of the WSDE of each population group, we can appreciate a high level of heterogeneity among their spatial distributions. Romanians have a quite disperse spatial distribution, covering the entire municipality surface, in particular the eastern and the south-eastern quadrant of the city. Filipinos are quite concentrated in the center of the municipality, their WSDE is in fact smaller than the GSDE and it is located in the central part of the city. Peruvians have a WSDE located in city center as well, with a circular shape covering a smaller area than the one identified by the GSDE. Chinese and Bangladeshis communities have both a very small WSDE, oriented to the north-eastern quadrant of the municipality. Both population groups are concentrated in the city center, with Bangladeshis scoring the smallest WSDE. With regard to the ratio with geographical axis, low values are recorded for both parameters by Filipinos, Bangladeshis and Chinese, with Bangladeshis recording the lowest values. Italians, as in the case of Milan, show a spatial distribution quite similar to the geographical one (Fig. 2 and Table 1).

78

F. Benassi and F. Lipizzi

Fig. 3 GMC, GSDE, WMC, WSDE. Italians and selected foreign groups. Naples. Source Our own elaboration on 2011 Population and Housing Census data, Istat

In the case of Naples (Fig. 3 and Table 1), as in Milan and Rome, the center of gravity of Italians is the closest to the geographical one, almost 0.5 km, which is located in “Avvocata” area. Similar distances are recorded by Ukrainians, Romanians and Sri Lankans. The mean centers of the first two communities are located in the “Avvocata” area as well but, compared to the Italians, they are located more on the southern part of the municipality. The mean center of Romanians is located in the north-east part of the city, in the “Stella” area. On the other hand, Filipinos and Chinese record the highest distance, almost 2 km. The WMC of the first community is located in the “Chiaia” area near the harbour, while the mean center of the Chinese community is located between the “Mercato” area and the industrial area. The Italians have a scattered spatial distribution, the surface covered by their WSDE is bigger than the one covered by the GSDE, with particular concentration in the northern and western quadrants of the city. The Ukrainians have a WSDE smaller than the geographical one, oriented towards the central-western part of the municipality. Therefore, they are quite concentrated in these municipality areas. The shape of Romanians WSDE indicates instead a quite disperse distribution oriented towards the central-eastern part of the municipality. This foreign community is therefore less concentrated in the center of the city, with an orientation towards the eastern side of the municipality area. The case of Sri Lankans, Chinese and Filipinos is interesting (Fig. 3). The shape of their WSDE shows that they are concentrated in the central

Spatial Residential Patterns of Selected Foreign Groups …

79

Fig. 4 GMC, GSDE, WMC, WSDE. Italians and selected foreign groups. Palermo. Source Our own elaboration on 2011 Population and Housing Census data, Istat

part of the municipality. This is true especially for the Chinese community, which presents the highest level of concentration. In particular, Sri Lankans are distributed across the north-east and south-west axis, Filipinos are distributed across the same axis but closer to the coast and, finally, Chinese are concentrated in the eastern quadrant of the municipality. Comparative low values of the ratio with geographical axis are in fact recorded by the same communities, with Chinese recording the lowest values (Table 1). In the city of Palermo (Fig. 4 and Table 1), the smallest distance from the GMC is recorded by Romanians, 0.4 km. In any case, all the population groups present a distance from the GMC that is less than 1 km, with the Sri Lankans and the Ghanaians recording the highest distance (nearly 1 km). As for spatial distribution, from Fig. 4, we can observe that Italians have a disperse spatial distribution, with a preference for the northern and western part of the municipality. All of the other communities (in different degrees) are concentrated in the central part of the municipality. In particular, Ghanaians and Bangladeshis, who have a very small WSDE, are distributed across the north-west axis, Sri Lankans are distributed across the north-east axis and Filipinos across the north-west axis. The Bangladeshis present the highest level of concentration. The same communities record low values of the ratio with geographical axis, with Bangladeshis recording the lowest values (Table 1).

80

F. Benassi and F. Lipizzi

4 Conclusions In this study we showed the different spatial residential patterns of selected foreign groups (and Italians) residing four of the largest Italian cities. As we expected, the spatial distribution of Italians is the most similar to the geographical one in terms of both mean center and standard deviational ellipse. Conversely, the spatial residential pattern of Chinese community is the most dissimilar to the geographical one in Milan, Rome and Naples. A comparative high level of concentration is showed by Chinese in Milan; Chinese, Filipinos and Bangladeshis in Rome; Chinese, Sri Lankans and Filipinos in Naples; Bangladeshis, Sri Lankans and Ghanaians in Palermo. Romanian and Ukrainian communities present a spatial residential pattern characterized by a comparative high level of dispersion (this is true for Rome, Naples and Palermo). In coclusion, it seems that foreign communities coming from more distant countries show a spatial residential pattern characterized by a low level of spatial dispersion and a level of dissimilarity comparatively high with respect to the spatial patterns of Italians. Conversely, communities coming from less distant countries show a spatial residential pattern identified by a high level of spatial dispersion and a comparatively low degree of dissimilarity with respect to the spatial residential patterns of Italians.

References 1. Bonifazi, C.: L’immigrazione straniera in Italia. Il Mulino, Bologna (1998) 2. Feitosa, F.F., Câmara, G., Montiero, A.M.V., Koschitki, T., Silva, M.P.S.: Global and local indices of urban segregationInt. J. Geogr. Inf. Sci. 299–323 (2007) 3. Massey, D.S., Denton, N.A.: The Dimension of Residential Segregation. Social Forces (1988). https://doi.org/10.1093/sf/67.2.281 4. Wong, D.W.S.: Geostatistics as measures of spatial segregation. Urban Geogr. 20, 635–647 (1999) 5. Reardon, S.F., O’Sullivan, D.: Measures of spatial segregation. Sociol. Methodol. (2004). 10.111/j.0081-1750.2004.0150.x 6. Ferruzza, A., Dardanelli, S., Heins, F., Verrascina, M.: La geografia insediativa degli stranieri residenti: Verona, Firenze e Palermo a confronto. Stud. Emigr./Migr. Stud. 171, 602–628 (2008) 7. Ferrara, R., Forcellati, L., Strozza, S.: Modelli insediativi degli immigrati stranieri in Italia. Boll. della Soc. Geogr. Ital.3, 619–639 (2010) 8. Boeri, T., Philippis, M., Paracchini, E., Pellizzari, M.: Moving to segregation: evidence from 8 Italian cities. Institute for the Study of Labor (2012). http://ftp.iza.org/dp6834.pdf 9. Benassi, F., Ferrara, R., Gallo, G., Strozza, S.: La presenza straniera nei principali agglomerati urbani italiani: implicazioni demografiche e modelli insediativi. In: Donadio, P., Gabrielli, G., Massari, M. (eds.) Uno come te. Europei e nuovi europei nei percorsi di integrazione, pp. 186–198. Franco Angeli, Milano (2014) 10. Gini, C., Galvani, L.: Di talune estensioni dei concetti di media ai caratteri qualitativi. Metron VIII, 3–209 (1929)

Spatial Residential Patterns of Selected Foreign Groups …

81

11. Gini, C., Boldrini, M., Galvani, L., Venere, A.: Sui centri della popolazione e sulle loro applicazioni. Metron 8, 3–102 (1933) 12. Lefever, D.W.: Measuring geographic concentration by means of standard deviational ellipse. Am. J. Sociol. 32, 88–94 (1926) 13. Lee, J., Wong, D.W.S.: Statistical Analysis with ArcView Gis. Wiley, New York (2001)

Minority Segregation Processes in an Urban Context: A Comparison Between Paris and Rome Oliviero Casacchia, Luisa Natale and Gregory Verdugo

Abstract The process of minority segregation within global cities is a complex phenomenon. In the European urban context, the process of minority segregation seems to differ in the old immigration countries (France, UK, Netherlands, Germany) from the new immigration countries (Italy, Greece, Spain, Portugal). The analysis compares two global cities (Paris and Rome), taking into consideration the current evolution of the minority segregation pattern. The study shows that in both cities no traces of significant increasing segregation emerge over the last twenty years. Keywords Minority residential segregation · Paris and Rome comparison · Isolation and dissimilarity index

1 Introduction 1

Paris and Rome are global cities with large foreign populations, currently amounting to about 1.8 million in Paris (in 2007 18% of the total population) and 334 thousand in Rome (or 8.2% in 2011: Table 1). Both urban areas had seen an increase in the foreign population, moderate in Paris, quite striking in Rome [4]. The city has always exerted a force of attraction for the newly arrived [9], who tend to settle on the basis of spatial localisation models so characteristic that they have been widely addressed in the literature [1, 3, 11, 12].

1 The territory considered in this study is not limited to the municipalities of Paris and Rome alone,

but extends further over what is defined as is the urban area. It is generally recognised that study of the residential inclusion of foreign populations cannot be limited to the core given the tendency of foreigners to settle also in the outlying areas. In the paper we use the definition of aire urbain for Paris, according to the official INSEE (Institut National de la Statistique et des Études Économiques) proposal. We create a metropolitan area for Rome, which includes the Rome municipality itself and three strata of contiguous municipalities. In sum 104 municipalities were included. O. Casacchia · L. Natale (B) · G. Verdugo University of Cassino and Southern Latium, Cassino, Italy e-mail: [email protected] © Springer Nature Switzerland AG 2019 C. Crocetta (ed.), Theoretical and Applied Statistics, Springer Proceedings in Mathematics & Statistics 274, https://doi.org/10.1007/978-3-030-05420-5_9

83

84

O. Casacchia et al.

Table 1 Dynamic of de jure foreign population from ‘90 to ‘10. Paris and Rome urban areas. Absolute and percentage values

Years Paris urban area

Years Rome urban area

Absolute value

Absolute value

%

%

1990 1,379,808

14.9

1991 55,496*

2.3

1999 1,606,359

15.5

2001 131,706

3.5

2007 1,790,582

18.0

2011 334,432

8.2

Note* 85,619 de facto foreign population Source population censuses, various years

The aim of this study is to compare the two settlement models [10], seeking to detect some aspects regarding the territorial process of assimilation of the foreign population. The focus of interest has thus shifted to analysis of the recent dynamics of the territorial integration process shown by the foreign population in the urban areas of Paris and Rome over the last twenty years. In both cities we observe the spatial integration process by analyzing what is termed the de jure population.

2 Data and Methods The analysis proposed is based on data from the three last censuses. Different criteria are applied to define what is meant by foreign population: in Italy we consider citizenship, while in France the definition of immigré is used, an immigrant being defined as an individual born abroad from parents both lacking French citizenship at the moment of birth. Two measures were used to analyze the territorial distribution of foreign population: a. the index of dissimilarity (ID), a measure of evenness [7] which compares an observed spatial distribution to a theoretical, absolutely even distribution. The Index is defined as: D  0.5

 i

|Ai /A − Ni /N|

where, in our case, Ai /A represents the share of the population belonging to group A in urban zone i,2 while Ni /N similarly refers to the autochthon population;

2 The territorial grid used in the study includes IRIS

in Paris, the Ilots Regroupés pour l’Information Statistique, a socio-spatial division equivalent to a census tract introduced by INSEE (see, for the definition, [8]). As far as Rome is concerned, we used two territorial grid levels: first, a macro one including the 122 urban zones - or toponomastic zones - in which the Rome municipality is subdivided plus the 103 municipalities surrounding Rome; second, a micro territorial grid including census blocks in which at least one resident is counted in census years.

Minority Segregation Processes in an Urban Context

85

b. the isolation index (IS), a measure of the degree of exposure of each subgroup in each area of residence [2, 8]. The index was calculated – for each urban zone - as the subgroup-weighted mean of the subgroup proportion. In formal terms, with Pi that represents the amount of the population in urban zone i: IS 

 i

Ai /A × Ai /Pi

The isolation index is particularly sensitive to the size of the group examined, and it is therefore difficult to make longitudinal and cross-sectional comparison among groups unless they are constant as size in time and space. To try to correct this measure we calculate an adjusted isolation index (ISa ) simply calculating the difference between the observed IS and the expected IS, that of the overall population, assuming an even distribution of the subgroup. For example, let us suppose that the overall percentage of the group out of the total population is 10% and that IS yields the value of 30%; in other words, that the probability that a member of our ethnic subgroup meet another member of the same subgroup, calculated considering the different weights of this population in each of the zones, is three times that obtained by simply considering the population as a whole. In this case, the difference between the IS and the proportion of the subgroup in the population is 20%. The more uneven the distribution of the subgroup across the city, the higher will be the value of the IS, and thus also the difference shown by the latter with respect to the proportion of the subgroup in the overall population.

3 Results Paris showed considerable variability in the dissimilarities between the various groups. The territorial model for Turks proved the most distant from that of the native population, while that of the European community’s came closest. The dynamics over the period 1990–2007 showed no significant variations. An appreciable reduction of dissimilarity was observed in the case of the North African community, while the communities showing greater similarity reveal a tendency towards differentiation over time (Table 2).3 This stability is relatively surprising given the large increase in the relative number of non-European immigrants over the period. Several mechanisms can explain these results but an important change mentioned in the recent literature is the generalization of the access of immigrants to public housing, which had two main consequences. First, it decreased the most extreme forms of segregation which where correlated with quite difficult housing conditions 3 Considering

the adjusted version of the Dissimilarity Index (for the sake of brevity not reported here), on checking for the random component of the territorial distribution (see [8]) no trace of increase emerges in the 1990–2007 years in the index constructed for Italians and Spanish immigrants (on the contrary, the index shows a decrease).

86

O. Casacchia et al.

Table 2 Dissimilarity Index by national origin and geo-cultural origin. Paris Urban Area. 1990–2007

National and geo-cultural origin

Observed indices 1990

1999

2007

Algeria

38.6

37.6

36.6

Morocco

38.8

37.8

36.3

Tunisia

39.4

38.3

36.2

Turkey

55.3

58.4

55.1

Italy

29.7

33.2

32.2

Spain

29.1

31.5

31.2

Portugal

25.8

27.4

28.2

Sub-Saharan Africa

38.8

35.0

35.4

North Africa

34.3

34.0

34.0

East Asia

40.7

39.5

37.2

Middle East

37.3

40.5

36.8

South Europe

20.3

22.6

22.7

Other European

28.7

28.9

28.8

Total

22.1

22.6

23.6

Source French Population Censuses, various years

[13]. Second, it decreased the share of immigrants locating in Paris region as immigrant tend to choose increasingly to locate in regions where public housing was more easily available [14]. Rome showed a decidedly sharp fall in the dissimilarity index over the 20-year period 1991–2011. This finding is borne out both by the evidence of analysis applied to the macro-zones (i.e. the 122 urban zones into which the Rome municipality is divided plus the 103 municipalities in its hinterland) and on constructing the indicator at the level of census blocks (Table 3).4 Analysis of the isolation index also yields results quite consistent with the dissimilarity analysis. In this case, following the method previously described (see above), the difference between the IS value observed and the percentage of foreigners out of the total population is analysed. In fact, considering that the isolation index is particularly sensitive to the size of the foreign percentage in the population (which itself accounts for an increase in the index), the index reflects fairly closely the exposure dynamics of the groups studied if considered net of the percentage of each group in the total population. As far as Paris is concerned (Table 4), a slight increase in the isolation of the foreign population emerges (IS ranging from 18.6% in 1990 to 22.3 in 2007), but once measurement is adjusted no such increase is observed. Moreover, the index level also appears relatively low (about 4%: Table 4), and extremely low in the case

4 Similar

results were obtained by Heins and Strozza [6: 582] and by Barbagli and Pisati [1: 257].

Minority Segregation Processes in an Urban Context

87

Table 3 Dissimilarity and Isolation Index by geo-cultural origin. Rome Area. 1991–2011 Groups

Dissimilarity Index (D)

Total

28.3

by census block

57.0

Isolation Index (IS) [1]

%[2]

Difference [1, 2]

2.7

1.5

1.2

6.2

1.5

4.7

1.0

1991

2001 Europe

19.9

2.6

1.6

Africa

21.4

0.9

0.5

0.4

America

24.0

0.8

0.5

0.3

Asia

33.8

1.8

0.8

1.0

Total

18.2

5.0

3.5

1.5

by census block

43.7

9.8

3.5

6.3

2011 Total

14.4

9.7

8.2

1.5

by census block

34.8

16.5

8.2

8.4

Source Italian Population Census

of the long-established European communities, namely the Spanish (0.3 in 2007) and Italians (0.4), as well as the Tunisians (0.9 in the last year). In the case of Rome, over the period between 1991 and 2011 a significant increase in IS is observed (from 2.7 to 9.7%: Table 3), but it practically disappears when the values of the adjusted index are considered. It is, however, worth noting that the trend of the index constructed at the level of census blocks shows a systematic increase over the period in terms of both the index and the adjusted value. This datum hardly fits in with the pattern previously observed; it will be further investigated with closer analysis (see Table 3). In Paris at 2007 a small percentage of immigrants live in neighbourhoods with a high percentage of foreigners: bearing in mind the definition of a segregated neighbourhood as an area with at least 30% of foreigners, this applies to 26% of the foreigners in Paris (Fig. 1), while the majority lives in areas where this percentage is less than 30% (about 40% of the foreigners lives in areas where the foreign population percentage ranges from 15 to 25%).

88

O. Casacchia et al.

Table 4 Isolation Index by national origin and geo-cultural origin. Paris Area. 1990–2007 National and geo-cultural origin

Observed indices [1]

% on total population [2]

Difference [1, 2]

1990

1999

2007

1990

1999

2007

1990

1999

2007

Algeria

4.4

3.9

4.6

2.1

2.1

2.5

2.3

1.9

2.1

Morocco

3.9

3.9

4.2

1.4

1.6

2.0

2.5

2.3

2.2

Tunisia

2.1

1.9

1.8

0.9

0.9

0.9

1.2

1.0

0.9

Turkey

2.8

3.5

3.2

0.4

0.5

0.6

2.4

3.0

2.6

Italy

1.3

1.1

1.0

0.8

0.6

0.6

0.5

0.5

0.4

Spain

1.3

1.0

0.8

0.8

0.6

0.5

0.5

0.4

0.3

Portugal

3.7

3.7

3.4

2.5

2.4

2.2

1.2

1.3

1.2

Sub-Saharan Africa

3.9

4.1

5.9

1.6

2.2

3.2

2.3

1.9

2.7

North Africa

8.0

7.9

9.0

4.4

4.5

5.4

3.6

3.4

3.6

East Asia

5.0

5.4

5.7

1.4

1.8

2.2

3.6

3.6

3.6

Middle East

2.8

3.2

3.0

1.1

1.0

1.1

1.7

2.2

1.9

South Europe

5.1

4.8

4.3

4.0

3.5

3.2

1.1

1.3

1.1

Other European

2.8

2.8

3.1

1.8

1.8

2.0

1.0

1.0

1.1

18.6

19.3

22.3

14.9

15.5

18.0

3.7

3.8

4.3

Total

% of immigrants in each group of zones

Source French Population Censuses, various years 40.0% 35.0% 30.0% 25.0%

Rome

Paris

20.0% 15.0% 10.0% 5.0% 0.0%

Percentage of immigrants by zone Fig. 1 Distribution of immigrant by share of immigrants in each zone (IRIS in Paris, 2007; urban zone plus municipalities in Rome, 2011)

Minority Segregation Processes in an Urban Context

89

4 Discussion We can therefore consider trends in segregation in Paris and Rome over the last twenty years within a single framework. No trace of increasing of segregation emerges in these two areas: neither of the segregation indexes shows an increase in segregation for the foreign population as a whole. Moreover, where it is possible to carry out dynamic analysis distinguishing amongst the nationalities of the groups under examination – at the moment, only in the urban area of Paris – segregation shows a generally declining trend, more notable among the groups from Maghreb.5 Thus the urban area of Paris, too, appears to show the same pattern of trends in segregation observed, according to some authors [8], for the urban areas of France as a whole. Nevertheless, as for the rest of the country, there remain large differences between European and non-European immigrants. In a context in which the relative share of European immigration decreased rapidly, this has led to an overall moderate increase in immigrant segregation levels in Paris. More importantly, recent research suggests that second-generation immigrants from non-European origins tend to concentrate disproportionately on neighbourhoods in which first-generation immigrants are already overrepresented. This has potentially led to an increase in the share of neighbourhoods concentrating inhabitants from non-European origins. However, these evolutions are difficult to measure empirically as information on second-generation status is not available in the French census data. The first findings thus appear to bear out the conclusions arrived at in analysing segregation in Italian cities [1] over a shorter period (the last decade) with a reference to a more limited territory of than considered here (not the urban areas but only the municipal areas). In general, the authors argue that the moderate fall in the level of segregation observed between 2001 and 2011 is not to be accounted for with variables like the rate of increase in immigrant communities or the geographical areas of origin (ibid, 253). We may conclude that the recent moderate changes in the global levels of residential segregation of foreigners are not clearly attributable to any factors that can be identified with a reasonable degree of certainty. In the case of Paris, too, factors are at work that are difficult to identify at this aggregate level of analysis, while it has emerged fairly evidently that the classical explanatory factors such as the length of residence play mostly marginal roles.6 It is, moreover, to be borne in mind that the analysis is presented here is limited to comparing recent trends in segregation between Paris and Rome, without being 5 In

fact it must be borne in mind that a substantial part of Maghreb origin population has acquired French citizenship in the past. In this case, it is possible that the rising degree of consistency between the geographical distribution of immigré from Maghreb and those of autochthones population (whether French citizenship was acquired at birth or not) is also caused by the presence of a significantly component of French citizens originated from Maghreb. 6 «La ségrégation ne baisse que faiblement avec l’ancienneté de l’arrivée des immigrés en logement privé (… et …) les taux de ségrégation moyens ne diminuent que faiblement au fil du séjour en France pour les personnes vivant en logement social » [13: 188].

90

O. Casacchia et al.

able to address comparison between the levels. In any case, comparison between the levels of segregation in the two urban areas cannot be performed on the basis of construction of the indexes proposed insofar as the indexes are affected by the numerousness, form and extensiveness of the territorial units taken for analysis. Further stages of research will see the possibility to define measurements that can conform to the rules for generalisation of segregation level assessment over time and in different areas. This possibility lies, we believe, both in the definition of spatial association measurements and in some recent contributions based on social interactions [5].

References 1. Barbagli M., Pisati M.: Dentro e fuori le mura. Città e gruppi sociali dal 1400 ad oggi. Bologna: Il Mulino (2012) 2. Bell, V.: A probabiliy model for the measurement of ecological segregation. Soc. Forces 43, 357–364 (1954) 3. Borjas G.J.: Ethnicity, neighborhoods, and human capital externalities. Am. Econ. Rev. 85, 365–390 (1995) 4. Casacchia, O., Natale, L., Martino, G.: La presenza straniera all’interno della città: Roma e Parigi a confronto. CISU, Roma (2012) 5. Echenique, F., Fryer Jr., R.G.: A measure of segregation based on social interactions. Q. J. Econ. 122(2), 441–485 (2007) 6. Heins, F., Strozza, S.: La geografia insediativa degli stranieri all’interno delle province italiane: differenze e determinanti. Studi Emigrazione 45(171), 573–601 (2008) 7. Massey D.S., Denton N.A.: The dimensions of residential segregation. Soc. Forces, 67, 281:315 (1988) 8. Pan Ké Shon J.L., Verdugo G.: Forty years if immigrant segregation in France: 1968–2007. How different is the new immigration? Urban Stud. 52(5), 823–840 (2015) 9. Park R., Burgess E. (eds.): The Urban Community. The University of Chicago Press, Chicago (1926) 10. Peach C.: London and New York: Contrasts in British and American models of segregation. Int. J. Popul. Geogr. 5, 319–351 (1999) 11. Poppe, W.: Patterns and meanings of housing: residential mobility and homeownership among former refugees. Urban Geogr. 34(2), 218–241 (2013) 12. Préteicelle, E.: La ségrégation ethno-sociale dans la métropole parisienne. Revue française de Sociologie 50(3), 489–515 (2009) 13. Verdugo, G.: Logement social et ségrégation résidentielle des immigrés en France, 1968–1999. Population-F 66(1), 171–196 (2011) 14. Verdugo, G.: Public Housing magnets: public housing supply and immigrants’ location choices. J. Econ. Geogr. 16(1), 237–265 (2016)

Similarity of GPS Trajectories Using Dynamic Time Warping: An Application to Cruise Tourism Mauro Ferrante, Christian Bongiorno and Noam Shoval

Abstract The aim of this research is to propose an analysis of the trajectories of cruise passengers at their destination using Dynamic Time Warping algorithm. Data collected by means of GPS devices relating to the behavior of cruise passengers in the port of Palermo have been analyzed in order to show similarities and differences among their spatial trajectories at destination. A cluster analysis has been performed in order to identify segments of cruise passengers, based on the similarity of their trajectories. The results have been compared in terms of several metrics derived from GPS tracking data in order to validate the proposed approach. Our findings are of interest from a methodological perspective concerning the analysis of GPS data and the management of cruise tourism destinations. Keywords Cruise tourism · Dynamic time warping · GPS trajectories

1 Introduction Cruise lines and their passengers contribute to the economy of many destinations. Although most companies and port authorities collect and distribute data on the number of cruises and of related passengers, there is a general paucity of information relating to the behavior of cruise passengers at their destination. The majority of M. Ferrante (B) Dipartimento Culture e Società, Università degli Studi di Palermo, Palermo, Italy e-mail: [email protected] C. Bongiorno Dipartimento Fisica e Chimica, Università degli Studi di Palermo, Palermo, Italy e-mail: [email protected] N. Shoval The Department of Geography, The Hebrew University of Jerusalem, Jerusalem, Israel e-mail: [email protected] N. Shoval The University Center for Urban and Social Research, The University of Pittsburgh, Pittsburgh, PA, USA © Springer Nature Switzerland AG 2019 C. Crocetta (ed.), Theoretical and Applied Statistics, Springer Proceedings in Mathematics & Statistics 274, https://doi.org/10.1007/978-3-030-05420-5_10

91

92

M. Ferrante et al.

studies concerning the behavior of cruise passengers is conducted by means of a questionnaire-based survey, or with the observational method [9, 19]. In recent years, the use of tracking technologies in social sciences has permitted the collection of accurate information on human behavior. In turn, this has determined the rise of new fields of investigation into human mobility patterns, based on data collected by means of Global Positioning Systems (GPS) technology [31]. The widespread availability of such types of data has resulted in a growing interest in designing techniques for identifying trajectories which are similar to each other. The aim of the work outlined in this Paper is to analyze data relating to the mobility of cruise passengers in the city of Palermo, by means of data collected with the use of GPS tracking devices. The aim is to implement a methodology which is capable of comparing GPS trajectories and to segment cruise passengers based on their spatial behavior at the destination. The knowledge of the most recurrent itineraries followed by cruise passengers at their destination can help destination managers in transportation planning and in the promotion and management of destination attractions.

2 Background In recent years, several contributions have focused their attention on the behavior of cruise passengers at their destination [3, 7–9, 19]. An improved understanding of this behavior is an essential prerequisite for the management of tourism destinations, given the challenges of cruise tourism at many coastal destinations. Nonetheless, the collection and analysis of data relating to the mobility of cruise passengers at their destination is a complex task. Traditional approaches for analyzing large-scale human mobility in travel surveys, including travel diaries, are not easily applicable to the field of cruise tourism [18]. An analysis of the mobility of cruise passengers from diaries or self-reported routes may entail several pitfalls [23]: cruise passengers may find the effort required taxing and, therefore, leave the diary incomplete or skip certain parts. Given the limited amount of time for visiting the destination, it is unlikely that cruise passengers would be willing to participate in this type of survey. Moreover, cruise passengers are not always able to easily locate the site to be marked in the diary/on the map [27]. For example, Guyer and Pollard [17] have investigated cruise passengers’ impressions of the environment through a survey of cruising on the interlinked systems of the Shannon-Erne Waterways in north-west Ireland. In this study, the authors pointed out that establishing points of attraction is not an easy matter, particularly in a rural environment where the names of so many landscape features are unknown to the visitor in most cases. More recently, the widespread availability of GPS technologies offers new effective surveying tools, which facilitate the collection of accurate information on human mobility in a cost-effective way [32]. In the field of cruise tourism, GPS technology enables the accurate recording of temporal and spatial behavior of cruise passengers, that is: which attractions they visit on site, how long they spend at each attraction, the sequence of attractions visited, etc. [13]. This method provides objective information

Similarity of GPS Trajectories Using Dynamic Time Warping …

93

which is not biased by the visitors’ perceptions; it can be useful from several perspectives, including transportation planning and, more generally, the management of tourism services.

3 Materials and Methods In the spring of 2014, a survey relating to cruise passengers docking in the port of Palermo was conducted. In order to collect information relating to the behavior of cruise passengers at their destination, a GPS device was distributed to every sampled cruise passenger. Moreover, two questionnaires (one pre-visit and one postvisit) were administered, aiming at collecting information on socio-demographic characteristics and on other elements related to cruise passengers’ experience at their destination [9]. The survey varied, depending on whether passengers decided to take part in a tour organized by the cruise company or those who decided to visit the destination by themselves, henceforth named ‘independent cruise passengers’. For the purposes of this study, only information related to independent cruise passengers has been considered, with the aim of comparing their trajectories at the destination. After pre-processing the information, which involved data cleaning and validation [13], several concise measures related to the itinerary were derived. These included: the total duration of the itinerary, the total length of tour, the maximum distance from the port and the 90th percentile of speed. The total duration of the itinerary could be easily derived by considering the difference between the time at the end and beginning of the tracking; the total length of tour was given by the sum of all the distances between consecutive pairs of coordinates (recorded every 10 s). By considering that cruise passengers must return to the same point at the end of their visit (i.e. to the cruise ship), important information relating to their mobility was given by the distance from the port. In particular, the maximum distance from the port could provide concise information regarding the extent of exploratory behavior at the destination. A very common element used in analyzing GPS tracking data is related to the speed of movement [4]. Speed is often used to detect different kinds of transportation modes [15, 34]. In this study, the 90th percentile of speed was used as a concise measure indicating the use of some kind of transportation mode during the visit, by considering that the average walking speed is generally below 5 km/hr. In order to compare the trajectories which cruise passengers followed at their destination, a Dynamic Time Warping algorithm (DTW) was used. DTW is an algorithm for measuring distance between two temporal sequences. It was introduced to analyze database communities [5]. Initially used for aligning 1D sequences, some authors have generalized DTW for 2D sequences [35], and it has been applied to obtain a distance between GPS tracking [20]. Although DTW may be computationally demanding, it has been applied in many fields, including bioinformatics [1], signatures [25], and fingerprints [22]. The major advantage of DTW over Euclidean distance relies on its ability to take into account the stretching and compression of

94

M. Ferrante et al.

sequences. A comparison of similarity measures for comparing trajectories has been proposed by Wang et al. [36]. This method calculates an optimal match between two given time series obtained by a local rescaling instead of performing global expansions of the series. As a result, this method produces a distance-like metric, which is independent of the velocity of the two temporal sequences. Specifically, let X = (x1 , . . . , xN ) and Y = (y1 , . . . , yM ) two generic time-series of points in a 2Dspace, and Dij = dist(xi , yj ) the cross-distance matrix of dimension N × M defined from the Euclidean distance. The DTW algorithm [28] remaps the time indexes of series X , Y to produce series of the same length T , which are termed warping curves φ(k) = (φx (k), φy (k)) with k = (1, . . . , T ). The monotonicity of the warping curve is usually imposed as a constraint to preserve time ordering. The DTW distance is therefore defined as the minimum Euclidean distance between each warping curves: DT W = minφ

T 

d (φx (k), φy (k))

k=1

However, when we are comparing alignments between time series of different lengths, it is usually convenient to manage an average per-step distance along the warping curve [24, 26, 30]. Therefore, the DTW distance is divided by the number of steps of the warping curve T . Having obtained the distance matrix among the trajectories, and in order to segment cruise passengers in relation to their behavior at their destination – of the wide range of methods for clustering observations [37] – an average linkage hierarchical clustering was implemented [12, 21]. Hierarchical clustering is a method of cluster analysis which provides a hierarchy of clusters. In this work we have adopted an agglomerative approach by initially assigning each observation to its own cluster, then by computing the similarity between each cluster and recursively joining the two most similar. Many similarity metrics can be used to agglomerate clusters [10, 16], and in this work the average linkage has been selected [33]. The distance between two clusters in the average linkage is defined as the average distance between each point in one cluster to every point in the other cluster:  1 d (x, y) |A | · |B| x∈A y∈B

where |A | and |B| are the size of the clusters and the distance is given by the DTW. This procedure leads to a rearrangement of the elements order which can be used to permute the rows of the distance matrix to reveal patterns of similarity, which would be otherwise hidden. Although the hierarchical clustering provides the complete hierarchy of the similarity, we are showing in this work a selected partition of trajectories of cruise passengers obtained by cutting the tree at a fixed level of similarity. This arbitrary choice has been made in order to have reasonable cluster sizes in order to compare

Similarity of GPS Trajectories Using Dynamic Time Warping …

95

the clusters to each other. Finally, after comparing clusters of cruise passengers based on the trajectories followed at their destination in terms of spatial patterns, the abovedefined metrics from GPS tracking data (i.e. total duration of tour, total length of tour, maximum distance from the port, and 90th percentile of speed) were compared among the different clusters by means of one-way ANOVA tests, in order to improve the characterization of the obtained clusters.

4 Results A total of 323 cruise passengers were sampled during the survey period. For the purposes of this study, only independent cruise passengers were considered. Moreover, those cruise passengers who visited destinations outside the city of Palermo (with a maximum distance from the port greater than 3.5 km) were excluded from the analysis. The final sample comprises 230 cruise passengers. By implementing DTW algorithm [14], the distance matrix among trajectories of cruise passengers was obtained (Fig. 1 left). Normalized distance among trajectories ranged from approximately 4 m to 1.3 km, with a strong concentration of observations at around 250 m. This revealed a general high degree of similarity among trajectories of cruise passengers at their destination. Figure 1 shows the dissimilarity plot derived from the distance matrix among trajectories followed by cruise passengers. We used a heat color scale – instead of numbers – in order to facilitate visualization, where the lighter colors indicate the shortest distance among trajectories and the units were ordered in relation to the results derived from the average linkage dendrogram [6]. Clusters of trajectories with homogeneous values of distances are clearly observable with the present ordering,

Fig. 1 DTW distance matrix: the original order of the units, shown in the left panel, does not reveal any useful information, however by sorting the units according to the hierarchical tree an evident pattern of similarity is highlighted, since lighter blocks represent very similar trajectories

96

M. Ferrante et al.

Fig. 2 Patterns of trajectories of the six largest clusters

since lighter blocks represent very similar trajectories. By considering a cut on a value of distance at 320 m, seven clusters of at least two elements were produced. Having produced the clusters based on similarities among trajectories, patterns of trajectories of the six largest clusters are reported in Fig. 2 (cluster 6 – with only 2 units – is not reported), thereby highlighting the different behavior of cruise passengers at their destination, for the different segments under consideration. Specifically, in the graphs reported in Fig. 2 the color of each polygon is proportional to the share of participants belonging to each cluster (with dark colors indicating a higher share), and the height of the polygons is proportional to the average time spent on each cell. Consequently, dark and high polygons represent places to which a high share of

Similarity of GPS Trajectories Using Dynamic Time Warping …

97

participants went and where they spent a high average period of time, the converse regards low and light polygons. Several considerations can be made by analyzing Fig. 2. Cluster 2 seems to be composed of those passengers who did not spent much time at their destination (with a lower height of polygons, compared to the other patterns), with minimal exploratory behavior in the city. Only the city center in the frontal port area was visited by those belonging to cluster 2. On the contrary, those passengers represented in clusters 3, 4, 5 and 7 seemed to reflect patterns related to a sightseeing tours in the main streets, including outwith the city center. Nonetheless, the tours of these clusters were rather different. More concentrated in the south-west part of the city were those passengers represented by cluster 4, with large numbers in both the eastern and the western areas of the city for cluster 7; less concentrated in the city center was cluster 3, and cluster 1 seemed to be the most heterogeneous in terms of trajectories. This is probably due to the fact that cruise passengers belonging to this group displayed more autonomous behavior on their visit, with less regularity in their movements, compared to the other groups. From an analysis of polygon color, it appears that almost all cruise passengers passed from the main street in front of the port, and only those belonging to cluster 4 were instead directed to the western part of the city along the coast. The tallest polygons highlighted places where the average time spent was higher. This permitted the identification of the most attractive areas, where cruise passengers spent much of their time. In order to check the validity of clustering based on DTW algorithm, several indexes of spatial behavior at the destination derived from GPS tracking data have been compared among clusters. Table 1 shows, for each cluster, mean values and standard errors of: total duration of tour, total length of tour, maximum distance from the port, and 90th percentile of speed. These results have been compared by means of ANOVA tests. An analysis of the results in Table 1 reveals strong differences regarding metrics derived from GPS tracking data as it appears among the different clusters of trajectories. The analysis of the metrics related to cruise passenger behavior at their destination enhance our understanding of the different itineraries. Specifically, cluster 2 describe those who made a very brief tour at the destination (of approximately 2 h, compared to an average duration of 3.7 h), shorter in length (11 km vs. 12.4 on average), with lowest values also for the maximum distance from the port and the 90th percentile of speed: the latter being approximately 4 km/hr indicates that no mode of transportation was used. Cluster 7 is characterized by the highest values in terms of duration of tour, and length. Nonetheless, the maximum distance from the port was lower compared to other groups. This was also evident by analyzing the pattern reported in Fig. 2, where the east-west shape of the tour followed determined lowest values for this index, compared to those who had visited attractions in the northern part of the city. The 90th percentile of speed relating to cluster 7 was also rather high, at approximately 10 km/hr, which confirmed the use of some kind of transportation mode during the visit.

98

M. Ferrante et al.

Table 1 One-way ANOVA tests on indexes of cruise passengers’ behavior at their destination in relation to cluster membership Variables

Cluster number #1

#2

#3

n = 151 n = 44 n = 8 Total dura- Mean tion of tour

#5

#7

n=8

n=7

n = 10 Total

F-test

P-value

3.921

2.080

3.109

4.658

5.077

5.539

3.669 15.157 0.000

0.136

0.164

0.541

0.457

0.637

0.363

0.117

Mean 13.424

5.524

Std. Err.

0.458

0.449

2.330

2.464

1.775

2.717

0.447

1.899

1.054

1.996

2.036

2.566

2.111

1.774 45.653 0.000

0.025

0.049

0.315

0.232

0.250

0.087

0.035

6.646

4.088

7.579

6.148

5.211

9.915

6.267

0.260

0.336

1.759

0.899

0.872

1.054

0.219

Std. Err. Total length of tour

#4

Maximum Mean distance from the port Std. Err. 90th per- Mean centile of speed Std. Err.

11.111 15.605 16.533 23.850 12.448 25.390 0.000

8.152 0.000

Cluster 4 and 5 looked very similar in terms of the metrics considered, with values of the total duration of tour at approximately around 5 h with a total length of tour at around 16 km. However, their patterns were quite different, as shown in Fig. 2, and this demonstrates the advantage of DTW in providing a complete comparison of trajectories; the latter in certain cases, may not be easily distinguishable when based on simple metrics. In other words, whereas the concise indexes of spatial behavior may be useful to answer the question on ‘how’ cruise passengers visited their destination, DTW also takes into account ‘where’ cruise passengers went. Similar considerations can be made for cluster 3, whereas cluster 1 presents values for the indexes which are quite similar to the average, also due to its higher size.

5 Conclusion Despite the importance of understanding the behavior of cruise passengers at their destination to improve the planning of services and the infrastructure of cruise destinations, there is a paucity of information and tools capable of monitoring and

Similarity of GPS Trajectories Using Dynamic Time Warping …

99

analyzing human mobility in time and space. Nowadays, the widespread availability of GPS technology permits for the collection of detailed information on human behavior, with a degree of precision at approximately 10 m. However, the complexity of this huge amount of information requires methods which are capable of compiling human trajectories. DTW allows for the comparison of GPS sequences of different lengths, and it is characterized by a relatively high degree of complexity (approximately 1,000 points for each trajectory). The analysis of human mobility is currently attracting the attention of researchers from many fields [29] and, although it can be perceived as random and unpredictable, we have found in this study that trajectories followed by cruise passengers at their destination are very similar. The results presented in this Paper may provide several insights for the urban planning of cruise destinations. Moreover, the implementation of cluster analysis on the trajectories of cruise passengers can assist in the identification of the relevant itineraries, thereby facilitating the exploration of more salient determinants of the mobility of cruise passengers at their destination. As a drawback of the proposed approach and future extensions, although timebased moving averages and other data-cleaning methods can help in correcting outlier observations, the efficiency of the DTW deteriorates if the two sequences are noisy, especially if they differ for some outliers. This behavior of the DTW is due to the exact matching of the sequences points. An alternative approach, proposed by [2], is the Longest Common Subsequence (LCSS). The idea of this method is to find the longest sub-sequence in common and, as a result, it is more resilient to outlier observations. In conclusion, GPS technologies can greatly improve the quality of data relating to the spatial-temporal activities of cruise passengers at their destination. Destination managers can use this information to plan decision-making, redirect visitor flows to avoid overcrowding, minimize an adverse impact on sensitive sites, and more broadly distribute expected benefits [11]. From a wider perspective, the knowledge of factors affecting the mobility of cruise passengers at their destination can expand our understanding of human mobility, in relation to the characteristics of a particular destination and individual factors, including the relationship between the characteristics of cruise passengers and their mobility.

References 1. Aach, J., Church, G.M.: Aligning gene expression time series with time warping algorithms. Bioinformatics 17(6), 495–508 (2001) 2. Andrienko, G., Andrienko, N., Rinzivillo, S., Nanni, M., Pedreschi, D., Giannotti, F.: Interactive visual clustering of large collections of trajectories. In: IEEE Symposium on Visual Analytics Science and Technology, 2009. VAST 2009. IEEE (2009) 3. Andriotis, K., Agiomirgianakis, G.: Cruise visitors experience in a Mediterranean port of call. Int. J. Tour. Res. 12(4), 390–404 (2010) 4. Bauder, M.: Using GPS supported speed analysis to determine spatial visitor behaviour. Int. J. Tour. Res. 17(4), 337–346 (2015)

100

M. Ferrante et al.

5. Berndt, D.J., Clifford, J.: Using dynamic time warping to find patterns in time series. KDD Work. 10(16), 359–370 (1994) 6. Bonanno, G., Lillo, F., Mantegna, R.N.: Levels of complexity in financial markets. Phys. A Stat. Mech. Appl. 299(1), 16–27 (2001) 7. Brida, J.G., Fasone, V., Scuderi, R., Zapata-Aguirre, S.: Exploring the determinants of cruise passengers expenditure at ports of call in Uruguay. Tour. Econ. 20(5), 1133–1143 (2014) 8. Cessford, G.R., Dingwall, P.R.: Tourism on New Zealands Sub-antarctic islands. Ann. Tour. Res. 21(2), 318–332 (1994) 9. De Cantis, S., Ferrante, M., Kahani, A., Shoval, N.: Cruise passengers’ behavior at the destination: investigation using GPS technology. Tour. Manag. 52, 133–150 (2016) 10. Defays, D.: An efficient algorithm for a complete link method. Comput. J. 20(4), 364–366 (1977) 11. Edwards, D., Griffin, T., Hayllar, B., Dickson, T.: Making Tracks and Collecting Images: New Methods for Examining Tourists’ Spatial Behaviour in Cities. In: Council for Australian University Tourism and Hospitality Education (Hrsg.), CAUTHE 2009, See Change: Tourism & Hospitality in a Dynamic World. Perth, pp. 2023–2026 (2009) 12. Eisen, M.B., Spellman, P.T., Brown, P.O., Botstein, D.: Cluster analysis and display of genomewide expression patterns. Proc. Natl. Acad. Sci. 95(25), 14863–14868 (1998) 13. Ferrante, M., De Cantis, S., Shoval, N.: A general framework for collecting and analyzing the tracking data of cruise passengers at the destination. Curr. Issues Tour. 1–26 (2016) 14. Giorgino, T.: DTW: Dynamic Time Warping algorithms. R package version 1.17.1 (2013) 15. Gong, H., Chen, C., Bialostozky, E., Lawson, C.T.: A GPS/GIS method for travel mode detection in New York City. Spec. Issue: Geoinformatics 2010 36(2), 131–139 (2012) 16. Gower, J.C., Ross, G.J.S.: Minimum spanning trees and single linkage cluster analysis. J. R. Stat. Soc. Ser. C 18(1), 54–64 (1969) 17. Guyer, C., Pollard, J.: Cruise visitor impressions of the environment of the Shannon-Erne waterways system. J. Environ. Manag. 51(2), 199–215 (1997) 18. Hallo, J.C., Manning, R.E., Valliere, W., Budruk, M.: A case study comparison of visitor selfreported and GPS recorded travel routes. In: Proceedings of the 2004 Northeastern Recreation Research Symposium, GTR-NE-326, Newton Square, PA: Forest Service, pp. 172–177 (2004) 19. Jaakson, R.: Beyond the tourist bubble? cruiseship passengers in port. Ann. Tour. Res. 31(1), 44–60 (2004) 20. Johnson, D., Trivedi, M.M.: Driving style recognition using a smartphone as a sensor platform. In: Proceedings of the 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), pp. 1609–1615 (2011) 21. Johnson, S.C.: Hierarchical clustering schemes. Psychometrika 2, 241–254 (1967) 22. Kovács-Vajna, Z.M.: A fingerprint verification system based on triangular matching and dynamic time warping. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1266–1276 (2000) 23. McKercher, B., Zoltan, J.: Tourists flows and spatial behavior. In: Lew, A.A., Hall, M.C., Williams, A.M. (eds.) The Wiley Blackwell Companion to Tourism, pp. 33–44. Wiley, Malden (2014) 24. Mori, A., Uchida, S., Kurazume, R., Taniguchi, R., Hasegawa, T., Sakoe, H.: Early recognition and prediction of gestures. In: Proceeding of the 18th International Conference on Pattern Recognition 2006, vol. 3, pp. 560–563 (2006) 25. Munich, M.E., Perona, P.: Continuous dynamic time warping for translation-invariant curve alignment with applications to signature verification. In: The Proceedings of the Seventh IEEE International Conference on Computer Vision, 1999. IEEE vol. 1, pp. 108–115 (1999) 26. Myers, C., Rabiner, L.R., Rosenberg, A.E.: Performance tradeoffs in dynamic time warping algorithms for isolated word recognition. IEEE Trans. Acoust. Speech Signal Process. 28(6), 623–635 (1980) 27. Puczkó, L., Bárd, E., Füzi, J.: Methodological triangulation: the study of visitor behaviour at the Hungarian open air museum. In: Richards, G., Munsters, W. (eds.) Cultural Tourism Research Methods, pp. 61–74. CABI, Wallingford (2010)

Similarity of GPS Trajectories Using Dynamic Time Warping …

101

28. Rabiner, L.R., Juang, B.-H.: Fundamentals of Speech Recognition. Tsinghua University Press, Beijing (1999) 29. Rhee, I., Shin, M., Hong, S., Lee, K., Kim, S.J., Chong, S.: On the levy-walk nature of human mobility. IEEE/ACM Trans. Netw. (TON) 19(3), 630–643 (2011) 30. Sakoe, H., Chiba, S.: Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans. Acoust. Speech Signal Process. 26(1), 43–49 (1978) 31. Shoval, N.: Tracking technologies and urban analysis. Cities 25(1), 21–28 (2008) 32. Shoval, N., Isaacson, M.: Tracking tourists in the digital age. Ann. Tour. Res. 34(1), 141–159 (2007) 33. Sokal, R., Michener, C.: A statistical method for evaluating systematic relationships. Univ. Kans. Sci. Bull. 38, 1409–1438 (1958) 34. Tsui, S.Y.A., Shalaby, A.: An enhanced system for link and mode identification for GPS-based personal travel survey. Transp. Res. Rec. 1972, 38–45 (2006) 35. Vlachos, M., Kollios, G., Gunopulos, D.: Discovering similar multidimensional trajectories. In: Proceedings of the 18th International Conference on Data Engineering. IEEE, pp. 673–684 (2002) 36. Wang, H., Su, H., Zheng, K., Sadiq, S., Zhou, X.: An effectiveness study on trajectory similarity measures. In: Proceedings of the Twenty-Fourth Australasian Database Conference, vol. 137, pp. 13–22 (2013) 37. Xu, R., Wunsch, D.: Survey of clustering algorithms. IEEE Trans. Neural Netw. 16(3), 645–678 (2005)

The Financial Stress Spillover: Evidence from Selected Asian Countries Zulfiqar Ali Shah, Muhammad Ejaz Majeed, Biagio Simonetti and Corrado Crocetta

Abstract The objective of the study is to analyze financial stress spillover among selected Asian countries, namely, China, Pakistan, Sri Lanka, Malaysia and India for the period from Jan 2001 to Dec 2009. The financial stress is measured by Financial Stress Index (FSI), a specially designed comprehensive measure of financial stress. The methodology of Yimlam 2012 is adopted for analyzing dynamics of variance decomposition among countries using FSI for the selected countries. The results of the study confirm that China and Pakistan are the largest transmitters of spillover towards other selected countries. Also the net spillover of China and Pakistan indicated to be positive whereas all other countries show up negative net spillovers. The economic and geographic linkages are suggested to be responsible for influencing magnitude of spillover among selected countries. Finally, the response of each country to shocks in other countries is found to be positive. Keywords Financial stress spillover · Asian countries · Inequality

1 Introduction Financial crises of 2008 has, once again, attained the attention of world to find out its causes to rectify these issues. The crisis has not limited to certain markets rather has spilled over across markets and across regions of the world. This certainly, has provoked the need to measure and tackle the contagion. Z. A. Shah · M. E. Majeed International Islamic University, Islamabad, Pakistan e-mail: [email protected] M. E. Majeed e-mail: [email protected] B. Simonetti (B) University of Sannio, Sannio, Italy e-mail: [email protected] C. Crocetta University of Foggia, Foggia, Italy e-mail: [email protected] © Springer Nature Switzerland AG 2019 C. Crocetta (ed.), Theoretical and Applied Statistics, Springer Proceedings in Mathematics & Statistics 274, https://doi.org/10.1007/978-3-030-05420-5_11

103

104

Z. A. Shah et al.

Substantial research has been undertaken in the past exploring the intensity and spillover of financial stress across countries. A rich literature is found addressing financial spillover from advanced to emerging countries. The research studies of Balakrishnan et al. [10] are leading in this direction. They utilized the Financial Stress Index developed by Cardarelli, Elekdag and Lall [15] and investigated the spillover of financial stress from advanced to emerging countries. The literature provides two principal channels of spillover: financial markets and trade. Studies of Eichengreen and Rose [22], Glick and Rose [28], and Forbes [24] evidenced the spillover on account of association of trade among countries. On the other hand Caramazza et al. [13] and Chui, Hall, and Taylor [19] documented the both channels for financial spillover across various countries simultaneously. The study of Balakrishnan et al. [10] evidenced strong co-movement of emerging and advanced economies during crisis. The intensity of contagion caused in emerging countries is proportional to linkages of financial markets in these countries with those of advanced countries. It was found that linkages of emerging markets with banking sector caused economic down turn in response to stress to banking sector in Europe due to decline in capital inflow. In study of Abbas et al. [1], it was argued that security markets of countries are influenced by linkages of economic, institutional and political factors and also by geographical closeness of these countries. Also linkages across boarder are weaken by business cycles. The crisis of 2008 results in increased correlation among markets across countries [36]. In addition, openness of investment opportunity among countries also contributes to co-movement of markets, particularly in crisis time [29]. Geographical integration is another factor of financial spillover evidenced by Aggarwal and Kyaw [2] in NAFTA equity market. Geographic integration was evidenced to be one of the determinants contributing to spillover among countries that are integrated geographically [33]. A rich literature is available focusing spillover across markets and regions. A majority of studies have adopted the methodology of GARCH for measuring this spillover. A new methodology has been introduced in recent past by Diebold and Yilmaz [21] who utilized Generalized Vector Autoregressive model to measure volatility spillover. According to Frobes and Rigobon [25], in high volatility periods, results obtained by correlation technique suffer from biasness in explaining spillover effects. The present study is first attempt in investigating empirically, the dynamic relationship between financial spillover during period from 2001 to 2009 among selected Asian countries including: Pakistan, China, India, Sri Lanka and Malaysia. The south Asian region is at very strategic geographical position. The region bridges East Asia, Middle East and Africa. Also these countries comprise the biggest consumer market of the world. Particularly, Pakistan, China and India have strong socio-economic association. The China-Pakistan Economic Corridor would make South Asia a center of the world to connect Europe, Asia, Africa and Australia contents. The objective of the study is to explore whether there exists volatility spillover among financial markets of these selected south Asian countries on account of geographic and economic integration among these countries.

The Financial Stress Spillover …

105

2 Literature Review This section documents in details the recent existing research work on the issue of spillover effects of financial markets’ crisis on other economies. The literature firstly highlights the contagion of crisis on developed economies, followed by emerging markets. Finally, the issue is discussed in context of developing economies/countries, particularly South Asia and Pakistan. The research of Balakrishnan et al. [10] was one of the leading studies in exploring the financial spillover effects. They provided comprehensive methodology of defining and constructing financial stress index. The study addresses the financial spillover from advanced countries to emerging economies with sample of 18 advanced countries and 26 emerging countries of the world with a data set of 15 years from 1997 to 2009. In addition, country specific and global factors are investigated as determinants of financial stress. The results of the study evidenced a quick transmission of volatility from advanced countries to emerging countries. Also the global financial crisis 2008 documented more severe impact on financial stress than Asian crisis and resulted in increased volatility of financial stress index in emerging countries. Cardarelli et al. [14] examine the impact of financial stress index on banking sector, stock market and exchange rate in advanced countries by providing an empirical framework. The study encompasses the period from 1981 to 2010 with 17 advanced countries of the world. The results of their study show that economic recession follows the financial stress. In particular, banking sector has greater impact on the economic downturn. Greenwood et al. [30] addressed financial spillover issue in G10 countries for the period from 1999 to 2014. They employed the Generalized VAR model to investigate the variance transmission of currencies across G10 countries. The results demonstrated significant transmission among returns of countries with strong linkages, particularly, exposition to common shocks. Inflated spillover is observed during GFC and sovereign debt crisis. Antonakakis and Vergos [6] investigated impact of financial crisis on the Euro bond market. The study adopted Diebold and Yilmaz [21] methodology and by applying GVAR, they concluded that shocks to bonds market are responsive to latest news. The results of study depict greater outwards transmission in currency market. Lee et al. [38] undertook a comprehensive study in order to explore the linkage between commodity market and stock market. The study presented a comparative analysis of oil exporting and importing countries by using generalized VAR model. They observed significant impact on return and volatility by oil exporting countries on oil importing countries. Moreover, this spillover indicated higher level in crisis times. Allen et al. [3] adopted univariant GARCH and VARMA-GARCH model to prove spillover among stock markets. The study focused on spillover effect from Chinese market to markets of its trading partners countries with a sample data from 1991 to 2010.

106

Z. A. Shah et al.

Allen et al. [4] employed Cholesky-GARCH model in order to explore the contagion effect from stock return of from US, China, Japan and Korea to Australia during period [18] to 2014 to capture the influence of financial crisis of 2008. By adopting Yilmaz [47] methodology, they used Financial Stress Index to carry out their research. US found to be dominant transmitter to Australia. GARCH proved significant impact of crisis of 2008 on the contagion effects in these countries. Similar study was conducted by Fernández et al. [23] by implementing VAR model to study the dynamic spillover within EMU region. The study encompassed period from 1999 to 2014 to capture the impact of financial crisis of 2008 on severity of spillover. The results proved prominent change in spillover during crisis period with varying magnitude for different countries. The study argues different factors for spillover and not only the financial crisis. Apostolakisa and Papadopoulos [7] presented in their study the security market to be the dominant driver of spillover among G7 countries. With a sample of 29 years (from 1981 to 2009) and using methodology of Yilmaz [21], they employed generalized VAR model in order to analyze dynamic spillover between Financial Stress Index and Consumer Price Index. Their study provided strong evidence of directional spillover from security markets to other markets. Another study of Apostolakisa and Papadopoulos [8] confirms the results of their previous study, by implementing similar methodology in banking, securities and exchange markets. According to their work, the securities market showed to be dominant transmitter of spillover to other markets. Grobys [31] examines the currency and exchange markets volatility spreads of US along with Asian countries. The study estimates spread in a different manner as those of Diebold and Yilmaz [21]. Using VAR model, a volatility spillover index is constructed. The study evidenced a sharp pre-crisis increment in spillovers index. In pre-crisis period individual financial market influenced by certain factor whereas all markets exhibited similar causes for volatility. Claeys and Vasicek [20] carried out research study focusing bond markets of European Union countries in period from 2000 to 2012 by utilizing Diebold and Yilmaz [21] model of GVAR. A significant spillover among EU countries is documented during the crisis time. The economic and financial linkages inflated with weak economic policy and imbalances were suggested to be main cause of rapid transmission of shocks among EU countries. The study of Babecky et al. [9] presented certain domestic factors responsible for economic crisis. They conducted research study that encompasses 40 years from 1970 to 2010 and focused EU and OECD countries. They adopted specially developed methodology of measuring crisis by economic costs. Their study found that hype of housing prices, share price and credit growth lead to economic crisis and argued these factors to be considered as signals of future economic crisis. Rimanl et al. [44] investigated stock market of Nigeria in relation with international oil prices for the period from 1980 to 2010. By utilizing VAR model, the study provides evidence of negative response of Nigerian stock market to international oil prices. The stability of market is confronted with declining international market and rising volatility.

The Financial Stress Spillover …

107

Mukulu et al. [40] attempted to examine US and Australian markets from 1989 to 2011. The distinct feature of their study is the construction of stress index for both countries by including factors from trade, securities and stock markets. The results of study indicate no causality of financial stress among both countries during crisis period. They argued that rapid adjustment of financial institutions in Australia in response to shock in US resulted in decrease in trade dominance and this might be considered as a sign to predict contagion risk of Australia in response to US markets. Mensi et al. (2015) undertook research to study US market along with top emerging markets of the world during period from 1997 to 2013 that encompasses the crisis of 2008. They applied DCCFIAPARCH and ICSS and VaR to investigate the spillover effects among selected countries of US, China, India, South Africa and Russia. Their work documented significant long memory and dynamic association among stock markets. Aloui et al. [5] carried out research addressing BRIC market along with US stock market in order to investigate conditional interdependence of these selected countries. The study presented the results of strong dependency between those markets that have commodity price dependence. Russia and Brazil proved strong dependency whereas India evidenced export-oriented markets dependency. Beirne et al. [12] provided evidence of strong spillovers of stock returns among emerging markets of Asia, Europe and MENA countries. The VAR-GARCH model was implemented to carry out hypothesis testing. The strength of linkages among markets of these countries differs from country to country however, spillover of returns was found to be leading one. The study of Wang et al. [46] addresses the linkages of stock markets pre and post Asian crisis of 1997. Their study focused US market in comparison with five emerging markets of Africa. They concluded decline in strength of linkages among stock markets following crisis. Mukherji [39] investigated the spillover effect of US stock returns on three emerging markets including China, India and Brazil. The study adopted VAR and FMOLS methods in order to examine the short run and long run relationships among the selected markets by controlling macroeconomic variables. The results indicated significant positive impact of US stock market returns in both short run and long run on other selected countries. Heryan and Ziegelbauer [32] studied the sovereign debt crisis in EMU and presented a comparison of spillover of bond volatilities among Greece and GIIPS countries before and after period of global crisis. The study applied GARCH and TARCH model in order to investigate the yield spread of Greece bond after the crisis. The results evidenced a significant change in bond yield and spreads of Greece due to crisis. Chiang et al. [17] made an attempt to explore U.S stock market in relation with other advanced and emerging countries of China, Russia, Brazil, and India during the crisis of 2007–08. The method of ARCJI model is utilized in order to test the hypothesis of transmission of effects of shocks among these countries. The results present Russia to be the largest receiver of effects of shocks from US whereas Vietnam

108

Z. A. Shah et al.

showed up the most intensive shock receiver. On the other hand China and Brazil documented the most risky among all countries. Basu [11] addressed the spillover of crisis from bond market during 1997 in East Asian region. The investor sentiment was considered to be main cause of spillover on account of their perception about the market. Kamil Yilmaz [47] presented a comparative analysis of return and volatility of stock among East Asian countries by utilizing variance decomposition through VAR. The sample period spans from 1991 to 2009 and monthly data of stock return was used. The study finds a prominent difference between return and volatility during the period. Return spillover evidenced strong linkages of East Asian countries whereas volatility spillover showed sharp increase during crisis. Similarly, Diebold and Yilmaz [21] carried out research in US market. By adopting generalized VAR they provided linkages among stock, bond and foreign exchange markets of US for the period ranging from 1999 to 2010. They found limited transmission of volatility among four markets, however, an increased spillover was documented during crisis of 2008. The epidemic of Asian Financial crisis of 1997 provoked a large number of researches to explore the causes of crisis and to search mechanism for protection in face of such collapse of financial markets. Among these research works, the study of [18, 26, 35, 45] and Wang et al. [46] addressed spillover of various markets in Asian region countries. Using MGARCH model, Khalid and Rajaguru [34] analyzed exchange rate linkages among South Asian exchange markets from 1996 to 2003 period. The results of the study evidenced no linkages of financial markets among selected countries. Chi et al. [16] undertook research to analyze financial association in East Asian countries for period of 1991 to 2005. They employed CAPM on three stock markets. The results proved strong linkage between financial markets and market integration increased during period. Similarly, Neaime [42] utilized GARCH model in order to setup linkages of stock markets. The study focus on MENA countries and results depicted spillover among countries. Trade openness showed up main factor of spillover in addition to country specific macro policies. In analyzing linkages between markets, the study of Khan and Sajid [35] employed interest rate parity hypothesis. The study of Gerard et al. [27] documented a strong integration of stock markets in East Asia. They adopted GARCH (1, 1) model and ICAMP approach. Neaime [41] utilized GARCH, TARCH and VAR model in order to examine dynamic volatility of stock markets in MENA countries. Strong integration resulted in easy transmission of stock volatility in MENA. The geographical integration and volatility spillover in stock markets was focus of study of Choudhry [18]. The sample comprised of Pakistan, India, Israel, Jordan, Greece and Turkey. By employing GARCH-t model, the study documented no association of stock returns and geographic closeness or/and good relations of countries. Qayyum and Kemal [43] undertook research work to address Pakistan stock market. The objective of their study remained to explore linkage of equity market and

The Financial Stress Spillover …

109

exchange market and transmission of volatility from one market to other. With weekly data of KSE 100 and foreign exchange, they found high linkage between the two markets. Strong regional influence was found among stock markets of Asia and Europe in the study of Sing et al. [45]. Abbas, Khan and Shah [1] carried out research for Asian countries and studied the stock markets for the period from 1997 to 2009. Their work indicated the volatility spillover among countries with good relations. In view of above literature, it can be concluded that a large number of studies are devoted to address the issue of financial spillover among various markets in different regions of the world. It is noted that most of the studies utilized various models of ARCH and GARCH in order to measure volatility spillover among markets and a few studies adopted VAR model. However, this study is unique in the respect that it adopted GVAR model introduced by Diebol and Yilmaz [21] to analyze financial spillover among selected south Asian countries.

3 Research Methodology 3.1 Methodology The present study has adopted the methodology of Diebold and Yilmaz [21]. The study is based on the generalized VAR framework developed by Diebold and Yilmaz [21] and attempt to explore stress spillovers between the developing selected South Asian countries. The model applied would be an N-variable, pth order VAR.

3.2 Financial Stress Index (FSI) A new methodology has been introduced in recent past by Diebold and Yilmaz [21] using Generalized Vector Autoregressive model to measure spillover volatility. DY devised a new index to measure financial stress. The index is a comprehensive measure that is constructed by three sectors that have been found important contributors for financial stress in the past literature. These sectors are banking sector, equity sector and foreign exchange market. A detailed of construction of Financial Stress Index (FSI) is presented in Appendix E. The index has certain advantages. All the data for variables included in the index can be obtained from market with high frequency. The index has additive edge over discrete measures used in the past studies for financial stress i.e. binary codes (yes/no) [37]. The index can be utilized for comparison of two different markets and different countries (advanced and developing). Although DY method provides advantage of measuring volatility spillover across market and across countries, it is suffered from some constraints. The VAR model used by DY is

110

Z. A. Shah et al.

reliant on the orders of variables included in the model. In addition, the methodology lacks in providing directions of spillover from one country, market or asset to other, since it provides aggregate volatility spillover. Another limitation of the DY methodology is homogeneity of variables (i.e. assets or markets) included in VAR model. Since financial crisis has infected and spilled over across different markets, it is of interest to study the spillover across diverse assets or markets. Diebold and Yilmaz [21] presented an enhanced version of their previous methodology by implementing Generalized Vector Autoregressive model. This model overcomes the dearth of DY 2009 methodology. With the DY methodology, three spillover effects can be measured. First is the Directional Spillover Transmitted from one country to other. Second is the Directional Spillover Received by one country from others. And third is the Net Spillover (Transmitted out – Transmitted in the country).

3.3 Sample Data The objective of the study is to analyze the financial stress spillover among selected countries of the South Asia. On account of availability of data from Asian countries, the sample is developed with five Asian countries including: China, Pakistan, India, Sri Lanka, and Malaysia. Whereas, Bangladesh, Bhutan, Maldives, Nepal were to be excluded from sample because of unavailability of data of these countries. The sample set consists of monthly data of Financial Stress Index from Jan 2001 to Dec 2009. The selection of these countries is on the basis of geographic integration as well as socio-economic linkage among countries. The data is collected from IFS prepared by IMF and World Development Index. The data for all the variables is primarily on monthly frequency. All the variables were standardized first, and then index is constructed by taking arithmetic mean of all five variables.

4 Results and Analysis 4.1 Descriptive and Correlation Analysis Table 1 depicts the descriptive statistics of financial stress index for all five countries for the sample period from Jan 2001 to Dec 2009. The results show that FSI for Malaysia is the highest among all the selected countries i.e. 0.0028 on average whereas the least average FSI is for Sri Lanka at 0.001. In regards to the volatility of FSI, it can be seen that there is not much difference among all the countries. Standard deviation of Pakistan, Malaysia and India are comparatively higher than Sri Lanka and China. Table represents slightly positive skewness of FSI of all the countries except Sri Lanka. Overall, the Jaque-Bera statistic confirms the normality of FSI for

The Financial Stress Spillover …

111

Table 1 Descriptive analysis China

India

Malaysia

Pakistan

Srilanka

Mean

0.0015

0.0018

0.0028

0.0012

0.0001

Maximum

14.4178

15.0298

14.5369

9.8135

9.0351

Minimum

−4.5461

−5.2339

−5.0076

−4.0473

−6.2141

Std. Dev.

2.7364

2.9815

2.9617

3.0709

2.3897

Skewness

1.3405

1.7852

1.3253

1.2166

0.6314

Kurtosis

8.6083

8.5260

7.1409

4.2155

4.1205

Jarque-Bera

173.8856

194.7790

108.7755

33.2891

12.8258

ADF

−3.862154

−4.66044

−5.009164

−3.271634

−5.405749

Table 2 Correlation analysis China India Malaysia Pakistan Sri Lanka

China

India

Malaysia

Pakistan

Sri Lanka

1

0.794

0.693

0.578

0.403

0.715

0.698

0.474

1

0.559

0.330

1

0.226

1

1

all the countries during the sample period. In addition, all the series (FSIs for all the countries) are found, through Augmented Dickey Fuller test, to be stationary at level. Correlation analysis is depicted in Table 2 that reveals the co-movements of financial stress among the selected Asian countries. The correlation is the highest between India and China i.e. positive 79.4%. The geographical and economic similarities of the two countries are indicators for this high association of markets. Both the countries have long boarders adjacent to each other and are the largest countries in the region with respect to population, geographical area and consumer markets size. This is followed by about 70% positive association among China, India, Pakistan and Malaysia. The economic, political and social integration among these countries is evident for this high co-movements of markets of these four countries. On the other hand, Sri Lanka has weak correlation with Pakistan and Malaysia but it is still positive. This result is also in line with the intensity of integration of Sri Lanka with these countries in the region. It is interesting to notice that financial stress of all selected countries move in the same direction during the period but with varying strength of association. The overall results confirms that all the selected countries are strongly integrated except Sri Lanka.

112

Z. A. Shah et al.

4.2 Analysis of Financial Stress Spillover The Financial Stress Spillover analyzed by VAR is presented in Appendix A. In order to select the lag length in VAR model, Lag Selection test is applied. All tests LR, FPE, AIC, SC and HQ criteria recommend 1 lag length except LogL (see Appendix B). So we have applied VAR [1] model to analyze financial stress spill over. For detection of autocorrelation, Residual Serial Correlation LM Test is applied that indicates no serial correlation of residuals in the model. Further, the VAR residual Heteroskedasticity test confirms the homogeneity in FSI of selected countries. The stability of the VAR [1] model is inspected through AR Roots test that indicates all roots are below 1 and within circle of 1 unit root radius (see Appendix C).

4.3 Directional Spillover Analysis The summary of Variance Decomposition is portrayed in Table 3. It is evident that China and Pakistan found to be the largest spillover transmitters towards other countries in the region with about 101.8% and 60.7% of error variance in 10 months ahead forecasting, respectively (Fig. 1). China is the most influential country in the south Asian region with the largest economic and political strength and hence is expected to be the largest transmitter of spillover in the region. but the results for Pakistan does not comply with prediction. Sri Lanka, as expected, showed up transmitting least financial spread towards other countries. The result supports with the study of Abbas et al. [1]. On the receiver end, India and Malaysia confirmed the largest countries that received financial contagion from other countries with values 65% and 54% of error variance respectively (Fig. 2). The higher values of volatility coefficients confirm the dependency and integration of markets of India and Malaysia on other countries in the region.

Table 3 Directional spillover analysis China China

India

Malaysia

17.9

Sri Lanka 0.0

Received from Others

79.2

2.3

India

37.9

35.3

2.4

24.4

0.1

64.7

Malaysia

35.1

5.2

46.5

12.6

0.6

53.5

Pakistan

14.1

6.6

1.0

78.4

0.0

21.6 33.1

Sri Lanka

0.7

Pakistan

14.7

11.6

0.9

5.9

66.9

Towards others

101.8

25.7

4.9

60.7

0.8

Net spillovers

80.9

−39.0

−48.6

39.1

−32.3

20.8

The Financial Stress Spillover …

113

101.8 60.7 25.7

China

Pakistan

India

4.9

0.8

Malysia

Sri Lanka

21.6

20.8

Pakistan

China

India

Malysia

-39.0

-48.6

Fig. 1 Spillover to other countries

64.7

53.5 33.1

India

Malysia

Sri Lanka

Fig. 2 Spillover received from other countries

80.9 39.1

China

Pakistan

Sri Lanka -32.3

Fig. 3 Net spillover

The Net Spillover (Transmitted Spillover minus Received Spillover) documented China and Pakistan to have positive spillover towards other countries with 81% and 39% of error variance forecasted in 10 months ahead, respectively. The positive net spillover for China is expected on the basis of its influential power but it is unexpected in case of Pakistan. All other three countries (Sri Lanka, India and Malaysia) remained net receivers of spillovers with −32%, −39% and −49%, respectively (Fig. 3). The negative net spillover for Malaysia and Sri Lanka can be expected but unexpected for India on account of its market size and economic position in south Asia. The lowest value of volatility transmission (both outward and inward) in case of Sri Lanka is supported by previous study of Khalid and Rajaguru [34] indicating no integration of Sri Lanka with other south Asian countries. By analyzing bi-country spillovers effects, Table 3 shows that India is influenced by 37.9% of error variance received from China and 24.4% error variance due to Pakistan. On contrary Pakistan received 14.1% spillover effects from China and only 6.6% from India. Similarly, the spillover statistics between Pakistan and Malaysia

114

Z. A. Shah et al.

prove that Pakistan is responsible for 12.6% of error variance in Malaysia while Malaysia contributes only 1% of error variance towards Pakistan. The analysis evidenced that socio-economic linkages among countries may contribute to enhance financial spillover effects.

4.4 Impulse Response Analysis The generalized Impulse response analysis is presented in Appendix D which is independent of ordering of countries in VAR. The results show positive and higher in magnitude response to a shock in all countries. As in accordance to variance decomposition the own country response is more than 2 SD (shown in diagonal boxes in Appendix D). It shows that the own country shock prevail for longer time due to longer time taken in developing and implementing reversal policies or the rigidity in the institutional set up to accept changes in policies. The off-diagonal figures show that response of each country to shock in other country is near to positive 1 SD.

5 Conclusion The objective of the study is to analyze financial stress spillover among selected Asian Countries, namely, China, Pakistan, India, Sri Lanka, and Malaysia. These countries have strong geographic and socio-economic linkages among them. The past studies addressing this issue and focusing Asian countries used Stock Market, Banking Sector or Exchange market alone, in order to measure financial stress of markets/countries and utilized modified ARCH or/and GARCH models to analyze volatility transmission in markets/countries. This study is unique in the respect that it utilizes Financial Stress Index (FSI), a comprehensive measure of financial stress, constructed by three sectors (i.e. Banking sector, Equity sector and exchange rate market) that have been found important contributors for financial stress in the past literature. The methodology of Yimlam [21] is adopted to provide dynamic of stress transmission among the selected countries. The sample period encompasses from Jan 2001 to Dec 2009. The results of the study confirm that China and Pakistan are the largest transmitters of spillover to other selected countries. Also China and Pakistan indicated positive net spillover whereas other three countries showed up negative net spillovers. The results for China (outward, inward and net spillover) are evident of China’s economic and political position in the region and are supported by previous study of Abbas et al. [1]. But the results are unpredictable in case of Pakistan. India and Malaysia showed up the largest receivers of financial contagion from other countries indicating their strong economic interaction with other three countries in the region. The bi-countries spillover results document India as receiver of financial stress from China. Pakistan has more outward transmission towards India as compare to inwards

The Financial Stress Spillover …

115

transmission from India. Similarly, Pakistan has more outward transmission than inward transmission of financial spillover effects from Malaysia. Sri Lanka showed up very least integration with other countries in the region with least coefficients of variance spillover. This may be attributed by least direct connection of Sri Lanka with these countries. The result is supported by Khalid and Rajaguru [34] of no integration of Sri Lanka with other south Asian countries. The overall results confirm that economic, geographical and social bonding exists among these selected Asian countries and this linkages contribute to enhance financial spread among countries. Moreover, the response of all countries to shocks to other countries is positive.

Appendix A: VAR Analysis

CHINA(−1)

India(−1)

Malaysia(−1)

Pakistan(−1)

Sri Lanka(−1)

China

India

Malaysia

Pakistan

0.302823

0.052225

0.121179

0.102787

Sri Lanka 0.00603

−0.13367

−0.12939

−0.13788

−0.11002

−0.1188

[ 2.26550]

[ 0.40363]

[ 0.87890]

[ 0.93422]

[ 0.05076]

0.036415

0.227214

−0.150422

−0.049573

0.191282

−0.14968

−0.14489

−0.1544

−0.12321

−0.13304

[ 0.24328]

[ 1.56818]

[−0.97427]

[−0.40236]

[1.43780]

0.042012

0.149417

0.486591

0.032349

−0.028551

−0.10854

−0.10507

−0.11196

−0.08934

−0.09647

[ 0.38705]

[ 1.42208]

[ 4.34605]

[ 0.36206]

[−0.29595]

0.290666

0.375003

0.250893

0.779128

0.011282

−0.0985

−0.09535

−0.1016

−0.08108

−0.08755

[2.95081]

[3.93291]

[2.46930]

[ 9.60934]

[ 0.12886]

0.007378

0.030333

0.086871

0.001282

0.456337

−0.10428

−0.10094

−0.10756

−0.08583

−0.09268

[ 0.07075]

[0.30051]

[0.80765]

[ 0.01494]

[4.92372]

0.024338

0.033375

−0.03342

−0.00558

−0.031657

−0.21202

−0.20523

−0.21869

−0.17452

−0.18844

[ 0.11479]

[0.16262]

[−0.15282]

[−0.03197]

[−0.16799]

R-squared

0.392873

0.520204

0.441596

0.671718

0.363847

Adj. R-squared

0.362817

0.496451

0.413953

0.655467

0.332355

Sum sq. resids

485.4769

454.8872

516.5255

328.9219

383.5076

C

S.E. equation

2.192419

2.122224

2.261441

1.80462

1.948616

F-statistic

13.07147

21.9012

15.97456

41.33251

11.55339

Log likelihood

−232.7346

−229.2527

−236.0513

−211.9065

−220.1208

Mean dependent

0.013182

0.018067

−0.031475

−0.02356

−0.026511

S.D. dependent

2.746576

2.990684

2.954057

3.074469

2.384808

116

Z. A. Shah et al.

Appendix B: VAR Lag Order Selection Criteria Lag

LogL

LR

FPE

AIC

SC

0

−1096.44

NA

1648.249

21.59685

21.72553

21.64896

1

−1003.83

174.3199*

438.1272*

20.27121*

21.04327*

20.58384*

2

−986.398

31.10711

509.9942

20.41957

21.835

20.99273

3

−971.511

25.10486

627.3644

20.61785

22.67666

21.45153

4

−954.259

27.40018

742.6635

20.76978

23.47195

21.86398

5

−933.901

30.33715

836.2091

20.8608

24.20635

22.21553

6

−913.877

27.87567

961.145

20.95838

24.94731

22.57363

* Indicates lag order selected by the criterion

Appendix C: Stability Test: AR Roots of Characteristic Polynomial Root

Modulus

0.828249

0.828249

0.515858

0.515858

0.301387 − 0.092044i

0.315129

0.301387 + 0.092044i

0.315129

0.305214

0.305214

No root lies outside the unit circle. VAR satisfies the stability condition

HQ

The Financial Stress Spillover …

117

Appendix D: Generalized Impulse Response Analysis

Appendix E Construction of Index The Financial Stress Index (FSI) comprises five variables, which are aggregated into an overall index to capture credit conditions in three financial market segments (banking, securities markets, and exchange markets). These five components all help associate the degree of financial stress with large swings in asset prices, abrupt changes regarding uncertainty and the appetite for risk, (international) liquidity conditions, credit availability and/or financial intermediation. The five components of the FSI are presented in table below: The choice of sub-indices was limited by data considerations and a preference for parsimony. To obtain the aggregate Financial Stress Index for each country the five components are standardized and summed up: FSI = β + Stock market returns + Stock market volatility + Sovereign debt spreads + EMPI

118

Z. A. Shah et al. Category

Variable

Measurement

1

Banking sector

Banking beta

CAPM: banking sector equity index

2

Security market

Stock return

3

Exchange markets

Stock volatility

GARCH (1, 1)

Sovereign debt spread

Bond Yield – 10y US T-Yield. Using JP Morgan EMBI Global Spread. o/w 5 year Credit Default Swap Spread

Exchange market pressure (EMPI)

[%age change Exchange Rate] – [%age Change of Total Reserve-Gold]

Further details on the definition of the five components (before standardization) and the aggregation method are given below: 1. Banking Sector: The Banking-sector beta is the standard capital asset pricing model (CAPM) beta, and is defined as follows:  M M C O V ri,t , ri,t βi,t  , 2 σi,M where: rM : The market returns rB : The banking returns The beta greater than one shows that banking stocks move more than proportionately with the overall stock market—suggests that the banking sector is relatively risky, and would be associated with a higher likelihood of a banking crisis. 2. Stock Market Stock Market Returns are the percentage change in the stock index. A decrease in equity prices corresponds to increased securities-market-related stress. Stock market volatility is a time-varying measure of market volatility obtained from a GARCH(1, 1) specification, using month-over-month real returns and modeled as an autoregressive process with 12 lags. Sovereign debt spreads is defined as the bond yield minus the 10-year United States Treasury yield using JPMorgan EMBI Global spreads. When EMBI data were not available, five-year credit default swap spreads were used. 3. Foreign Exchange Market The EMPI captures exchange rate depreciations and declines in international reserves, and is defined for country i in month t as:

The Financial Stress Spillover …

E M P Ii,t

119

    ei,t − μi,e R E Si,t − μi,R E S  − , σi,e σi,R E S

where: e: RES: μ: σ:

percentage change in exchange rate percentage change in total reserves minus gold the mean the standard deviation

All variables are in monthly or daily frequency. The index is constructed by taking the average of the components after adjusting for the sample mean and standardizing by the sample standard deviation. Then it is converted into quarterly frequency by taking the average of the monthly data.

References 1. Abbas, Khan S., Shah, Z.A.: Volatility spillover among regional Asian stock markets. Emerg. Mark. Rev. 16, 66 (2013) 2. Aggarwal, R., Kyaw, N.A.: Equity market integration in the NAFTA region: evidence from unit root and co-integration tests. Int. Rev. Financ. Anal. 14, 393–406 (2005) 3. Allen, D.E., Amram, R., McAleer, M.: Volatility spillovers from the Chinese Stock market to economic neighbours. Math. Comput. Simul. 94, 238–257 (2013). https://doi.org/10.1016/j. matcom.2013.01.001 4. Allen, D.E., McAleer, M., Powell, R.J., Singh, A.K.: Volatility spillovers from Australia’s major trading partners across the GFC. Int. Rev. Econ. Financ. (2016). http://dx.doi.org/10. 1016/j.iref.2016.10.007 5. Aloui, R., Ben Aissa, M.S., Nguyen, D.K.: Global financial crisis, extreme interdependences, and contagion effects: the role of economic structure? J. Bank. Financ. 35, 130–141 (2011) 6. Antonakakis, N., Vergos, K.: Sovereign bond yield spillovers in the Euro zone during the financial and debt crisis. J. Int. Financ. Mark. Inst. Money 26, 258–272 (2013). https://doi.org/ 10.1016/j.intfin.2013.06.004 7. Apostolakis, G., Papadopoulos, A.P.: Financial stress spillovers in advanced economies. J. Int. Financ. Mark. Inst. Money 32, 128–149 (2014). https://doi.org/10.1016/j.intfin.2014.06.001 8. Apostolakis, G., Papadopoulos, A.P.: Financial stress spillovers across the banking, securities and foreign exchange markets. J. Financ. Stab. 19, 1–21 (2015). https://doi.org/10.1016/j.jfs. 2015.05.003 9. Babecky, Jan, Havranek, Tomas, Mateju, Jakub, Rusnak, Marek, Smidkova, Katerina, Vasicek, Borek: Leading indicators of crisis incidence: evidence from developed countries. J. Int. Money Financ. 35(2013), 1–19 (2013) 10. Balakrishnan, R., Danninger, S., Tytell, I., Elekdag, S.A.: The transmission of financial stress from advanced to emerging economies (No. 09/133). Working Paper Series IMF (2009) 11. Basu, R.: Financial contagion and investor learning: an empirical investigation. International Monetary Fund (2002) 12. Beirne, J., Caporale, G.M., Schulze-Ghattas, M., Spagnolo, N.: Global and regional spillovers in emerging stock markets: a multivariate GARCH-in-mean analysis. Emerg. Mark. Rev. 11, 250–260 (2010) 13. Caramazza, F., Ricci, L.A., Salgado, R.: Trade and financial contagion in currency crises. IMF Working Paper WP/00/55, International Monetary Fund, Washington (2000)

120

Z. A. Shah et al.

14. Cardarelli, R., Elekdag, S., Lall, S.: Financial stress and economic contractions. J. Financ. Stab. 7, 78–97 (2011). http://dx.doi.org/10.1016/j.jfs.2010.01.005 15. Cardarelli, R., Elekdag, S., Lall, S.: Financial stress, downturns, and recoveries. (No. 09/100) IMF Working Paper (2009) 16. Chi, J., Li, K., Young, M.: Financial integration in East Asian equity markets. Pac. Econ. Rev. 11(4), 513–526 (2006) 17. Chiang, S.M., Chen, H.F., Lin, C.T.: The spillover effects of the sub-prime mortgage crisis and optimum asset allocation in the BRICV stock markets. Glob. Financ. J. 24, 30–43 (2013) 18. Choudhry, T.: International transmission of stock returns and volatility: empirical comparison between friends and foes. Emerg. Mark. Financ. Trade 40(4), 33–52 (2004) 19. Chui, Hall, Taylor: Crisis spillovers in emerging market economies: interlinkages, vulnerabilities and investor behavior. Bank of England, Working Paper No. 212 (2004) 20. Claeys, P., Vasícek, B.: Measuring bilateral spillover and testing contagion on sovereign bond markets in Europe. J. Bank. Financ. (2014). http://dx.doi.org/10.1016/j.jbankfin. Accessed 5 Nov 2014 21. Diebold, F.X., Yilmaz, K.: Better to give than to receive: predictive directional measurement of volatility spillovers. Int. J. Forecast. 28, 57–66 (2016). https://doi.org/10.1016/j.ijforecast. 2011.02.006 22. Eichengreen, B., Rose A.: Contagious currency crises: channels of conveyance. In: Ito, T., Krueger, A. (eds.) Changes in Exchange Rates in Rapidly Developing Economies. University of Chicago Press, Chicago (1999) 23. Fernández, F., Puig, M., Rivero, S.: Volatility spillovers in EMU sovereign bond markets. Research Institute of Applied Economics. Working Paper 2015/10- 1/32 (2015) 24. Forbes, K.: Are trade linkages important determinants of country vulnerability to crises? In: Paper Prepared for the NBER Conference on Currency Crises Prevention, National Bureau of Economic Research, Cambridge, Massachusetts, January 2001 25. Forbes, K.J., Rigobon, R.: No contagion, only interdependence: measuring stock market comovements. J. Financ. 57, 2223–2261 (2002) 26. Gallo, G.M., Velucchi, M.: Market interdependence and financial volatility transmission in East Asia. Int. J. Financ. Econ. 14, 24–44 (2009) 27. Gerard, B., Thanyalakpark, K., Batten, J.A.: Are the East Asian markets integrated? Evidence from the ICAPM. J. Econ. Bus. 55(5), 585–607 (2003) 28. Glick, R., Rose, A.K.: Contagion and trade: why are currency crises regional? J. Int. Money Financ. 18, 603–617 (1999) 29. Goetzmann, W., Ingersoll, J., Spiegel, M.I., Welch, I.: Sharpening sharpe ratios. National Bureau of Economic Research (2002) 30. Greenwood, M., Nguyen, V., Rafferty, B.: Risk and return spillovers among the G10 currencies. J. Financ. Mark. (2016). http://dx.doi.org/10.1016/j.finmar.2016.05.001 31. Grobys, Klaus: Are volatility spillovers between currency and equity market driven by economic states? Evidence from the US economy. Econ. Lett. 127(2015), 72–75 (2015) 32. Heryan, T., Ziegelbauer, J.: Relations between yields of government bonds in GIPS countries during the sovereign debt crisis in The EMU. 13th International Scientific Conference “Economic Policy in the European Union Member Countries” September 2–4, 2015, Karolinka, CZECH REPUBLIC (2015) 33. Janakiraman, S., Lamba, A.S.: An empirical examination of linkages between Pacific-basin stock markets. J. Int. Financ. Mark. Inst. Money 8, 155–173 (1998) 34. Khalid, A.M., Rajaguru, G.: Financial market linkages in South Asia: evidence using a multivariate GARCH model. Pak. Dev. Rev. 43(4), 585–603 (2004) 35. Khan, M.A., Sajid, M.Z.: Integration of financial markets in SAARC countries: evidence based on uncovered interest rate parity hypothesis. Kashmir Econ. Rev. 16(1), 1–16 (2007) 36. King, M., Wadhwani, S.: Transmission of volatility between stock markets. Rev. Financ. Stud. 1(3), 5–33 (1990) 37. Laeven, L., Valencia, F.: Systemic banking crises: a new database. Working Paper, unpublished, International Monetary Fund, Washington (2008)

The Financial Stress Spillover …

121

38. Lee, H., Liao, T., Huang, Y., Huang, T.: Dynamic spillovers between oil and stock markets: new approaches at spillover index. Int. J. Financ. Res. 6(2), 2015 (2015) 39. Mukherji, Ronit: Stock market efficiency in developing economies. J. Appl. Econ. Res. 9(4), 402–429 (2015) 40. Mukulu, Sandra, Hettihewa, Samanthala, Wright, Christopher S.: Financial contagion: an empirical investigation of the relationship between financial-stress indexes of Australia and the US. J. Appl. Bus. Econ. 16(3), 2014 (2014) 41. Neaime, S.: Volatilities in emerging MENA stock markets. Thunderbird Int. Bus. Rev. 48(4), 455–484 (2006) 42. Neaime, S.: The global financial crisis, financial linkages and correlations in returns and volatilities in emerging MENA stock markets. Emerg. Mark. Rev. 13(3), 268–288 (2012) 43. Qayyum, A., Kemal, A.R.: Volatility spillover between the stock market and the foreign exchange market in Pakistan. MPRA paper 1715. University Library of Munich, Germany (2006) 44. Riman1, H.B., Offiong, A.I., Egbe, I.E.: Effect of volatility transmission on domestic stock returns: evidence from nigeria. J. Int. Bus. Econ. 2, 189–219 (2014) 45. Singh, P., Kumar, B., Pandey, A.: Price and volatility spillovers across North American, European, and Asian stock markets. Int. Rev. Financ. Anal. 19, 55–64 (2010) 46. Wang, Z., Yang, J., Bessler, D.A.: Financial crisis and African stock market integration. Appl. Econ. Lett. 10, 527–533 (2003) 47. Yilmaz, K.: Return and volatility spillovers among the East Asian equity markets. J. Asian Econ. 21, 304–313 (2010). https://doi.org/10.1016/j.asieco.2009.09.001